The Regression Equation

Learning objectives

Data rarely fit a straight line exactly. Usually, you must be satisfied with rough predictions. Typically, you have a set of data with a scatter plot that appear to fit a straight line. This is called a line of best fit or least-squares regression line.

A random sample of 11 statistics students produced the data in element-50, where x is the third exam score out of 80 and y is the final exam score out of 200. Can you predict the final exam score of a random student if you know the third exam score?

The third exam score, x, is the independent variable, and the final exam score, y, is the dependent variable. We will plot a regression line that best fits the data. If each of you were to fit a line by eye, you would draw different lines. We can obtain a line of best fit using either the median-–median line approach or by calculating the least-squares regression line.

Let'’s first find the line of best fit for the relationship between the third exam score and the final exam score using the median-median line approach. Remember that this is the data from element-22 after the ordered pairs have been listed by ordering x values. If multiple data points have the same x values, then they are listed in order from least to greatest x (see data values where x = 71). We first divide our scores into three groups of approximately equal numbers of x values per group. The first and third groups have the same number of x values. We must remember first to put the x values in ascending order. The corresponding y values are then recorded. However, to find the median, we first must rearrange the y values in each group from the least value to the greatest value. fs-id1170308472605 shows the correct ordering of the x values but does not show a reordering of the y values.

With this set of data, the first and last groups each have four x values and four corresponding y values. The second group has three x values and three corresponding y values. We need to organize the x and y values per group and find the median x and y values for each group. Let’s now write out our y values for each group in ascending order. For group 1, the y values in order are 126, 133, 153, and 175. For group 2, the y values are already in order. For group 3, the y values are also already in order. We can represent these data as shown in fs-id1170308944039, but notice that we have broken the ordered pairs; (65, 126) is not a data point in our original set:

When this is completed, we can write the ordered pairs for the median values. This allows us to find the slope and y-intercept of the –median-median line.

The ordered pairs are (66.5, 143), (69, 159), and (71, 174).

The slope can be calculated using the formula $m-\frac{y_2-y_1}{x_2-x_1}\text{.}$ Substituting the median x and y values from the first and third groups gives $m=\frac{174-143}{71-66.5}\text{,}$ which simplifies to $m\approx 6.9\text{.}$

The y-intercept may be found using the formula $b=\frac{\Sigma y-m\Sigma x}{3}$, which means the quantity of the sum of the median y values minus the slope times the sum of the median x values divided by three.

The sum of the median x values is 206.5, and the sum of the median y values is 476. Substituting these sums and the slope into the formula gives $b=\frac{476-6.9\left(206.5\right)}{3}$, which simplifies to $b\approx -316.3\text{.}$

The line of best fit is represented as $y=mx+b\text{.}$

Thus, the equation can be written as y = 6.9x − 316.3.

The median–median line may also be found using your graphing calculator. You can enter the x and y values into two separate lists; choose Stat, Calc, Med-Med, and press Enter. The slope, a, and y-intercept, b, will be provided. The calculator shows a slight deviation from the previous manual calculation as a result of rounding. Rounding to the nearest tenth, the calculator gives the –median-median line of $y=6.9x-315.5\text{.}$ Each point of data is of the the form (x, y), and each point of the line of best fit using least-squares linear regression has the form (x, ŷ).

The ŷ is read y hat and is the estimated value of y. It is the value of y obtained using the regression line. It is not generally equal to y from data, but it is still important because it can help make predictions for other values.

The term y₀ – ŷ₀ = ε₀ is called the error or residual. It is not an error in the sense of a mistake. The absolute value of a residual measures the vertical distance between the actual value of y and the estimated value of y. In other words, it measures the vertical distance between the actual data point and the predicted point on the line, or it measures how far the estimate is from the actual data value.

If the observed data point lies above the line, the residual is positive and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative and the line overestimates that actual data value for y.

In linrgs_regeq2, y₀ – ŷ₀ = ε₀ is the residual for the point shown. Here the point lies above the line and the residual is positive.

ε = the Greek letter epsilon

For each data point, you can calculate the residuals or errors, y_i – ŷ_i = ε_i for i = 1, 2, 3, . . . , 11.

Each |ε| is a vertical distance.

For the example about the third exam scores and the final exam scores for the 11 statistics students, there are 11 data points. Therefore, there are 11 ε values. If you square each ε and add them, you get the sum of ε squared from i = 1 to i = 11, as shown below.

${\left({\epsilon }_1\right)}^2+{\left({\epsilon }_2\right)}^2+\mathrm{...}+{\left({\epsilon }_{11}\right)}^2=\stackrel{11}{\underset{i=1}{\Sigma }}{\epsilon }^2\text{.}$

This is called the sum of squared errors (SSE).

Using calculus, you can determine the values of a and b that make the SSE a minimum. When you make the SSE a minimum, you have determined the points that are on the line of best fit. It turns out that the line of best fit has the equation

where

$a=\overline{y}-b\overline{x}$

and $b=\frac{\sum \left(x-\overline{x}\right)\left(y-\overline{y}\right)}{\sum {\left(x-\overline{x}\right)}^2}$.

The sample means of the x values and the y values are $\overline{x}$ and $\overline{y}$, respectively. The best-fit line always passes through the point $(\overline{x},\overline{y})$.

The slope (b) can be written as $b=r\left(\frac{s_y}{s_x}\right)$ where s_y = the standard deviation of the y values and s_x = the standard deviation of the x values. r is the correlation coefficient, which shows the relationship between the x and y values. This will be discussed in more detail in the next section.

Least-Squares Criteria for Best Fit

The process of fitting the best-fit line is called linear regression. We assume that the data are scattered about a straight line. To find that line, we minimize the sum of the squared errors (SSE), or make it as small as possible. Any other line you might choose would have a higher SSE than the best-fit line. This best-fit line is called the least-squares regression line.

Third Exam vs. Final Exam Example

The graph of the line of best fit for the third exam/final exam example is as follows:

The least-squares regression line (best-fit line) for the third exam/final exam example has the equation

Understanding and Interpreting the y-intercept

The y-intercept, a, of the line describes where the plot line crosses the y-axis. The y-intercept of the best-fit line tells us the best value of the relationship when x is zero. In some cases, it does not make sense to figure out what y is when x = 0. For example, in the third exam vs. final exam example, the y-intercept occurs when the third exam score, or x, is zero. Since all the scores are grouped around a passing grade, there is no need to figure out what the final exam score, or y, would be when the third exam was zero.

However, the y-intercept is very useful in many cases. For many examples in science, the y-intercept gives the baseline reading when the experimental conditions aren’'t applied to an experimental system. This baseline indicates how much the experimental condition affects the system. It could also be used to ensure that equipment and measurements are calibrated properly before starting the experiment.

In biology, the concentration of proteins in a sample can be measured using a chemical assay that changes color depending on how much protein is present. The more protein present, the darker the color. The amount of color can be measured by the absorbance reading. fs-id1170308505621 shows the expected absorbance readings at different protein concentrations. This is called a standard curve for the assay.

The scatter plot figure_12_03_01 includes the line of best fit.

The y-intercept of this line occurs at 0.0226 mAU. This means the assay gives a reading of 0.0226 mAU when there is no protein present. That is, it is the baseline reading that can be attributed to something else, which, in this case, is some other non-protein chemicals that are absorbing light. We can tell that this line of best fit is reasonable because the y-intercept is small, close to zero. When there is no protein present in the sample, we expect the absorbance to be very small, or close to zero, as well.

Understanding Slope

The slope of the line, b, describes how changes in the variables are related. It is important to interpret the slope of the line in the context of the situation represented by the data. You should be able to write a sentence interpreting the slope in plain English.

Interpretation of the Slope: The slope of the best-fit line tells us how the dependent variable (y) changes for every one unit increase in the independent (x) variable, on average.

Third Exam vs. Final Exam Example

The Correlation Coefficient r

Besides looking at the scatter plot and seeing that a line seems reasonable, how can you determine whether the line is a good predictor? Use the correlation coefficient as another indicator (besides the scatter plot) of the strength of the relationship between x and y.

The correlation coefficient, r, developed by Karl Pearson during the early 1900s, is numeric and provides a measure of the strength and direction of the linear association between the independent variable x and the dependent variable y.

If you suspect a linear relationship between x and y, then r can measure the strength of the linear relationship.

What the Value of r Tells Us

• The value of r is always between –1 and +1. In other words, –1 ≤ r ≤ 1.
• The size of the correlation r indicates the strength of the linear relationship between x and y. Values of r close to –1 or to +1 indicate a stronger linear relationship between x and y.
• If r = 0, there is absolutely no linear relationship between x and y (no linear correlation).
• If r = 1, there is perfect positive correlation. If r = –1, there is perfect negative correlation. In both these cases, all the original data points lie on a straight line. Of course, in the real world, this does not generally happen.

What the Sign of r Tells Us

• A positive value of r means that when x increases, y tends to increase and when x decreases, y tends to decrease (positive correlation).
• A negative value of r means that when x increases, y tends to decrease and when x decreases, y tends to increase (negative correlation).
• The sign of r is the same as the sign of the slope, b, of the best-fit line.

The correlation coefficient is calculated as the quantity of data points times the sum of the quantity of the x-coordinates times the y-coordinates, minus the quantity of the sum of the x-coordinates times the sum of the y-coordinates, all divided by the square root of the quantity of data points times the sum of the x-coordinates squared minus the square of the sum of the x-coordinates, times the number of data points times the sum of the y-coordinates squared minus the square of the sum of the y-coordinates. It can be summarized by the following equation:

$r=\frac{n\Sigma \left(xy\right)-\left(\Sigma x\right)\left(\Sigma y\right)}{\sqrt{\left[n\Sigma x^2-{\left(\Sigma x\right)}^2\right]\left[n\Sigma y^2-{\left(\Sigma y\right)}^2\right]}}$

where n is the number of data points.

The Coefficient of Determination

The variable r² is called the coefficient of determination and it is the square of the correlation coefficient, but it is usually stated as a percentage, rather than in decimal form. It has an interpretation in the context of the data:

• $r^2\text{,}$ when expressed as a percent, represents the percentage of variation in the dependent (predicted) variable y that can be explained by variation in the independent (explanatory) variable x using the regression (best-fit) line.
• 1 – $r^2\text{,}$ when expressed as a percentage, represents the percentage of variation in y that is not explained by variation in x using the regression line. This can be seen as the scattering of the observed data points about the regression line.

Consider the third exam/final exam example introduced in the previous section.

• The line of best fit is: ŷ = –173.51 + 4.83x.
• The correlation coefficient is r = .6631.
• The coefficient of determination is r² = .6631² = .4397.

Interpret r² in the context of this example.

• Approximately 44 percent of the variation (0.4397 is approximately 0.44) in the final exam grades can be explained by the variation in the grades on the third exam, using the best-fit regression line.
• Therefore, the rest of the variation (1 – 0.44 = 0.56 or 56 percent) in the final exam grades cannot be explained by the variation of the grades on the third exam with the best-fit regression line. These are the variation of the points that are not as close to the regression line as others.

Chapter Review

A regression line, or a line of best fit, can be drawn on a scatter plot and used to predict outcomes for the x and y variables in a given data set or sample data. There are several ways to find a regression line, but usually the least-squares regression line is used because it creates a uniform line. Residuals, also called errors, measure the distance from the actual value of y and the estimated value of y. The sum of squared errors, or SSE, when set to its minimum, calculates the points on the line of best fit. Regression lines can be used to predict values within the given set of data but should not be used to make predictions for values outside the set of data.

The correlation coefficient, r, measures the strength of the linear association between x and y. The variable r has to be between –1 and +1. When r is positive, x and y tend to increase and decrease together. When r is negative, x increases and y decreases, or the opposite occurs: x decreases and y increases. The coefficient of determination, r², is equal to the square of the correlation coefficient. When expressed as a percentage, r² represents the percentage of variation in the dependent variable, y, that can be explained by variation in the independent variable, x, using the regression line.

Use the following information to answer the next five exercises. A random sample of 10 professional athletes produced the following data, where x is the number of endorsements the player has and y is the amount of money made, in millions of dollars.

Homework

Glossary

coefficient of correlation

a measure developed by Karl Pearson during the early 1900s that gives the strength of association between the independent variable and the dependent variable; $r=\frac{n{{\sum }^{\text{}}}^{\text{}}xy-\left[{{\sum }^{\text{}}}^{\text{}}x\right]\left[{{\sum }^{\text{}}}^{\text{}}y\right]}{\sqrt{(n{{\sum }^{\text{}}}^{\text{}}{x}^2-{\left[{{\sum }^{\text{}}}^{\text{}}x\right]}^2)(n{{\sum }^{\text{}}}^{\text{}}{y}^2-{\left[{{\sum }^{\text{}}}^{\text{}}y\right]}^2)}}$ where n is the number of data points
The coefficient cannot be more than 1 and less than –1. The closer the coefficient is to ±1, the stronger the evidence of a significant linear relationship between x and y.