Least squares regression

Least squares regression is a method for finding a line of best fit for a set of data. The goal is to choose the line that makes the sum of the squared vertical differences between the data points and the line as small as possible. Those vertical differences are called residuals.

We’ll use the slope-intercept form of a line:

$y = m x + b$

On the AP Statistics exam, you’ll have a calculator that can compute the least squares regression line from a list of data points. You should know how to do this on your calculator. The short video below shows how to find the line of best fit on a TI-84 (a common calculator for the AP exam), which you’ll use in the next example.

Example 1:

Consider the three points $(1, 5)$ , $(2, 8)$ , $(3, 14)$ . Use the least squares regression method to find the smallest possible sum of the squared errors.

Using the calculator, the line of best fit is:

$y = 4.5 x$

Next, find each residual (observed minus predicted), square it, and add the results. The original points are $(1, 5)$ , $(2, 8)$ , $(3, 14)$ . The predicted points on the regression line (using the same $x$ -values) are $(1, 4.5)$ , $(2, 9)$ , $(3, 13.5)$ .

$(5 - 4.5)^{2} (8 - 9)^{2} (14 - 13.5)^{2} = (0.5)^{2} = 0.25 = (- 1)^{2} = 1 = (0.5)^{2} = 0.25$

Solution to Example 1:

(spoiler)

$0.25 + 1 + 0.25 = 1.5$

Thus, the smallest possible value that the sum of squared errors can be is $1.5$ .

Definitions

Residual: The vertical difference between an observed value and a predicted value. The residual equals observed minus predicted.

Key facts about residuals

When the data point is above the regression line (i.e. the observed value is greater than the predicted value), then the residual is positive. When the data point is below the regression line (i.e. the observed value is less than the predicted value), then the residual is negative.
The sum (and mean) of the residuals is always zero.
The regression line minimizes the sum of the squares of the residuals.
A residual plot with a nonlinear pattern indicates that a nonlinear model would show a better fit to the data. If there is no apparent pattern or a linear pattern, then a linear model is appropriate.

Practice problem

Part a.

Example 2:

A scientist recorded the coffee drinking habits and sleeping habits of $10$ different people, where $x$ represents the number of coffee cups that the participant drank each day and $y$ represents the number of hours that they sleep each night.

The results were as follows: $(0, 8.5)$ , $(1, 8.0)$ , $(2, 7.0)$ , $(2, 7.5)$ , $(3, 6.5)$ , $(3, 6.8)$ , $(4, 6.0)$ , $(4, 6.2)$ , $(5, 5.5)$ , $(6, 5.0)$

a. What are the explanatory and response variables?

Solution to part a:

(spoiler)

The explanatory variable is coffee intake per day ( $x$ ) and the response variable is hours of sleep per night ( $y$ ).

Part b.

b. Create a scatter plot.

Solution to part b:

(spoiler)

Scatter plot showing coffee consumption compared to hours of sleep. — Coffee consumption vs hours of sleep

Part c.

c. Describe the relationship as either positive, negative, or neither.

Consider whether hours of sleep tend to increase or decrease as coffee consumption increases.

Solution to part c:

(spoiler)

The relationship is negative because, in general, as coffee consumption increases, hours of sleep decrease.

Part d.

d. Use your graphing calculator to find the equation of the regression line.

Use the method shown in the video from Example 1 to find the regression equation.

Solution to part d:

(spoiler)

$Y = - 0.6 X + 8.5$

Part e.

e. Interpret the slope in this context.

The slope tells you how $y$ (hours of sleep) changes for each one-unit increase in $x$ (cups of coffee).

Solution to part e:

(spoiler)

For the average participant, sleep decreases by $0.6$ hours for every additional daily cup of coffee consumed.

Part f.

f. Predict how much sleep someone would get who drank $3.5$ cups of coffee.

Substitute $x = 3.5$ into the regression equation.

Solution to part f:

(spoiler)

$Y = - 0.6 (3.5) + 8.5 = 6.4$

Thus, a person who drank $3.5$ cups of coffee is predicted to have gotten about $6.4$ hours of sleep.

Least Squares Regression

Finds line of best fit by minimizing sum of squared residuals
Uses slope-intercept form: $y = m x + b$
Residual = observed value minus predicted value

Calculating Regression Line and Errors

Use calculator to compute regression line from data points
Residuals: vertical differences between observed and predicted values
Sum of squared residuals (errors) is minimized by regression line

Key Facts About Residuals

Positive residual: observed value above regression line; negative residual: below line
Sum (and mean) of residuals always zero
Regression line minimizes sum of squared residuals
Nonlinear pattern in residual plot suggests nonlinear model may fit better

Variables in Regression

Explanatory variable: independent variable ( $x$ ), e.g., coffee intake per day
Response variable: dependent variable ( $y$ ), e.g., hours of sleep per night

Interpreting Regression Output

Slope: change in $y$ for each unit increase in $x$
- Example: sleep decreases by 0.6 hours per additional cup of coffee
Use regression equation to predict $y$ for given $x$ value

Scatter Plots and Relationship Direction

Scatter plot visualizes relationship between $x$ and $y$
Negative relationship: as $x$ increases, $y$ decreases

Summary Points

Equation of a line: $y = m x + b$
Line of best fit minimizes residuals
Positive/negative residuals indicate position relative to regression line
Correlation does not imply causation; confounding variables may exist

Least squares regression

We’ll use the slope-intercept form of a line:

$y = m x + b$

Example 1:

Consider the three points $(1, 5)$ , $(2, 8)$ , $(3, 14)$ . Use the least squares regression method to find the smallest possible sum of the squared errors.

Using the calculator, the line of best fit is:

$y = 4.5 x$

Next, find each residual (observed minus predicted), square it, and add the results. The original points are $(1, 5)$ , $(2, 8)$ , $(3, 14)$ . The predicted points on the regression line (using the same $x$ -values) are $(1, 4.5)$ , $(2, 9)$ , $(3, 13.5)$ .

$(5 - 4.5)^{2} (8 - 9)^{2} (14 - 13.5)^{2} = (0.5)^{2} = 0.25 = (- 1)^{2} = 1 = (0.5)^{2} = 0.25$

Solution to Example 1:

(spoiler)

$0.25 + 1 + 0.25 = 1.5$

Thus, the smallest possible value that the sum of squared errors can be is $1.5$ .

Definitions

Residual: The vertical difference between an observed value and a predicted value. The residual equals observed minus predicted.

Key facts about residuals

When the data point is above the regression line (i.e. the observed value is greater than the predicted value), then the residual is positive. When the data point is below the regression line (i.e. the observed value is less than the predicted value), then the residual is negative.
The sum (and mean) of the residuals is always zero.
The regression line minimizes the sum of the squares of the residuals.
A residual plot with a nonlinear pattern indicates that a nonlinear model would show a better fit to the data. If there is no apparent pattern or a linear pattern, then a linear model is appropriate.

Practice problem

Part a.

Example 2:

A scientist recorded the coffee drinking habits and sleeping habits of $10$ different people, where $x$ represents the number of coffee cups that the participant drank each day and $y$ represents the number of hours that they sleep each night.

The results were as follows: $(0, 8.5)$ , $(1, 8.0)$ , $(2, 7.0)$ , $(2, 7.5)$ , $(3, 6.5)$ , $(3, 6.8)$ , $(4, 6.0)$ , $(4, 6.2)$ , $(5, 5.5)$ , $(6, 5.0)$

a. What are the explanatory and response variables?

Solution to part a:

(spoiler)

The explanatory variable is coffee intake per day ( $x$ ) and the response variable is hours of sleep per night ( $y$ ).

Part b.

b. Create a scatter plot.

Solution to part b:

(spoiler)

Part c.

c. Describe the relationship as either positive, negative, or neither.

Consider whether hours of sleep tend to increase or decrease as coffee consumption increases.

Solution to part c:

(spoiler)

The relationship is negative because, in general, as coffee consumption increases, hours of sleep decrease.

Part d.

d. Use your graphing calculator to find the equation of the regression line.

Use the method shown in the video from Example 1 to find the regression equation.

Solution to part d:

(spoiler)

$Y = - 0.6 X + 8.5$

Part e.

e. Interpret the slope in this context.

The slope tells you how $y$ (hours of sleep) changes for each one-unit increase in $x$ (cups of coffee).

Solution to part e:

(spoiler)

For the average participant, sleep decreases by $0.6$ hours for every additional daily cup of coffee consumed.

Part f.

f. Predict how much sleep someone would get who drank $3.5$ cups of coffee.

Substitute $x = 3.5$ into the regression equation.

Solution to part f:

(spoiler)

$Y = - 0.6 (3.5) + 8.5 = 6.4$

Thus, a person who drank $3.5$ cups of coffee is predicted to have gotten about $6.4$ hours of sleep.

Least squares regression

We’ll use the slope-intercept form of a line:

$y = m x + b$

Example 1:

Consider the three points $(1, 5)$ , $(2, 8)$ , $(3, 14)$ . Use the least squares regression method to find the smallest possible sum of the squared errors.

Using the calculator, the line of best fit is:

$y = 4.5 x$

Next, find each residual (observed minus predicted), square it, and add the results. The original points are $(1, 5)$ , $(2, 8)$ , $(3, 14)$ . The predicted points on the regression line (using the same $x$ -values) are $(1, 4.5)$ , $(2, 9)$ , $(3, 13.5)$ .

$(5 - 4.5)^{2} (8 - 9)^{2} (14 - 13.5)^{2} = (0.5)^{2} = 0.25 = (- 1)^{2} = 1 = (0.5)^{2} = 0.25$

Solution to Example 1:

(spoiler)

$0.25 + 1 + 0.25 = 1.5$

Thus, the smallest possible value that the sum of squared errors can be is $1.5$ .

Definitions

Residual: The vertical difference between an observed value and a predicted value. The residual equals observed minus predicted.

Key facts about residuals

When the data point is above the regression line (i.e. the observed value is greater than the predicted value), then the residual is positive. When the data point is below the regression line (i.e. the observed value is less than the predicted value), then the residual is negative.
The sum (and mean) of the residuals is always zero.
The regression line minimizes the sum of the squares of the residuals.
A residual plot with a nonlinear pattern indicates that a nonlinear model would show a better fit to the data. If there is no apparent pattern or a linear pattern, then a linear model is appropriate.

Practice problem

Part a.

Example 2:

A scientist recorded the coffee drinking habits and sleeping habits of $10$ different people, where $x$ represents the number of coffee cups that the participant drank each day and $y$ represents the number of hours that they sleep each night.

The results were as follows: $(0, 8.5)$ , $(1, 8.0)$ , $(2, 7.0)$ , $(2, 7.5)$ , $(3, 6.5)$ , $(3, 6.8)$ , $(4, 6.0)$ , $(4, 6.2)$ , $(5, 5.5)$ , $(6, 5.0)$

a. What are the explanatory and response variables?

Solution to part a:

(spoiler)

The explanatory variable is coffee intake per day ( $x$ ) and the response variable is hours of sleep per night ( $y$ ).

Part b.

b. Create a scatter plot.

Solution to part b:

(spoiler)

Part c.

c. Describe the relationship as either positive, negative, or neither.

Consider whether hours of sleep tend to increase or decrease as coffee consumption increases.

Solution to part c:

(spoiler)

The relationship is negative because, in general, as coffee consumption increases, hours of sleep decrease.

Part d.

d. Use your graphing calculator to find the equation of the regression line.

Use the method shown in the video from Example 1 to find the regression equation.

Solution to part d:

(spoiler)

$Y = - 0.6 X + 8.5$

Part e.

e. Interpret the slope in this context.

The slope tells you how $y$ (hours of sleep) changes for each one-unit increase in $x$ (cups of coffee).

Solution to part e:

(spoiler)

For the average participant, sleep decreases by $0.6$ hours for every additional daily cup of coffee consumed.

Part f.

f. Predict how much sleep someone would get who drank $3.5$ cups of coffee.

Substitute $x = 3.5$ into the regression equation.

Solution to part f:

(spoiler)

$Y = - 0.6 (3.5) + 8.5 = 6.4$

Thus, a person who drank $3.5$ cups of coffee is predicted to have gotten about $6.4$ hours of sleep.

Least Squares Regression

Finds line of best fit by minimizing sum of squared residuals
Uses slope-intercept form: $y = m x + b$
Residual = observed value minus predicted value

Calculating Regression Line and Errors

Use calculator to compute regression line from data points
Residuals: vertical differences between observed and predicted values
Sum of squared residuals (errors) is minimized by regression line

Key Facts About Residuals

Positive residual: observed value above regression line; negative residual: below line
Sum (and mean) of residuals always zero
Regression line minimizes sum of squared residuals
Nonlinear pattern in residual plot suggests nonlinear model may fit better

Variables in Regression

Explanatory variable: independent variable ( $x$ ), e.g., coffee intake per day
Response variable: dependent variable ( $y$ ), e.g., hours of sleep per night

Interpreting Regression Output

Slope: change in $y$ for each unit increase in $x$
- Example: sleep decreases by 0.6 hours per additional cup of coffee
Use regression equation to predict $y$ for given $x$ value

Scatter Plots and Relationship Direction

Scatter plot visualizes relationship between $x$ and $y$
Negative relationship: as $x$ increases, $y$ decreases

Summary Points

Equation of a line: $y = m x + b$
Line of best fit minimizes residuals
Positive/negative residuals indicate position relative to regression line
Correlation does not imply causation; confounding variables may exist

Least squares regression

We’ll use the slope-intercept form of a line:

$y = m x + b$

Example 1:

Consider the three points $(1, 5)$ , $(2, 8)$ , $(3, 14)$ . Use the least squares regression method to find the smallest possible sum of the squared errors.

Using the calculator, the line of best fit is:

$y = 4.5 x$

Next, find each residual (observed minus predicted), square it, and add the results. The original points are $(1, 5)$ , $(2, 8)$ , $(3, 14)$ . The predicted points on the regression line (using the same $x$ -values) are $(1, 4.5)$ , $(2, 9)$ , $(3, 13.5)$ .

$(5 - 4.5)^{2} (8 - 9)^{2} (14 - 13.5)^{2} = (0.5)^{2} = 0.25 = (- 1)^{2} = 1 = (0.5)^{2} = 0.25$

Solution to Example 1:

(spoiler)

$0.25 + 1 + 0.25 = 1.5$

Thus, the smallest possible value that the sum of squared errors can be is $1.5$ .

Definitions

Residual: The vertical difference between an observed value and a predicted value. The residual equals observed minus predicted.

Key facts about residuals

When the data point is above the regression line (i.e. the observed value is greater than the predicted value), then the residual is positive. When the data point is below the regression line (i.e. the observed value is less than the predicted value), then the residual is negative.
The sum (and mean) of the residuals is always zero.
The regression line minimizes the sum of the squares of the residuals.
A residual plot with a nonlinear pattern indicates that a nonlinear model would show a better fit to the data. If there is no apparent pattern or a linear pattern, then a linear model is appropriate.

Practice problem

Part a.

Example 2:

A scientist recorded the coffee drinking habits and sleeping habits of $10$ different people, where $x$ represents the number of coffee cups that the participant drank each day and $y$ represents the number of hours that they sleep each night.

The results were as follows: $(0, 8.5)$ , $(1, 8.0)$ , $(2, 7.0)$ , $(2, 7.5)$ , $(3, 6.5)$ , $(3, 6.8)$ , $(4, 6.0)$ , $(4, 6.2)$ , $(5, 5.5)$ , $(6, 5.0)$

a. What are the explanatory and response variables?

Solution to part a:

(spoiler)

The explanatory variable is coffee intake per day ( $x$ ) and the response variable is hours of sleep per night ( $y$ ).

Part b.

b. Create a scatter plot.

Solution to part b:

(spoiler)

Part c.

c. Describe the relationship as either positive, negative, or neither.

Consider whether hours of sleep tend to increase or decrease as coffee consumption increases.

Solution to part c:

(spoiler)

The relationship is negative because, in general, as coffee consumption increases, hours of sleep decrease.

Part d.

d. Use your graphing calculator to find the equation of the regression line.

Use the method shown in the video from Example 1 to find the regression equation.

Solution to part d:

(spoiler)

$Y = - 0.6 X + 8.5$

Part e.

e. Interpret the slope in this context.

The slope tells you how $y$ (hours of sleep) changes for each one-unit increase in $x$ (cups of coffee).

Solution to part e:

(spoiler)

For the average participant, sleep decreases by $0.6$ hours for every additional daily cup of coffee consumed.

Part f.

f. Predict how much sleep someone would get who drank $3.5$ cups of coffee.

Substitute $x = 3.5$ into the regression equation.

Solution to part f:

(spoiler)

$Y = - 0.6 (3.5) + 8.5 = 6.4$

Thus, a person who drank $3.5$ cups of coffee is predicted to have gotten about $6.4$ hours of sleep.

Least Squares Regression

Finds line of best fit by minimizing sum of squared residuals
Uses slope-intercept form: $y = m x + b$
Residual = observed value minus predicted value

Calculating Regression Line and Errors

Use calculator to compute regression line from data points
Residuals: vertical differences between observed and predicted values
Sum of squared residuals (errors) is minimized by regression line

Key Facts About Residuals

Positive residual: observed value above regression line; negative residual: below line
Sum (and mean) of residuals always zero
Regression line minimizes sum of squared residuals
Nonlinear pattern in residual plot suggests nonlinear model may fit better

Variables in Regression

Explanatory variable: independent variable ( $x$ ), e.g., coffee intake per day
Response variable: dependent variable ( $y$ ), e.g., hours of sleep per night

Interpreting Regression Output

Slope: change in $y$ for each unit increase in $x$
- Example: sleep decreases by 0.6 hours per additional cup of coffee
Use regression equation to predict $y$ for given $x$ value

Scatter Plots and Relationship Direction

Scatter plot visualizes relationship between $x$ and $y$
Negative relationship: as $x$ increases, $y$ decreases

Summary Points

Equation of a line: $y = m x + b$
Line of best fit minimizes residuals
Positive/negative residuals indicate position relative to regression line
Correlation does not imply causation; confounding variables may exist