Correlation and linear regression

Bivariate data contain two measurements for each individual or object. If $x$ is a person’s height and $y$ is their arm span, each person supplies one pair $(x,y)$. We use correlation to describe the strength and direction of an association, and regression to model one variable in terms of the other.

These ideas answer different questions:

• correlation asks how closely the points follow a straight line;
• regression asks which straight line should be used to estimate one variable from the other.

Neither, by itself, establishes that one variable causes the other.

Prerequisites

You should be able to:

• plot coordinates and interpret the gradient and intercept of $y=a+bx$;
• calculate means and standard deviations;
• use summation notation and calculator statistics modes;
• distinguish a population from a sample.

Review averages and measures of spread if ar x, variance or standard deviation are unfamiliar.

Scatter diagrams

A scatter diagram plots one point for each paired observation. The variable thought to explain or predict is usually placed on the horizontal axis and called the explanatory variable. The variable being predicted is placed on the vertical axis and called the response variable.

Look for four features:

1. Direction: positive, negative or no apparent association.
2. Form: linear, curved, clustered or another pattern.
3. Strength: how tightly the points follow the form.
4. Unusual observations: outliers or influential points.

In positive correlation, larger $x$ values tend to occur with larger $y$ values. In negative correlation, larger $x$ values tend to occur with smaller $y$ values. A strong curved relationship can have weak linear correlation, so always inspect the scatter diagram before relying on one calculated number.

Correlation is not causation

A strong association can arise because $x$ causes $y$, because $y$ causes $x$, because a third variable affects both, or by coincidence. Ice cream sales and sunburn cases may be positively correlated because warm weather increases both. Ice cream does not cause sunburn.

Worked example 1: interpreting a context

A sample shows a negative association between the age of a used car and its selling price.

A careful conclusion is:

Older cars in this sample tended to have lower selling prices.

This describes association, direction and context. It does not claim that age is the only cause. Mileage, condition and model may also affect price.

If one point represents a very old but unusually expensive classic car, it may be an outlier. Removing it merely to strengthen the pattern would be unjustified. Investigate whether it is an error, a member of a different population, or a genuine observation.

Pearson’s product moment correlation coefficient

Pearson’s product moment correlation coefficient, usually written $r$, measures the strength and direction of linear association in a sample.

$$\boxed{-1\leq r\leq 1}$$

Its interpretation is:

Value of $r$	Linear association
close to $1$	strong positive
close to $0$	weak or no linear association
close to $-1$	strong negative
exactly $1$ or $-1$	perfect linear association

There are no universal boundaries at which correlation suddenly becomes “strong”. Context and sample size matter. An $r$ of $0.7$ may be impressive for noisy biological data but disappointing for repeated measurements from calibrated equipment.

Define the corrected sums of squares and products by

$$\begin{aligned} S_{xx}&=\sum(x-\bar x)^2 =\sum x^2-\frac{(\sum x)^2}{n},\\ S_{yy}&=\sum(y-\bar y)^2 =\sum y^2-\frac{(\sum y)^2}{n},\\ S_{xy}&=\sum(x-\bar x)(y-\bar y) =\sum xy-\frac{(\sum x)(\sum y)}{n}. \end{aligned}$$

Then

$$\boxed{r=\frac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}}.$$

The denominator is positive, so the sign of $r$ is the sign of $S_{xy}$. The division standardises the result, making $r$ dimensionless and unaffected by the units used.

Why the formula measures association

For a point with $x>\bar x$ and $y>\bar y$, both deviations are positive, so their product is positive. A point with both values below their means also gives a positive product. These points support positive association.

If one value is above its mean and the other below, the product is negative. Such points support negative association. Adding the products gives $S_{xy}$; standardising gives a value between $-1$ and $1$.

Worked example 2: calculating $r$

For five students, $x$ is the number of hours revised and $y$ is a test score.

$x$	$1$	$2$	$3$	$4$	$5$
$y$	$52$	$55$	$61$	$64$	$68$

The required totals are

$$\sum x=15,\quad \sum y=300,\quad \sum x^2=55,$$ $$\sum y^2=18130,\quad \sum xy=938,\quad n=5.$$

Therefore

$$S_{xx}=55-\frac{15^2}{5}=10,$$ $$S_{yy}=18130-\frac{300^2}{5}=130,$$

and

$$S_{xy}=938-\frac{15(300)}5=38.$$

Hence

$$r=\frac{38}{\sqrt{10\times130}}=1.053\ldots$$

This is impossible, since $r$ cannot exceed $1$. The contradiction exposes an arithmetic error: the correct value is

$$\sum xy=1(52)+2(55)+3(61)+4(64)+5(68)=941.$$

Thus $S_{xy}=41$ and

$$r=\frac{41}{\sqrt{1300}}=1.137\ldots$$

That is still impossible, revealing that $\sum y^2$ must also be checked. In fact,

$$\sum y^2=52^2+55^2+61^2+64^2+68^2=18170.$$

Now $S_{yy}=170$, so

$$\boxed{r=\frac{41}{\sqrt{10\times170}}=0.994\text{ to 3 s.f.}}$$

There is a very strong positive linear association between revision time and score in this sample. The deliberate checks illustrate an important safeguard: $|r|>1$ always signals an input or arithmetic error.

Calculator use

Enter paired values into the two variable statistics mode, then request $r$. Keep the original table available so that you can verify row pairing and recheck entries. A calculator output is only as reliable as its input.

Self-check 1

What can you conclude if $r=0$?

Answer

There is no measured linear association. You cannot conclude that the variables are unrelated. For example, points lying exactly on $y=x^2$ and arranged symmetrically about $x=0$ can have $r=0$ despite a perfect curved relationship.

The least squares regression line

The regression line of $y$ on $x$ predicts $y$ from a given value of $x$. Write it as

$$\boxed{y=a+bx},$$

where

$$\boxed{b=\frac{S_{xy}}{S_{xx}}} \qquad\text{and}\qquad \boxed{a=\bar y-b\bar x}.$$

The gradient $b$ is the predicted change in $y$ for a one unit increase in $x$. Its units are “units of $y$ per unit of $x$”. The intercept $a$ is the predicted $y$ when $x=0$, but it may have no practical meaning if $x=0$ is outside the observed range.

The line always passes through the mean point $(\bar x,\bar y)$ because

$$\bar y=a+b\bar x.$$

Why it is called least squares

For an observed point $(x_i,y_i)$, the fitted value is

$$\hat y_i=a+bx_i.$$

The vertical residual is

$$e_i=y_i-\hat y_i.$$

The least squares line chooses $a$ and $b$ to minimise

$$\sum e_i^2=\sum\left(y_i-a-bx_i\right)^2.$$

Residuals are squared so that positive and negative errors cannot cancel. A positive residual means the observed value lies above the line; a negative residual means it lies below.

Worked example 3: finding and interpreting a regression line

Return to the revision data. We found

$$S_{xx}=10,\qquad S_{xy}=41,$$

and the means are

$$\bar x=3,qquad \bar y=60.$$

So

$$b=\frac{41}{10}=4.1$$

and

$$a=60-4.1(3)=47.7.$$

The regression line of score on revision time is

$$\boxed{\hat y=47.7+4.1x}.$$

The gradient predicts an increase of $4.1$ marks for each additional hour of revision. This is an association based on the fitted model, not proof that forcing every student to revise one more hour will raise their score by exactly $4.1$.

For $x=3.5$ hours,

$$\hat y=47.7+4.1(3.5)=62.05.$$

The predicted score is about $62$ marks. This is an interpolation because $3.5$ lies within the observed range $1\leq x\leq5$.

For the student who revised for $4$ hours, the fitted score is

$$\hat y=47.7+4.1(4)=64.1.$$

The observed score was $64$, so the residual is

$$e=64-64.1=-0.1.$$

The model overestimates this score by $0.1$ marks.

Use unrounded coefficients

If your calculator stores more accurate values of $a$ and $b$, use those for predictions and round only the final answer. Early rounding can noticeably change an estimate.

Which regression line?

The regression line of $y$ on $x$ minimises vertical squared residuals and predicts $y$ from $x$. If the task is instead to predict $x$ from $y$, use the regression line of $x$ on $y$:

$$x=c+dy, \qquad d=\frac{S_{xy}}{S_{yy}}, \qquad c=\bar x-d\bar y.$$

These are generally different lines. Do not find $y$ on $x$ and rearrange it to predict $x$. Rearranging preserves the same geometric line, but it does not change which squared residuals were minimised.

Worked example 4: regression in the other direction

For the revision data,

$$d=\frac{S_{xy}}{S_{yy}}=\frac{41}{170}=0.241176\ldots$$

and

$$c=3-d(60)=-11.4705\ldots.$$

Thus the regression line of revision time on score is

$$\boxed{\hat x=-11.4705+0.241176y}.$$

To estimate revision time for a score of $63$,

$$\hat x=-11.4705+0.241176(63)=3.72\text{ hours}.$$

The $y$ on $x$ line would give $x=(63-47.7)/4.1=3.73$ hours. The answers happen to be close because the correlation is almost perfect, but only $x$ on $y$ is appropriate for predicting $x$.

Both regression lines pass through $(\bar x,\bar y)$. Their gradients also satisfy

$$\left(\frac{S_{xy}}{S_{xx}}\right) \left(\frac{S_{xy}}{S_{yy}}\right)=r^2.$$

Self-check 2

For a data set, $\bar x=8$, $\bar y=21$, $S_{xx}=40$ and $S_{xy}=-30$. Find the regression line of $y$ on $x$ and estimate $y$ when $x=10$.

Answer

$$b=\frac{-30}{40}=-0.75,$$ $$a=21-(-0.75)(8)=27.$$

Therefore

$$\hat y=27-0.75x.$$

When $x=10$,

$$\hat y=27-0.75(10)=19.5.$$

Interpolation and extrapolation

Interpolation predicts within the range of observed explanatory values. Extrapolation predicts beyond that range.

Interpolation is usually safer because it uses the model where data have actually been observed. Extrapolation assumes the same linear pattern continues into an unobserved region, which may be unrealistic.

Worked example 5: spotting an unsafe prediction

Suppose tree height was measured for ages from $2$ to $12$ years and the fitted model is

$$\hat h=0.8+0.55t,$$

where $t$ is age in years and $h$ is height in metres.

At $t=8$,

$$\hat h=0.8+0.55(8)=5.2\text{ m}.$$

This is interpolation and may be reasonable if the scatter diagram supports a linear model.

At $t=80$, the model predicts $44.8$ m. This is extreme extrapolation. Tree growth does not remain linear for a lifetime, so the numerical calculation is valid but the prediction is not credible.

Even interpolation deserves caution when there is an outlier, a curved pattern, changing variability, or a sample that is not representative of the target population.

Effect of coding and transformations

Adding a constant to every value, or multiplying every value by a positive constant, does not change $r$. Changing centimetres to metres therefore leaves correlation unchanged. Multiplying one variable by a negative constant reverses the sign of $r$.

The regression coefficients do change with units. If $x$ changes from metres to centimetres, a gradient measured per metre must be rescaled to a gradient per centimetre.

A curved relationship can sometimes be transformed into a linear one. For example, if

$$y=Ab^x,$$

then

$$\ln y=\ln A+x\ln b.$$

Correlation and regression may be applied to $(x,\ln y)$, not blindly to $(x,y)$. Learn more in linearising data.

Common misconceptions

• “A high $r$ proves causation.” It shows strong linear association in the data, not a causal mechanism.
• “An $r$ near zero means no relationship.” It means little linear association; a curved relationship may remain.
• “The regression line can be used both ways.” Use $y$ on $x$ to predict $y$, and $x$ on $y$ to predict $x$.
• “A precise calculator answer is an accurate prediction.” Model suitability and data range matter more than displayed decimal places.
• “An outlier should always be removed.” Remove data only for a defensible reason, such as a confirmed recording error or an explicitly different population.
• “The intercept must have a real interpretation.” It is algebraically necessary but may correspond to an impossible or unobserved input.

Mixed self-check

A study of six machines gives $\bar x=12$, $\bar y=38$, $S_{xx}=50$, $S_{yy}=200$ and $S_{xy}=-80$, where $x$ is machine age in years and $y$ is output in units per hour.

1. Calculate $r$ and interpret it.
2. Find the regression line of $y$ on $x$.
3. Estimate output for a machine aged $10$ years.
4. Explain one reason why predicting output at age $30$ years may be unreliable if observed ages were between $4$ and $16$ years.

Answers

$$r=\frac{-80}{\sqrt{50\times200}}=-0.8.$$

There is a strong negative linear association between machine age and output in this sample.

2. The gradient is

$$b=\frac{-80}{50}=-1.6,$$

so

$$a=38-(-1.6)(12)=57.2.$$

The line is

$$\hat y=57.2-1.6x.$$

3. At $x=10$,

$$\hat y=57.2-1.6(10)=41.2$$

units per hour.

4. An age of $30$ lies far beyond the observed range, so this is extrapolation. The linear trend may not continue that far. The machine might fail, be repaired or be replaced before then.

Exam checklist

Before finishing a correlation or regression question, check that you have:

• used paired observations in the correct rows;
• confirmed that $-1\leq r\leq1$;
• described association in context without claiming causation;
• selected the regression direction that matches the variable being predicted;
• interpreted the gradient with units and context;
• substituted into the unrounded regression equation;
• considered the observed range before accepting a prediction.

Next steps

Correlation strength can be judged formally using a hypothesis test. Continue to hypothesis tests for correlation. For broader issues about samples and conclusions, review populations, samples and sampling. For data quality and influential observations, see outliers and cleaning data.