Outliers and cleaning data

An outlier is an observation unusually far from the main body of data. It may be a recording error, a genuine rare event, or evidence that several populations have been mixed. Its cause matters: deleting every unusual value makes data look tidy but can destroy important evidence.

Data cleaning means finding and resolving errors, inconsistencies and ambiguities before analysis. Every change needs a reason that could be explained and reproduced.

Prerequisites

You should be able to:

• order data and find quartiles;
• calculate the interquartile range, mean and standard deviation;
• interpret box plots and summary statistics;
• distinguish qualitative, discrete and continuous variables.

Review averages and measures of spread and presenting and interpreting data if needed.

The $1.5\operatorname{IQR}$ rule

The interquartile range is

$$\operatorname{IQR}=Q_3-Q_1.$$

A common convention classifies $x$ as an outlier when

$$\boxed{x<Q_1-1.5\operatorname{IQR}} \qquad\text{or}\qquad \boxed{x>Q_3+1.5\operatorname{IQR}}.$$

The two boundary values are called the lower fence and upper fence. A value equal to a fence is not an outlier under these strict inequalities.

This is a flagging rule, not a definition that applies in every problem. If a question gives another rule, use that rule. Some questions define an outlier as more than $2$ standard deviations from the mean, for example.

Worked example 1: calculate the fences

For a set of journey times,

$$Q_1=18,qquad Q_3=30.$$

First calculate

$$\operatorname{IQR}=30-18=12.$$

The fences are

$$\begin{aligned} \text{lower fence}&=18-1.5(12)=0,\\ \text{upper fence}&=30+1.5(12)=48. \end{aligned}$$

Therefore a time below $0$ or above $48$ minutes is an outlier. The values $0$ and $48$ themselves are not outliers.

Do not compare with $1.5\operatorname{IQR}$ alone

The cutoffs are $Q_1-1.5\operatorname{IQR}$ and $Q_3+1.5\operatorname{IQR}$, not merely $\pm1.5\operatorname{IQR}$.

Worked example 2: raw data from start to finish

Consider the ordered data

$$2, 4, 5, 7, 8, 9, 10, 12, 13, 30.$$

There are $10$ values. Using the medians of the lower and upper halves,

$$Q_1=5,qquad Q_3=12.$$

Hence

$$\operatorname{IQR}=12-5=7$$

and

$$\begin{aligned} \text{lower fence}&=5-1.5(7)=-5.5,\\ \text{upper fence}&=12+1.5(7)=22.5. \end{aligned}$$

Only $30$ lies outside the fences, so only $30$ is flagged. Notice that $2$ is the same distance from $5$ as $15$ would be from $12$, but neither would be an outlier. Outlier status is determined by the fences, not by appearance.

Quartile conventions

Calculators and specifications can use slightly different quartile conventions. In an examination, use the convention expected by the question or awarding body. If quartiles are supplied, do not recalculate them.

Self-check 1

A data set has $Q_1=24$ and $Q_3=36$. Decide which of $5,6,54,55$ are outliers under the $1.5\operatorname{IQR}$ rule.

Answer

$$\operatorname{IQR}=36-24=12.$$

The fences are

$$24-1.5(12)=6,qquad 36+1.5(12)=54.$$

Thus $5$ and $55$ are outliers. The boundary values $6$ and $54$ are not.

Other stated outlier rules

An alternative rule may classify $x$ as an outlier if it lies more than $k$ standard deviations from the mean:

$$|x-\bar{x}|>k\sigma.$$

Equivalently,

$$x<\bar{x}-k\sigma \qquad\text{or}\qquad x>\bar{x}+k\sigma.$$

Worked example 3: a standard deviation rule

Suppose $\bar{x}=72.4$ and $\sigma=6.3$, and the rule is that values more than $2$ standard deviations from the mean are outliers.

$$\begin{aligned} \bar{x}-2\sigma&=72.4-2(6.3)=59.8,\\ \bar{x}+2\sigma&=72.4+2(6.3)=85.0. \end{aligned}$$

Therefore $59.8$ is not an outlier, $59.7$ is, $85.0$ is not, and $85.1$ is.

The mean and standard deviation are themselves sensitive to extreme values. This rule can therefore behave differently from the IQR rule, which uses resistant quartiles. Never switch rules simply because one produces the answer you expected.

What an outlier does to statistics

The mean, range and standard deviation are sensitive to extreme observations. The median and IQR are more resistant because they depend mainly on ordered positions.

Worked example 4: comparing before and after

For

$$8, 9, 10, 10, 11, 12, 45,$$

the mean is

$$\bar{x}=\frac{105}{7}=15,$$

while the median is $10$. Also,

$$\operatorname{range}=45-8=37.$$

If investigation establishes that $45$ was a transcription error for $15$, the corrected data have

$$\bar{x}=\frac{75}{7}=10.714\ldots,qquad \operatorname{median}=10,qquad \operatorname{range}=15-8=7.$$

The mean and range change greatly, but the median does not change. This does not justify replacing $45$ merely because it has a large effect. The correction is justified only by evidence about the recorded value.

For skewed data or data containing genuine extremes, reporting the median and IQR may represent centre and spread better than reporting the mean and standard deviation.

Investigate before changing

When a value is flagged, ask:

1. Is it possible? A negative age or a percentage above $100\%$ is usually impossible in its stated context.
2. Are the units consistent? A height recorded as $1.72$ among centimetre values may be in metres.
3. Does the source confirm it? Check the original form, instrument reading or database record.
4. Is it a duplicate or a distinct case? Matching values do not necessarily mean matching observations.
5. Could it be genuine? A record temperature or very long waiting time can be valid and important.
6. Does it belong to another population? An adult measurement in a children’s sample may indicate a selection error.

Possible decisions include:

Evidence	Defensible action
source confirms a typing error	correct it and record the correction
impossible value, correct value unknown	mark it missing or exclude that case from the relevant calculation
inconsistent units with known conversion	convert to common units and document it
genuine extreme observation	retain it, possibly reporting resistant summaries too
cause remains uncertain	flag it and compare analyses with and without it

An outlier is not automatically an error

The mathematical rule identifies unusual values. It cannot identify their cause. Contextual evidence determines whether a value should be corrected, excluded or retained.

A reproducible cleaning process

Preserve the raw data and work on a copy. For each variable:

1. check its meaning, type and units;
2. standardise category labels, such as Female, female and F, only when they mean the same thing;
3. distinguish zero from missing values;
4. test valid ranges and logical relationships;
5. identify duplicates using the fields that define one case;
6. calculate outlier flags where appropriate;
7. record every correction, exclusion and unresolved query.

A useful cleaning log contains

$$(\text{case identifier},\ \text{original value},\ \text{new value},\ \text{reason}).$$

This gives an audit trail and allows another analyst to reproduce the cleaned data.

Worked example 5: clean a small table

Suppose one row represents one runner in a $5$ km race.

Runner ID	Time (min)	Age	Club
101	$24.6$	$31$	North
102	$0$	$28$	South
103	$27.1$	$-4$	north
103	$27.1$	$-4$	north
104	$62.8$	$73$	North

Reason through each issue:

• A race time of $0$ is impossible for a finisher. Check whether it means missing, did not finish, or a recording error. Do not include it as a genuine time.
• Age $-4$ is impossible. The source is needed before it can be corrected.
• The repeated row for runner 103 is a likely duplicate because the identifier and all fields match. Confirm that runner ID uniquely identifies an entry before deleting one copy.
• North and north can be standardised if the data dictionary confirms they are the same club.
• $62.8$ minutes is plausible for a $73$ year old runner. Even if an outlier rule flags it, retain it unless evidence shows an error.

The example separates validity checks, which use context, from outlier checks, which use the distribution.

Analysing with and without a disputed value

If the status of a suspected value cannot be resolved, a sensitivity analysis shows whether it controls the conclusion.

Worked example 6: sensitivity analysis

Daily machine outputs are

$$98, 99, 100, 101, 102, 150.$$

With all values,

$$\bar{x}=\frac{650}{6}=108.33\ldots,qquad \operatorname{median}=100.5.$$

Without the disputed $150$,

$$\bar{x}=\frac{500}{5}=100,qquad \operatorname{median}=100.$$

The conclusion about typical output depends strongly on $150$ if the mean is used, but hardly changes if the median is used. A responsible report might state both analyses and explain that the source record for $150$ could not be verified. It should not quietly choose the version that supports a preferred claim.

Common misconceptions

• Every largest or smallest value is an outlier. Extremes are outliers only if they meet the stated rule.
• Every outlier should be deleted. Genuine extremes belong to the data and may be the most informative observations.
• A value inside the fences must be correct. Plausible values can still be mistyped, duplicated or assigned to the wrong case.
• A zero is the same as a blank. Zero is a numerical value; a blank may mean missing or not applicable.
• Cleaning is harmless preparation. Corrections and exclusions change results, so they must be justified and recorded.
• The IQR rule proves a distribution is abnormal. It only flags observations relative to the quartiles of that sample.

Final self-check

A sample has

$$Q_1=14.2,qquad Q_3=19.8.$$

1. Find the lower and upper outlier fences.
2. Classify $5.7$, $5.8$, $28.2$ and $28.3$.
3. A value of $28.3$ is confirmed by the original measuring instrument. Should it automatically be removed?
4. A value of $18.6$ was entered twice for the same unique case identifier. Is the fact that it is not an outlier enough reason to retain both rows?

Answer

First,

$$\operatorname{IQR}=19.8-14.2=5.6.$$

Therefore

$$\begin{aligned} \text{lower fence}&=14.2-1.5(5.6)=5.8,\\ \text{upper fence}&=19.8+1.5(5.6)=28.2. \end{aligned}$$

1. The fences are $5.8$ and $28.2$.
2. Values $5.7$ and $28.3$ are outliers. Values equal to the fences, $5.8$ and $28.2$, are not.
3. No. It is unusual but confirmed, so it should normally be retained and its effect discussed.
4. No. Outlier detection and duplicate detection are different checks. If the identifier truly defines a unique case, investigate the duplicate and retain only the genuine record or records.

Next steps

• Apply cleaning decisions in working with a large data set.
• Display outliers and compare groups in presenting and interpreting data.
• Examine how extremes affect correlation and regression.
• Review how biased collection can create misleading data in populations, samples and sampling.