Working with a large data set

A large data set is a real collection of many observations and several variables. At A level, its purpose is not to test whether you can memorise a spreadsheet. It is to make you use statistics as an investigation: understand the context, ask a precise question, select relevant data, analyse them with technology and communicate a justified conclusion.

Your awarding body prescribes the particular data set that you must study. Names and variables differ between boards, but the reasoning on this page applies to all of them.

Prerequisites

You should be able to:

• classify qualitative, discrete and continuous data;
• calculate the mean, median, quartiles, interquartile range and standard deviation;
• interpret box plots, histograms and scatter diagrams;
• distinguish a population from a sample.

Review presenting and interpreting data, averages and measures of spread or populations, samples and sampling if needed.

Know the structure before calculating

In a rectangular data table, each:

• row usually represents one case, observation or individual;
• column represents a variable;
• cell contains the value of one variable for one case.

This sounds simple, but an incorrect idea of what a row represents invalidates every later conclusion. A row might represent a person, one weather station on one day, or a monthly total for a region. These are not interchangeable.

Before analysing your prescribed data set, locate its documentation and answer:

1. What does one row represent?
2. Where and when were the data collected?
3. What are the units of every numerical variable?
4. Which variables are qualitative, discrete or continuous?
5. What do codes, blanks and symbols mean?
6. Are displayed values exact, rounded, estimated or measured?
7. Does the file contain the entire population of interest or a sample?

Worked example 1: reading metadata

Suppose a weather file has columns

Station	Date	Maximum temperature ($^\circ$C)	Rainfall (mm)	Wind direction
Northport	14 June	18.7	0.0	W
Hilltown	14 June	16.2	3.4	SW

One row represents one station on one date, not simply one day. Therefore two rows with the same date are not duplicates if their stations differ.

The maximum temperature and rainfall are continuous variables, even though they have been rounded. Wind direction is qualitative. A value of $0.0$ mm means that recorded rainfall was zero, while a blank might mean that rainfall was not recorded. Those meanings must not be confused.

Zero is not missing

A numerical zero is a measured or counted value. A blank, dash or special code may mean unavailable, not applicable or not recorded. Consult the data dictionary before changing anything.

Self-check 1

A transport table contains one row for each journey. Two rows have the same passenger category, date and duration, but different route identifiers. Should one automatically be deleted as a duplicate?

Answer

No. They may be two genuine journeys on different routes. A duplicate can only be identified using the variables that define a unique case, together with knowledge of how the data were collected.

The statistical enquiry cycle

A strong investigation follows a cycle:

$$\text{question}\longrightarrow\text{data}\longrightarrow\text{analysis} \longrightarrow\text{conclusion}\longrightarrow\text{refinement}.$$

1. Ask a precise question

“Investigate temperature” is too vague. A useful question identifies variables, groups and scope:

In the recorded June observations, was the typical maximum temperature higher at Northport than at Hilltown?

This suggests a filter, a grouping variable and suitable summary statistics. It does not yet assert anything about all future June weather.

2. Select relevant data

Filter only after deciding what evidence the question requires. Record every condition so the analysis is reproducible. For example,

$$S=\{x_i: \text{month}_i=\text{June and station}_i=\text{Northport}\}.$$

If a random sample is required, assign each eligible row a unique integer and select identifiers using a random number generator. Do not choose rows that look typical. See populations, samples and sampling for sampling methods and sources of bias.

3. Clean, but preserve the raw data

Work on a copy or add a cleaning flag. Never silently overwrite the original file. Check for:

• missing values;
• impossible or inconsistent values;
• duplicated cases;
• mixed units or category spellings;
• outliers that need investigation.

Cleaning is a reasoned decision, not a command to remove every unusual value. An extreme temperature may be genuine and scientifically important. An impossible temperature of $850^\circ$C is probably a recording or unit error. Learn formal outlier rules in outliers and cleaning data.

4. Summarise and display

Choose methods that answer the question:

Aim	Useful analysis
Compare typical values	median or mean
Compare consistency	IQR or standard deviation
Examine distribution shape	histogram or box plot
Examine association	scatter diagram, regression line and correlation
Compare category frequencies	frequency table or bar chart
Study change through time	time series plot

5. Conclude in context

A good conclusion states the evidence, answers the question and acknowledges limitations. It does not merely repeat calculator output.

Using a spreadsheet or statistical package

Technology lets you handle the full data set, but it does not decide what the output means. A reliable workflow is:

1. retain an untouched raw sheet;
2. give columns clear names and check units;
3. filter the intended rows;
4. check the filtered count;
5. calculate summaries and construct graphs;
6. inspect suspicious values against the raw data;
7. save the filters, formulae and decisions used.

For values $x_1,x_2,\ldots,x_n$, software may calculate

$$\bar{x}=\frac{\sum x_i}{n}, \qquad \sigma_x=\sqrt{\frac{\sum x_i^2}{n}-\bar{x}^{,2}}.$$

Some packages also report the sample standard deviation

$$s=\sqrt{\frac{\sum (x_i-\bar{x})^2}{n-1}}.$$

Check which statistic the software displays. At A level, a question may specify the convention or use calculator notation such as $\sigma_x$. The two answers are close for large $n$, but they are not identical.

Worked example 2: checking technological output

For the filtered values

$$12.1,\ 14.8,\ 15.3,\ 16.0,\ 20.8,$$

we have

$$\sum x=79.0, \qquad \sum x^2=1298.58, \qquad n=5.$$

Hence

$$\bar{x}=\frac{79.0}{5}=15.8$$

and

$$\sigma_x =\sqrt{\frac{1298.58}{5}-15.8^2} =\sqrt{10.076} \approx3.17.$$

The sample standard deviation is

$$s=\sqrt{\frac{5}{4}}\,\sigma_x\approx3.55.$$

If a spreadsheet returns $3.55$ when you expected $3.17$, the data may be correct but the chosen standard deviation function is different.

Always perform a plausibility check

The mean must lie between the minimum and maximum. A standard deviation cannot be negative. A frequency table should total the number of selected valid observations. These checks catch filters, ranges and missing values that were handled incorrectly.

Comparing subsets fairly

Large data sets invite comparisons, but the subsets must be comparable. Check that they use:

• the same variable and units;
• equivalent time periods;
• clearly defined groups;
• consistent treatment of missing values;
• enough observations to support the conclusion.

Worked example 3: comparing two locations

After cleaning, summer maximum temperatures give:

Location	$n$	Median ($^\circ$C)	IQR ($^\circ$C)	Mean ($^\circ$C)	SD ($^\circ$C)
Coast	$92$	$19.4$	$4.1$	$19.1$	$3.2$
Inland	$90$	$21.0$	$7.6$	$21.3$	$5.8$

Step 1: compare centre. The inland median is

$$21.0-19.4=1.6^\circ\text{C}$$

higher, and its mean is also higher.

Step 2: compare spread. The inland IQR is $3.5^\circ$C larger and its standard deviation is $2.6^\circ$C larger, so inland temperatures are less consistent.

Step 3: conclude within scope. In these recorded summer observations, the inland location was typically warmer but more variable. This does not prove that being inland causes the difference. Other variables, such as altitude, year or measurement conditions, may differ.

Self-check 2

Group A has mean $48.2$ and standard deviation $3.1$. Group B has mean $51.0$ and standard deviation $8.7$. Complete a contextual comparison without claiming that every value in B exceeds every value in A.

Answer

Group B has the higher mean, so its values are higher on average. Group B also has the larger standard deviation, so its values are more dispersed about their mean. The summaries do not imply that every B value is greater than every A value because the distributions may overlap.

Missing data, outliers and denominator traps

Suppose a filtered table has $200$ rows, but rainfall is missing in $12$ rows. If $47$ of the valid values record rain, the relevant proportion is

$$\frac{47}{200-12}=\frac{47}{188}=0.25,$$

provided the question asks for the proportion among observations with a recorded rainfall value. Using $47/200$ silently treats missing observations as “no rain”.

However, deleting missing values can also introduce bias. If instruments fail more often during severe weather, the $188$ complete cases may not be representative. State both the numerical treatment and its possible effect.

Worked example 4: an outlier decision

The lower and upper quartiles of wind speed are $Q_1=8.4$ and $Q_3=15.2$ knots. Then

$$\operatorname{IQR}=15.2-8.4=6.8.$$

The conventional outlier fences are

$$8.4-1.5(6.8)=-1.8, \qquad 15.2+1.5(6.8)=25.4.$$

A recorded speed of $28.1$ knots is therefore a potential outlier. This is a signal to investigate, not permission to delete it. Check the unit, source record, date and surrounding observations. If it is a genuine storm measurement, retaining it is usually correct. If the source confirms a transcription error, correct it and document the correction.

Bivariate data and lurking variables

When studying two variables, preserve the matched pairs $(x_i,y_i)$. A scatter diagram can reveal association, clusters, outliers and non-linear structure.

Suppose hotter days tend to have more ice cream sales. A positive correlation does not show that temperature is the only cause. Day type, holidays, location and visitor numbers may affect both variables. A lurking variable is an unrecorded or uncontrolled variable that may help explain the observed association.

Do not calculate a correlation after separately sorting the two columns. Sorting destroys the original pairings and can manufacture a strong but meaningless relationship. Continue with correlation and regression and correlation hypothesis tests.

From data to a probability model

Real data help you judge whether a model is plausible. For example, a Normal model may be questionable if a histogram is strongly skewed, has several distinct peaks or is truncated by a natural boundary. A binomial model needs a fixed number of trials, two outcomes, constant probability and independence.

The model is an approximation, not a description of every detail. Investigate choosing a probability distribution before applying the Normal distribution or the binomial distribution.

Exam reasoning and prescribed data set knowledge

Questions may provide a graph or numerical summary, but familiarity with your prescribed data set can give useful context. Know:

• what each case and variable represents;
• units, coding and rounding conventions;
• locations, categories and time periods;
• important gaps, anomalies and genuine unusual values;
• broad patterns you have discovered through exploration.

Do not rely on memorised numerical values that could be confused. Use supplied evidence first, then add accurate contextual knowledge.

Worked example 5: writing a complete conclusion

Question: “The median daily rainfall is $0$ mm at both sites. Conclude that their rainfall distributions are similar.”

A weak answer says: “They are similar because the medians are equal.”

A stronger analysis is:

1. Equal medians only show that the central recorded value is the same.
2. With rainfall, many dry days can make both medians zero.
3. The positive rainfall amounts, upper quartiles, maxima and proportion of wet days may differ greatly.
4. Compare spread and distribution shape before concluding similarity.

This is a common large data set theme: one statistic rarely captures the whole distribution.

Common misconceptions

• “Large means every member of the population.” A large file may still be a sample, and a large biased sample remains biased.
• “Technology guarantees correctness.” Software accurately performs the calculation you request, even if you selected the wrong rows or function.
• “All outliers are errors.” Some are errors; others are genuine extreme observations.
• “A blank equals zero.” Missing and zero values have different meanings.
• “Correlation explains cause.” Association alone cannot establish causation.
• “More decimal places mean greater accuracy.” Reported precision cannot exceed the precision justified by the original measurements.
• “A conclusion applies universally.” It applies first to the data, place, period and population actually studied.

Final self-check

A data file contains daily measurements from four sites over five years. You want to compare winter wind speeds at Sites A and B.

1. State a suitable precise question.
2. Give two filters needed.
3. Explain why unequal valid sample sizes do not by themselves prevent comparison.
4. Name two summaries that together compare centre and spread.
5. State one limitation of a causal conclusion.

Answer

1. For example: “In the recorded winter observations, how do the typical value and variability of wind speed compare between Sites A and B?”
2. Filter to winter months and to Sites A and B. Also exclude or flag missing wind speed values according to a stated rule.
3. Means, medians, IQRs and standard deviations can be compared for samples of different sizes. The differing sizes may affect reliability or reveal missingness, so the counts should still be reported.
4. For example, median and IQR, or mean and standard deviation.
5. Site is not randomly assigned and other variables such as altitude, exposure, season balance or instrument type may explain differences. Observational association does not establish causation.

Investigation checklist

Before finishing any large data set task, ask:

• Is my statistical question precise?
• Do I know what one row represents?
• Have I selected the correct variables, cases, units and time period?
• Have I treated missing values, errors and outliers transparently?
• Do my summaries and diagrams answer the question?
• Have I compared both centre and spread where appropriate?
• Is my conclusion contextual, supported and limited to a justified population?

The next step is to apply this workflow to your awarding body’s prescribed data set, then deepen the relevant technique through histograms, correlation and regression or hypothesis testing language.