Binomial hypothesis tests: critical regions and significance

A binomial hypothesis test uses sample data to assess a claim about the probability $p$ of success in a repeated trial. It asks whether the observed number of successes would be sufficiently unusual if a stated value of $p$ were true.

The test does not prove that a hypothesis is true or false. It makes a controlled decision from uncertain evidence.

Prerequisites

You should be able to:

• recognise when $X\sim\operatorname{B}(n,p)$ is an appropriate model;
• calculate individual and cumulative binomial probabilities;
• interpret $P(X\leq x)$, $P(X\geq x)$ and complements;
• understand the language of hypothesis testing.

Review the binomial distribution if you are not yet confident with cumulative probabilities.

The central idea

Suppose a coin is claimed to have probability $0.5$ of landing heads. It is tossed $20$ times and gives $17$ heads. If the claim is true, then

$$X\sim\operatorname{B}(20,0.5),$$

where $X$ is the number of heads.

Seventeen heads is possible under this model, but possibility is not the issue. We ask:

How probable is a result at least as extreme as the one observed, assuming the claim is true?

If that probability is at most the chosen significance level, usually $5\%$, the result is called significant and the null hypothesis is rejected.

Hypotheses and tails

The null hypothesis $H_0$ gives the value of $p$ used in the probability calculation. The alternative hypothesis $H_1$ expresses the departure being investigated.

Investigation	Null hypothesis	Alternative hypothesis	Test
Has $p$ increased?	$H_0:p=p_0$	$H_1:p>p_0$	upper tailed
Has $p$ decreased?	$H_0:p=p_0$	$H_1:p<p_0$	lower tailed
Has $p$ changed?	$H_0:p=p_0$	$H_1:p\neq p_0$	two tailed

The wording of the claim determines the alternative hypothesis. Write the hypotheses before looking at whether the observed result is high or low.

The null hypothesis contains equality

The probability model needs one precise value of $p$, so $H_0$ is written with equality. Statements such as $H_0:p\geq0.4$ are not used for these A level binomial tests.

Worked example 1: choosing the hypotheses

A seed supplier claims that $85\%$ of a variety germinate. A gardener suspects that the germination rate is lower.

Let $p$ be the probability that a randomly selected seed germinates. The suspicion is specifically about a decrease, so

$$\boxed{H_0:p=0.85}, \qquad \boxed{H_1:p<0.85}.$$

This is a lower tailed test.

If the question had asked whether the advertised rate was incorrect, the alternative would instead be $H_1:p\neq0.85$.

The significance level

The significance level $\alpha$ is the largest intended probability of rejecting $H_0$ when $H_0$ is actually true. Common choices are

$$\alpha=0.05 \quad\text{and}\quad \alpha=0.01.$$

A $1\%$ test demands stronger evidence than a $5\%$ test. A result can therefore be significant at $5\%$ but not at $1\%$.

Rejecting a true null hypothesis is a Type I error. The significance level controls the probability of that error. Because a binomial distribution is discrete, the actual probability of rejection is often less than, rather than exactly equal to, $\alpha$.

Method 1: compare a tail probability with $\alpha$

For an observed value $x$:

• upper tailed test: calculate $P(X\geq x)$;
• lower tailed test: calculate $P(X\leq x)$;
• two tailed test: use both tails, with the significance level shared between them.

The tail probability for a one tailed test is the probability, under $H_0$, of the observed result or one more extreme in the direction of $H_1$.

Then apply the decision rule

$$\boxed{\text{reject }H_0\text{ if the relevant probability is }\leq\alpha.}$$

Equality leads to rejection because the critical region is constructed to have probability at most $\alpha$.

Worked example 2: an upper tailed test

A machine normally produces acceptable components with probability $0.80$. After servicing, $25$ randomly selected components are inspected and $24$ are acceptable. Test at the $5\%$ significance level whether the probability of an acceptable component has increased.

Let $p$ be the probability that a component is acceptable. Then

$$H_0:p=0.80, \qquad H_1:p>0.80.$$

Under $H_0$,

$$X\sim\operatorname{B}(25,0.80).$$

The alternative points towards unusually large values, so calculate

$$\begin{aligned} P(X\geq24) &=P(X=24)+P(X=25)\\ &=\binom{25}{24}(0.8)^{24}(0.2)+(0.8)^{25}\\ &\approx0.02739. \end{aligned}$$

Since

$$0.02739<0.05,$$

reject $H_0$. There is sufficient evidence at the $5\%$ significance level to suggest that the probability of an acceptable component has increased.

Conclusion in context

Give the statistical decision and interpret it using the language of the question. Do not finish with only ”$H_0$ is rejected”.

Worked example 3: a lower tailed test

A player has historically made a free throw with probability $0.70$. In a training session, she makes $11$ of $20$ attempts. Test at the $5\%$ level whether her success probability has decreased.

Let $p$ be the probability that she makes a free throw.

$$H_0:p=0.70, \qquad H_1:p<0.70.$$

Under $H_0$,

$$X\sim\operatorname{B}(20,0.70).$$

Since $11$ is in the direction of $H_1$,

$$P(X\leq11)\approx0.1133.$$

As $0.1133>0.05$, do not reject $H_0$. There is insufficient evidence at the $5\%$ significance level to suggest that her free throw probability has decreased.

This does not establish that $p=0.70$. The sample has simply failed to provide sufficiently strong evidence against that value.

Self-check 1

A website has historically converted $12\%$ of visits into sales. After a redesign, $18$ sales occur in $100$ independent visits. A test of $H_0:p=0.12$ against $H_1:p>0.12$ gives

$$P(X\geq18)=0.0603.$$

State the conclusion at the $5\%$ significance level.

Answer

Since $0.0603>0.05$, do not reject $H_0$. There is insufficient evidence at the $5\%$ level to suggest that the redesign has increased the conversion probability.

The observed rate $18\%$ is greater than $12\%$, but the difference is not statistically significant at the stated level.

Critical regions

A critical region is the set of values of $X$ that cause $H_0$ to be rejected. Its boundary value is a critical value.

For an upper tailed test, the critical region has the form

$$X\geq c,$$

where $c$ is the smallest integer satisfying

$$P(X\geq c)\leq\alpha.$$

For a lower tailed test, it has the form

$$X\leq c,$$

where $c$ is the largest integer satisfying

$$P(X\leq c)\leq\alpha.$$

You must check the neighbouring value to show that the region is as large as possible without exceeding the significance level.

Worked example 4: upper critical region

Under $H_0$, $X\sim\operatorname{B}(20,0.4)$. Find the upper critical region for a $5\%$ test.

Calculator values give

$$P(X\geq12)=0.0565$$

and

$$P(X\geq13)=0.0210.$$

The region $X\geq12$ is too probable because $0.0565>0.05$. The next possible region is within the limit because $0.0210\leq0.05$. Therefore

$$\boxed{X\geq13}$$

is the critical region, with critical value $13$.

Its actual significance level is

$$\boxed{P(X\geq13)=0.0210}.$$

It is not $0.05$. No integer boundary produces every probability between $0.0210$ and $0.0565$.

Worked example 5: lower critical region

Under $H_0$, $X\sim\operatorname{B}(30,0.6)$. Find the lower critical region for a $5\%$ test.

We seek the largest $c$ for which $P(X\leq c)\leq0.05$. Suppose the cumulative probabilities are

$$P(X\leq13)=0.0481, \qquad P(X\leq14)=0.0971.$$

Including $14$ would make the probability exceed $0.05$, so

$$\boxed{X\leq13}$$

is the critical region. Its actual significance level is $0.0481$.

If the observed value were $14$, it would lie just outside the critical region, so $H_0$ would not be rejected at $5\%$.

Self-check 2

Under $H_0$, $X\sim\operatorname{B}(40,0.25)$. For a lower tailed test,

$$P(X\leq5)=0.0433, \qquad P(X\leq6)=0.0962.$$

State the $5\%$ critical region and its actual significance level.

Answer

The critical region is

$$\boxed{X\leq5}.$$

Its actual significance level is

$$\boxed{0.0433}.$$

The value $6$ cannot be included because $0.0962>0.05$.

Two tailed tests

A two tailed test uses

$$H_1:p\neq p_0.$$

Both unusually small and unusually large observations count as evidence against $H_0$. At overall significance level $\alpha$, A level questions normally place at most $\alpha/2$ in each tail.

For a $5\%$ two tailed test, find:

• a lower region with probability at most $0.025$;
• an upper region with probability at most $0.025$.

The two parts need not have equal actual probabilities. A binomial distribution is discrete and may be asymmetric.

Worked example 6: construct a two tailed critical region

A coin is claimed to land heads with probability $0.5$. It is tossed $20$ times. Find the critical region for a $5\%$ two tailed test.

Under $H_0$,

$$X\sim\operatorname{B}(20,0.5).$$

Each tail may contain at most $0.025$. For the lower tail,

$$P(X\leq5)=0.0207, \qquad P(X\leq6)=0.0577.$$

Thus the lower part is $X\leq5$.

The distribution is symmetric because $p=0.5$, so

$$P(X\geq15)=P(X\leq5)=0.0207,$$

while $P(X\geq14)=0.0577$. Hence the upper part is $X\geq15$.

The critical region is

$$\boxed{X\leq5\quad\text{or}\quad X\geq15}.$$

The actual significance level is the probability of the whole critical region:

$$\begin{aligned} P(X\leq5)+P(X\geq15) &=0.0207+0.0207\\ &=\boxed{0.0414}. \end{aligned}$$

If $4$ heads were observed, reject $H_0$. There would be sufficient evidence that the probability of heads differs from $0.5$.

Worked example 7: an asymmetric two tailed test

For $X\sim\operatorname{B}(15,0.3)$ under $H_0$, find the critical region for a $10\%$ two tailed test from the following values:

$$\begin{array}{c|cc} \text{Lower tail} & P(X\leq0)=0.00475 & P(X\leq1)=0.0353\\ \hline \text{Upper tail} & P(X\geq8)=0.050013 & P(X\geq9)=0.0152 \end{array}$$

Each tail is allowed at most $0.05$.

For the lower tail, $X\leq1$ is allowed because $0.0353\leq0.05$. For the upper tail, $X\geq8$ is not allowed because $0.050013>0.05$, so the boundary must move to $9$. Therefore

$$\boxed{X\leq1\quad\text{or}\quad X\geq9}.$$

The actual significance level is approximately

$$0.0353+0.0152=\boxed{0.0505}.$$

The tail probabilities are unequal. Trying to force equal probabilities would discard valid outcomes and make the test unnecessarily conservative.

Do not simply double every probability

For a symmetric null distribution, twice a one tail probability can produce a two sided p value. For a general binomial distribution with $p_0\neq0.5$, the two tails are not symmetric. In critical region questions, allocate $\alpha/2$ to each tail and construct both boundaries separately.

Calculator strategy

Calculator menus differ, but the probability identities do not.

For $X\sim\operatorname{B}(n,p)$:

$$P(X\leq x)=\operatorname{binomcdf}(n,p,x),$$

and

$$P(X\geq x)=1-P(X\leq x-1).$$

The shift to $x-1$ matters. For example,

$$P(X\geq12)=1-P(X\leq11),$$

not $1-P(X\leq12)$.

For an isolated value,

$$P(X=x)=\binom nx p^x(1-p)^{n-x}.$$

Keep unrounded calculator values when comparing with $\alpha$. Round only when presenting the probability. A displayed value of $0.0500$ might lie slightly above or below $0.05$ before rounding.

A complete exam method

1. Define $p$ in context.
2. State $H_0$ and $H_1$.
3. Under $H_0$, write $X\sim\operatorname{B}(n,p_0)$.
4. Identify whether the test is upper tailed, lower tailed or two tailed.
5. Calculate the relevant tail probability, or construct the critical region.
6. Compare with the significance level, or check whether the observation lies in the critical region.
7. State “reject $H_0$” or “do not reject $H_0$”.
8. Give a conclusion in the context of the question.

Worked example 8: full test using a critical region

A manufacturer claims that $4\%$ of its batteries are defective. A customer believes the defect probability is higher. In a random sample of $50$ batteries, $5$ are defective. At the $5\%$ significance level, test the customer’s belief.

Let $p$ be the probability that a randomly selected battery is defective.

$$H_0:p=0.04, \qquad H_1:p>0.04.$$

Under $H_0$,

$$X\sim\operatorname{B}(50,0.04).$$

This is upper tailed. The relevant probabilities are

$$P(X\geq5)\approx0.04895$$

and

$$P(X\geq4)\approx0.1391.$$

Therefore the $5\%$ critical region begins at $5$:

$$X\geq5.$$

The observed value $5$ lies in the critical region, so reject $H_0$. There is sufficient evidence at the $5\%$ significance level to support the customer’s belief that the probability of a defective battery is greater than $0.04$.

The result is only just significant. That does not change the decision, but it is useful context when interpreting the strength of evidence.

Assumptions behind the test

The test is valid only if a binomial model is reasonable. Check that:

• there is a fixed number $n$ of trials;
• each trial has two outcomes, labelled success and failure;
• trials are independent;
• the probability $p$ of success is constant across trials.

In sampling without replacement, independence is only approximate. It is generally reasonable when the population is much larger than the sample. Bias in the selection process is not repaired by a hypothesis test.

Worked example 9: challenge the model

A school surveys the first $40$ pupils entering the library and treats “supports the proposal” as success. It then uses a binomial test to make a claim about all pupils.

Even if each response has two outcomes, the sample may be biased because library users might differ systematically from the whole school. The calculation could be arithmetically correct while the conclusion is unreliable.

Statistical inference depends on how the data were obtained, not only on the formula used afterwards. Review sampling methods and probability modelling.

Common misconceptions

”Not significant” means the null hypothesis is true

No. It means the evidence was not strong enough to reject $H_0$ at the chosen level. A larger sample might detect a real difference that this sample did not.

The observed value alone is the probability

For an upper tailed observation $x$, use $P(X\geq x)$, not $P(X=x)$. Outcomes more extreme than $x$ also count as evidence against $H_0$.

The sample proportion replaces $p$ under $H_0$

If $18$ successes occur in $30$ trials, the sample proportion is $18/30=0.6$. But if $H_0:p=0.5$, probabilities for the test are calculated using $p=0.5$, not $0.6$.

The tail follows the data

The research question chooses the tail. If the stated alternative is $p>p_0$, an unexpectedly low result does not justify changing to a lower tailed test after seeing the data.

A $5\%$ critical region must have probability exactly $0.05$

Binomial values are discrete. The actual significance level is the largest available critical probability not exceeding the stated level, and is often smaller.

Mixed self-check

Question 1

A die is suspected of producing sixes more often than a fair die. In $30$ rolls, it produces $9$ sixes.

1. State suitable hypotheses.
2. Write the null distribution.
3. Given that $P(X\geq9)=0.0474$, complete a $5\%$ test.

Answer

Let $p$ be the probability of rolling a six.

$$H_0:p=\frac16, \qquad H_1:p>\frac16.$$

Under $H_0$,

$$X\sim\operatorname{B}\left(30,\frac16\right).$$

Since $0.0474<0.05$, reject $H_0$. There is sufficient evidence at the $5\%$ level to suggest that the die produces sixes with probability greater than $1/6$.

Question 2

For a lower tailed test, $X\sim\operatorname{B}(24,0.75)$ under $H_0$. The following probabilities are given:

$$P(X\leq13)=0.0187, \qquad P(X\leq14)=0.0551.$$

Find the $5\%$ critical region. State the decision if $14$ successes are observed.

Answer

The critical region is

$$\boxed{X\leq13},$$

with actual significance level $0.0187$. Since $14$ is outside the critical region, do not reject $H_0$.

Question 3

Explain why a two tailed $5\%$ test does not usually place $5\%$ in each tail.

Answer

Placing $5\%$ in each tail would give a total intended significance level of up to $10\%$. The overall $5\%$ allowance is shared, normally as at most $2.5\%$ in each tail.

Question 4

A test produces a p value of $0.008$. State the decisions at the $5\%$ and $1\%$ significance levels.

Answer

Since

$$0.008<0.01<0.05,$$

reject $H_0$ at both levels. The result is significant at both $5\%$ and $1\%$.

Summary

For $X\sim\operatorname{B}(n,p_0)$ under $H_0$:

$$\begin{array}{c|c|c} H_1 & \text{Evidence lies in} & \text{Typical critical region}\\ \hline p>p_0 & \text{upper tail} & X\geq c\\ p<p_0 & \text{lower tail} & X\leq c\\ p\neq p_0 & \text{both tails} & X\leq a\text{ or }X\geq b \end{array}$$

Reject $H_0$ when the relevant probability is at most $\alpha$, or when the observation lies in the critical region. Always finish with a cautious conclusion in context.

Next, compare this exact discrete procedure with normal hypothesis tests and apply the same decision language in correlation hypothesis tests.