Introduction to significance testing

Significance testing

Definitions

Significance testing

Also called hypothesis testing, this is a structured method for deciding whether results from a sample are likely due to natural random variation or whether they provide strong evidence for a claim about a population.

Null hypothesis ($H_0$)

The specific claim being tested. It usually represents “no effect” or “no difference.”

Alternative hypothesis ($H_a$)

The claim we suspect might be true instead of $H_0$.

For example, suppose you sample some students from high school A and find that their average academic score is much higher than the overall average for high school students. That difference could happen for two reasons:

• Students at high school A really do perform better on average than students at other schools.
• By chance, the particular students sampled from high school A happened to be unusually high-scoring.

Significance testing gives a structured way to judge whether the sample result is unusual enough (assuming the population claim is true) that we should doubt the claim.

Procedure for hypothesis testing:

1. Choose a specific hypothesis to be tested (this is called the null hypothesis and is denoted by $H_0$).
2. Take an appropriate random sample from the population.
3. Use measurements from the sample to determine how likely the null hypothesis is to be correct given the results of the sampling.

Significance level

Definitions

Significance level ($\alpha$)

The threshold chosen at the start of the test that sets how much evidence is required to reject $H_0$. It is also the probability of making a type I error.

Common choices are:

• $\alpha =0.05$ (a common default)
• $\alpha =0.01$ (used when you want stronger evidence before rejecting $H_0$)
• $\alpha =0.1$ (used when you’re willing to accept more risk of a false alarm)

To see what $\alpha$ means, consider a company testing a new drug.

• $H_0$: the drug has no effect
• $H_a$ (alternative hypothesis): the drug has an effect

If you use a significance level of $0.05$, you are accepting a $5\%$ chance of rejecting $H_0$ even when $H_0$ is actually true. In context, that would mean concluding the drug has an effect when it really doesn’t. This is called a type I error.

Type I and type II errors

Definitions

Type I error

A type I error occurs when the null hypothesis is rejected even though it is true. The significance level $\alpha$ is the probability of making a type I error.

Type II error

A type II error occurs when we fail to reject the null hypothesis even though it is false. The probability of a type II error is denoted by $\beta$.

There’s an important trade-off: choosing a smaller $\alpha$ reduces the chance of a type I error, but it increases the chance of missing a real effect. Missing a real effect is a type II error.

In the drug example, a type II error would mean the drug really does have an effect, but the test fails to reject $H_0$ (which says the drug has no effect).

Power of a test

Definitions

Power of a test

The probability that the test correctly rejects $H_0$ when $H_0$ is false. In other words, it is the probability of detecting a real effect when there is one.

$$\begin{array}{rl}\text{Power}& =1-\beta \end{array}$$

Two common ways to increase power are:

• Increase the sample size.
• Increase the significance level.

Keep in mind that increasing the significance level increases power, but it also increases the probability of a type I error. So you’re balancing two kinds of mistakes.

Practice problem

Example:

McDonald’s claims that during the USA McDonald’s monopoly game, $1$ in $5$ game pieces are food prizes. Denia and her family go to McDonald’s and collect $100$ game pieces and $13$ of them are food prizes. Based on the $1$ in $5$ claim, the expected number of food prizes for Denia and her family would have been $20$.

Using a significance level of $\alpha =0.05$, conduct a significance test to determine whether the observed number of $13$ out of $100$ food prizes provides significant evidence against the restaurant’s claim of $1$ in $5$.

State the hypotheses:

$$\begin{array}{rl}{H}_0& :p=0.2\\ H_a& :p<0.2\end{array}$$

From the problem:

$$\begin{array}{rl}\hat{p}& =\frac{13}{100}\\ & =0.13\end{array}$$

$$\begin{array}{rl}{p}_0& =0.2\end{array}$$

Check conditions:

• Random: The random sample condition is met because Denia and her family collected game pieces that were randomly chosen rather than chosen specifically for her family.
• Independence: The independence condition is also met because $100$ game pieces is most certainly less than $10\%$ of all game pieces produced by such a large corporation.
• Normality: Both large-counts values are at least $10$:

$$\begin{array}{rl}n{p}_0& =100(0.20)\\ & =20\ge 10\end{array}$$

$$\begin{array}{rl}n(1-p_0)& =100(0.80)\\ & =80\ge 10\end{array}$$

Find the standard error:

$$\begin{array}{rl}\text{SE}& =\sqrt{\frac{p_0(1-p_0)}{n}}\\ & =\sqrt{\frac{0.2(0.8)}{100}}\\ & =\sqrt{\frac{0.16}{100}}\\ & =0.04\end{array}$$

Find the z-score:

$$\begin{array}{rl}z& =\frac{\hat{p}-p_0}{\text{SE}}\\ & =\frac{0.13-0.2}{0.04}\\ & \approx \frac{-0.07}{0.04}\\ & \approx -1.75\end{array}$$

Find the p-value using the standard normal distribution:

$$\begin{array}{rl}P(z<-1.75)& \approx 0.0401\end{array}$$

This means that there is a $4.01\%$ chance of getting $13$ or fewer food prizes out of $100$ by random chance if the McDonald’s claim about $1$ in $5$ of their tickets being food prizes is correct.

Compare the p-value to $\alpha$:

Solution: