7.6 Hypothesis testing errors

Achievable AP Statistics

7. Quantitative data

Our AP Statistics course is currently in development and is a work-in-progress.

Hypothesis testing errors

4 min read

Font

Discuss

Feedback

Type I and Type II errors

In this section, you’ll look more closely at Type I and Type II errors, along with the real-world consequences of each. To start, here are the definitions and a concrete example of each.

Definitions

Type I error (false positive): Occurs when we incorrectly reject a true null hypothesis. In other words, we conclude there is an effect or difference when there actually isn’t one.

For example, if $H_{0}$ is that a drug has no effect, and in reality $H_{0}$ is true (the drug has no effect), but we conclude the drug is effective, then a Type I error has occurred. A likely consequence is that time and money get spent on a drug that doesn’t work.

Type II error (false negative): Occurs when we fail to reject a false null hypothesis. In other words, we conclude there is no significant difference when there actually is one.

For example, if $H_{0}$ is that a drug has no effect, and in reality the drug is effective, but our study concludes the drug has no effect, then a Type II error has occurred. A likely consequence is that a beneficial drug gets abandoned and never reaches the patients who could have used it.

Ideally, we want to avoid both errors:

A Type I error can lead to wasted resources on something ineffective.
A Type II error can lead to cutting support for something that actually helps.

Connecting errors to hypothesis-testing quantities

Now connect these ideas to the usual hypothesis-testing quantities:

Definitions

Significance level ( $α$ ): The probability of rejecting the null hypothesis when it is true. In other words, it is the probability of making a Type I error.
Power ( $1 - β$ ): The probability of correctly rejecting the null hypothesis when it is false. In other words, it is the probability of detecting a real effect.
$β$: The probability of making a Type II error.

The trade-off between $α$ and $β$ :

If you decrease $α$ , making it harder to reject $H_{0}$ , you reduce the chance of a false positive (Type I error) but increase the chance of not detecting an actual difference (Type II error).
If you increase $α$ , making it easier to reject $H_{0}$ , you increase the chance of a false positive (Type I error) but decrease the chance of not detecting an actual difference (Type II error).

There is a way to reduce both of these simultaneously: increase the sample size, which gives you more information. The downside is extra time and expense. Increasing the sample size can help a lot at first, but beyond a point you’ll see diminishing returns.

Improving power: blocking and stratification

Both blocking and stratification can improve power.

Definitions

Blocking: A technique used in experimental design to control for variables you aren’t directly studying but that can still affect the outcome. The idea is to compare like with like, so the main comparison is cleaner. Blocking helps reduce standard error by limiting the influence of these other variables.
Stratification: A technique used in sample selection and survey design to make sure the sample represents the whole population. You divide the population into distinct, non-overlapping groups called strata based on a chosen characteristic (for example, age, gender, race, or income bracket), and then sample from each stratum. Stratification helps reduce sampling error and improves representativeness.

Example of blocking:

Suppose the head lifeguard at a pool wants to know which teaching method is most effective for competitive swimmers. Blocking by age and sex is crucial. It wouldn’t be fair to compare the times of $6$ -year-old swimmers to those of $10$ -year-olds. Regardless of teaching method, you’d expect $6$ -year-olds to have slower times on average than $10$ -year-olds. If you ignore age, you could unfairly penalize instructors who were assigned younger swimmers. Similarly, if average swim times differ by sex, failing to account for sex could distort the comparison.

Example of stratification:

If you want to measure public opinion about a new property tax increase, it’s important to include people across income levels. Stratification across income brackets helps ensure the sample reflects the whole population.

Type I and Type II errors

Type I error: false positive; reject true null hypothesis
Type II error: false negative; fail to reject false null hypothesis
Real-world consequences:
- Type I: waste resources on ineffective interventions
- Type II: miss out on beneficial interventions

Hypothesis testing quantities

Significance level $α$ : probability of Type I error
$β$ : probability of Type II error
Power ( $1 - β$ ): probability of correctly detecting a real effect

Trade-offs and sample size

Decreasing $α$ : lowers Type I error, raises Type II error
Increasing $α$ : raises Type I error, lowers Type II error
Increasing sample size $n$ : reduces both Type I and Type II errors, but with diminishing returns

Blocking and stratification

Blocking: controls for confounding variables in experiments; reduces standard error
Stratification: ensures sample represents population; reduces sampling error

Key takeaways

Type I error = false positive; Type II error = false negative
$α$ = probability of Type I error; $β$ = probability of Type II error
Power ( $1 - β$ ) = probability of avoiding Type II error
Smaller $α$ : higher Type II error risk, lower Type I error risk
Larger $n$ : reduces standard deviations, lowers both error types
Power increases if (all else equal):
- $α$ increases
- Standard error decreases
- Sample size $n$ increases
- True parameter farther from null hypothesis

Next | 7.7 Paired data

Hypothesis testing errors

Type I and Type II errors

In this section, you’ll look more closely at Type I and Type II errors, along with the real-world consequences of each. To start, here are the definitions and a concrete example of each.

Definitions

Type I error (false positive): Occurs when we incorrectly reject a true null hypothesis. In other words, we conclude there is an effect or difference when there actually isn’t one.

Type II error (false negative): Occurs when we fail to reject a false null hypothesis. In other words, we conclude there is no significant difference when there actually is one.

Ideally, we want to avoid both errors:

A Type I error can lead to wasted resources on something ineffective.
A Type II error can lead to cutting support for something that actually helps.

Connecting errors to hypothesis-testing quantities

Now connect these ideas to the usual hypothesis-testing quantities:

Definitions

Significance level ( $α$ ): The probability of rejecting the null hypothesis when it is true. In other words, it is the probability of making a Type I error.
Power ( $1 - β$ ): The probability of correctly rejecting the null hypothesis when it is false. In other words, it is the probability of detecting a real effect.
$β$: The probability of making a Type II error.

The trade-off between $α$ and $β$ :

If you decrease $α$ , making it harder to reject $H_{0}$ , you reduce the chance of a false positive (Type I error) but increase the chance of not detecting an actual difference (Type II error).
If you increase $α$ , making it easier to reject $H_{0}$ , you increase the chance of a false positive (Type I error) but decrease the chance of not detecting an actual difference (Type II error).

Improving power: blocking and stratification

Both blocking and stratification can improve power.

Definitions

Blocking: A technique used in experimental design to control for variables you aren’t directly studying but that can still affect the outcome. The idea is to compare like with like, so the main comparison is cleaner. Blocking helps reduce standard error by limiting the influence of these other variables.
Stratification: A technique used in sample selection and survey design to make sure the sample represents the whole population. You divide the population into distinct, non-overlapping groups called strata based on a chosen characteristic (for example, age, gender, race, or income bracket), and then sample from each stratum. Stratification helps reduce sampling error and improves representativeness.

Example of blocking:

Suppose the head lifeguard at a pool wants to know which teaching method is most effective for competitive swimmers. Blocking by age and sex is crucial. It wouldn’t be fair to compare the times of $6$ -year-old swimmers to those of $10$ -year-olds. Regardless of teaching method, you’d expect $6$ -year-olds to have slower times on average than $10$ -year-olds. If you ignore age, you could unfairly penalize instructors who were assigned younger swimmers. Similarly, if average swim times differ by sex, failing to account for sex could distort the comparison.

Example of stratification:

If you want to measure public opinion about a new property tax increase, it’s important to include people across income levels. Stratification across income brackets helps ensure the sample reflects the whole population.

Achievable AP Statistics

7. Quantitative data

Our AP Statistics course is currently in development and is a work-in-progress.