Hypothesis testing, Z-test & t-test

Hypothesis testing is a formal procedure for making decisions about population parameters based on sample data, and is one of the most common tools available to engineers and scientists for making data-driven decisions. It provides a rigorous, systematic framework for drawing conclusions from sample data, allowing us to move beyond intuition and subjective judgment to conclusions supported by mathematical evidence.

Specifically, a hypothesis test is a procedure that answers the question: “Did these data come from a particular type of distribution?” or “Is there a significant difference between groups?” The most common hypothesis test determines whether a sample mean could have come from a population with a specified mean.

Components of hypothesis testing

Definitions

• Null Hypothesis ($H_0$): The claim to be tested (usually a statement of “no effect” or “no difference”).
• Alternative Hypothesis ($H_a$ or $H_1$): The claim we want to establish
• Test Statistic: A value calculated from sample data
• Significance Level ($\alpha$): The probability of rejecting $H_0$ when it is true (Type I error)
• $P$-value: The probability of obtaining results as extreme as observed, assuming $H_0$ is true

Every hypothesis test involves two competing hypotheses:

Null hypothesis ($H_0$): The statement being tested, typically representing “no effect” or “no difference.” For example: $H_0:\mu ={\mu }_0$, i.e. the mean of a distribution is the mean of the sample.

Alternative hypothesis ($H_1$ or $H_a$): The statement we conclude if we reject the null hypothesis. It can be:

• Two-tailed: $H_1:\mu \ne {\mu }_0$
• Left-tailed: $H_1:\mu <{\mu }_0$
• Right-tailed: $H_1:\mu >{\mu }_0$

The decision rule for hypothesis testing is:

• If $p$-value $\le \alpha$: Reject $H_0$
• If $p$-value $>\alpha$: Retain $H_0$

Types of errors

There are four possible outcomes, which are the combinations of whether $H_0$ is true or note and whether $H_0$ was retained or rejected, where the two “incorrect” outcomes are considered “errors” of two types, where type I is a false positive and type II is a false negative:

	$H_0$ True	$H_0$ False
Reject $H_0$	Type I Error ($\alpha$)	Correct Decision (Power = $1-\beta$)
Retain $H_0$	Correct Decision	Type II Error ($\beta$)

Type I error (false positive): Rejecting $H_0$ when it is actually true. The probability of a Type II error is $\alpha$.

Type II error (false negative): Failing to reject $H_0$ when it is actually false. The probability of a Type II error is $\beta$.

The power of a test is the probability of correctly rejecting a false null hypothesis: Power = $1-\beta$.

The hypothesis testing procedure

Step 1: State the null and alternative hypotheses.

Step 2: Choose the significance level α (commonly 0.05 or 0.01).

Step 3: Determine whether to use a one-tail or two-tail test:

• If testing whether the parameter has changed (in either direction), use a two-tail test
• If testing whether the parameter has increased or decreased (one direction only), use a one-tail test

Step 4: Find the critical value(s) from the appropriate table ($z$-table or $t$-table) based on α and the number of tails.

Step 5: Calculate the test statistic from the sample data.

Step 6: Make a decision:

• If |test statistic| ≥ critical value, reject H₀
• If |test statistic| < critical value, fail to reject H₀

Confidence levels

The results of experiments are seldom correct 100% of the time. Recognizing this fundamental reality, researchers accept a certain probability of being wrong. In order to minimize this probability, an experiment is repeated several times. The number of repetitions required depends on the level of confidence wanted in the results.

For example, if the results have a 5% probability of being wrong, the confidence level, denoted C, is 95% that the results are correct. Common confidence levels used in practice include 90%, 95%, and 99%. The choice of confidence level depends on the consequences of making an incorrect decision and the practical constraints of the situation. Higher confidence levels require more data or result in wider confidence intervals.

The significance level (denoted $\alpha$) represents the probability of making an error—specifically, the probability of rejecting a true hypothesis. The relationship between confidence level and significance level is:

$$𝛼=1-C$$

For a 95% confidence level, α = 0.05, meaning there is a 5% chance of incorrectly rejecting a true hypothesis.

The $z$-test

The z-test is used when:

• The population standard deviation σ is known, OR
• The sample size is large (typical rule of thumb: $n\ge 30$), allowing the sample standard deviation to approximate $\sigma$.

The $z$-test statistic

The test statistic for a $z$-test on a population mean is:

$$z=\frac{\bar{X}-{\mu }_o}{\sigma /\sqrt{n}}$$

where:

• $\bar{X}$ = sample mean
• ${\mu }_o$ = hypothesized population mean (from $H_0$)
• $\sigma$ = population standard deviation
• $n$ = sample size

Critical values for the $z$-test

Common critical $z$-values are:

Confidence Level	α	Two-Tail z	One-Tail z
90%	0.10	±1.645	1.282
95%	0.05	±1.960	1.645
99%	0.01	±2.576	2.326

The $t$-distribution and the $t$-test

The $t$-distribution (also called Student’s t-distribution) was developed by William Sealy Gosset, who published under the pseudonym “Student” in 1908. This distribution is fundamental to statistical inference when working with small samples or when the population standard deviation is unknown.

The $t$-distribution is similar to the standard normal distribution (“bell curve”) but has heavier tails, meaning it assigns more probability to extreme values. This accounts for the additional uncertainty introduced when estimating the population standard deviation from sample data.

Degrees of freedom

The shape of the $t$-distribution depends on a parameter called degrees of freedom (df), which is typically equal to $n-1$, where $n$ is the sample size. As the degrees of freedom increase, the $t$-distribution approaches the standard normal distribution. For practical purposes, when df > 30, the t-distribution is nearly identical to the normal distribution.

The exceedance probability

The exceedance (i.e., the probability of being incorrect), $\alpha$, is equal to the total area under the tail(s) of the distribution beyond the critical value(s).

• For a one-tail test: $\alpha =1-C$
• For a two-tail test: $\alpha /2$ is placed in each tail, so the total exceedance is $\alpha =1-C$

When to Use the $t$-test

The t-test is used when:

• The population standard deviation σ is unknown, AND
• The sample size is small ($n$ < 30)

The $t$-test statistic

The test statistic for a t-test on a population mean is:

$$t=\frac{\bar{X}-{\mu }_o}{s/\sqrt{n}}$$

where:

• $\bar{X}$ = sample mean
• ${\mu }_o$ = hypothesized population mean (from $H_0$)
• $s$ = sample standard deviation
• $n$ = sample size

$t$-test for comparing two means

When comparing the means of two independent samples (“1” and “2”), the t-test statistic is:

$$t=\frac{(\overline{X_1}-\overline{X_2})-({\mu }_1-{\mu }_2)}{\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}}$$

where:

• $\overline{X_i}$ = sample mean for sample $i$
• ${\mu }_i$ = hypothesized population mean for $i$
• $s_i$ = sample standard deviation for $i$
• $n_i$ = sample size for $i$

Choosing between the $z$-test and the $t$-test

Use the following guidelines to select the appropriate test:

1. Is σ known?

• Yes → Use z-test
• No → Continue to step 2

2. Is n ≥ 30?

• Yes → Use z-test (with s approximating σ)
• No → Use t-test

Summary comparison

Feature	Z-Test	T-Test
Population σ	Known	Unknown
Sample size	Any (typically large)	Any (typically small)
Distribution	Standard normal	Student’s t
Critical values	Fixed for given α	Depend on df
As n → ∞	—	Approaches z-test

Conclusion

Hypothesis testing provides a rigorous framework for making decisions based on sample data. The key concepts covered in this section include:

• Confidence levels quantify our certainty in statistical conclusions
• Confidence intervals provide ranges likely to contain population parameters
• Hypothesis tests follow a systematic six-step procedure
• The $z$-test is used when $\sigma$ is known or $n$ is large (more than 30)
• The $t$-test is used when $\sigma$ is unknown and $n$ is small (fewer than 30)
• Type I and Type II errors represent the two ways a hypothesis test can lead to incorrect conclusions

Understanding when to apply each test and how to interpret the results is essential for engineers and scientists who must make data-driven decisions under uncertainty.