Biased and unbiased estimators

When using a sample statistic to estimate a population parameter, it matters whether that statistic tends to hit the target on average. An estimator that consistently overshoots or undershoots is called biased, while one that averages out to the true parameter is called unbiased.

Definitions

Biased estimator (in the context of sampling distributions)

A statistic whose expected value is not equal to the true population parameter.

• For example, the sample variance computed as $\frac{1}{n}\sum (x_i-\bar{x}{)}^2$ is biased because it tends to underestimate the true population variance ${\sigma }^2$.

Unbiased estimator

A statistic whose expected value equals the true population parameter.

• For example, $\bar{x}$ estimates $\mu$, and $\hat{p}$ estimates $p$. More generally, an estimator $\hat{\theta }$ of a parameter $\theta$ is unbiased if

$$E[\hat{\theta }]=\theta$$

Sample mean as an estimator of the population mean

Let $X_1,X_2,X_3,\dots ,X_n$ be a random sample from a population with mean $\mu$. The sample mean is defined as:

$$\bar{X}=\frac{1}{n}\sum _{i=1}^n{X}_i$$

To check whether $\bar{X}$ is unbiased for $\mu$, we compute its expected value:

$$\begin{array}{rl}E[\bar{X}]& =E\left[\frac{1}{n}\sum _{i=1}^n{X}_i\right]\\ & =\frac{1}{n}\sum _{i=1}^{n}E(X_i)\\ & =\frac{1}{n}\sum _{i=1}^n\mu \\ & =\frac{1}{n}(n\mu )\\ & =\mu \end{array}$$

Solution:

Practice problems

Example 1:

Suppose we have a deck of $3$ cards with values $\{2,4,6\}$ and that random samples of size $n=2$ are taken with replacement. Find the population mean and the expected value of the sample mean in order to determine if the sample mean is a biased or an unbiased estimator of the population mean.

First, find the population mean:

$$\begin{array}{rl}\mu =E(X)& =\frac{2+4+6}{3}\\ & =\frac{12}{3}\\ & =4\end{array}$$

Next, list all equally likely samples of size $2$ (with replacement) and compute each sample mean:

$$\begin{array}{rl}(2,2)& \to \bar{x}=\frac{2+2}{2}=2\\ (2,4)& \to \bar{x}=\frac{2+4}{2}=3\\ (2,6)& \to \bar{x}=\frac{2+6}{2}=4\\ (4,2)& \to \bar{x}=\frac{4+2}{2}=3\\ (4,4)& \to \bar{x}=\frac{4+4}{2}=4\\ (4,6)& \to \bar{x}=\frac{4+6}{2}=5\\ (6,2)& \to \bar{x}=\frac{6+2}{2}=4\\ (6,4)& \to \bar{x}=\frac{6+4}{2}=5\\ (6,6)& \to \bar{x}=\frac{6+6}{2}=6\end{array}$$

There are $9$ equally likely cases, so each has probability $\frac{1}{9}$:

$$\begin{array}{rl}E[\bar{X}]& =\frac{1}{9}(2+3+4+3+4+5+4+5+6)\\ & =\frac{1}{9}(36)\\ & =4\end{array}$$

Solution:

Example 2:

The mayor of a city wants to estimate the average standardized test scores of high school students in the city. The city has three high schools: one public school, one private school, and one charter school. He randomly selects $100$ students from the private school and uses that sample mean to estimate the citywide average. Is this sample mean likely to be an unbiased or biased estimator of the population mean, and why? If biased, suggest at least one way the mayor could improve his approach.

The sample is drawn only from the private school, while the population includes students from all three schools. Students at the public and charter schools had no chance of being selected — a violation of the random sampling requirement needed for an unbiased estimator.

If private school students tend to score higher than students at other schools (due to differences in resources, tutoring access, or socioeconomic factors), then this sample will systematically overestimate the citywide mean.

To reduce bias, the mayor could:

• Take a simple random sample from the entire population of high school students, so every student has a chance of being selected.
• Use a stratified sample by drawing from each school in proportion to its enrollment, so the sample structure mirrors the population.

If the mayor has the resources, he could conduct a census of all students. In that case, the calculated mean is the true population parameter $\mu$ rather than an estimate, so the concept of bias does not apply.

Solution: