Sampling distribution simulations

Definitions

Sampling distribution

the probability distribution of a statistic computed from many repeated samples drawn from the same population.

When we collect data, we usually only see one sample — but different samples from the same population can give different results. To understand how sample statistics behave across many samples, we study their sampling distribution.

Simulating sample results

One of the clearest ways to see how a statistic varies from sample to sample is to simulate the process many times. The steps below outline how to carry out a simulation of a sampling distribution.

How to simulate a sample distribution:

1. Identify the population and the statistic: Specify the population and its key characteristics (such as mean and standard deviation, or median and IQR). Then choose the sample statistic you want to study (for example, the sample mean, sample proportion, etc.).
2. Select a sample size $n$: Choose the sample size you’ll draw from the population in each repetition of the simulation.
3. Repeat sampling many times ($N$ times): Randomly select a sample of size $n$ from the population and calculate the chosen statistic for that sample. Repeat this process $N$ times, recording the statistic each time.
4. Plot the distribution: Optionally, plot the results to get a clear visual of the simulated sampling distribution.
5. Analyze the distribution: Describe the shape, center, and spread of the simulated sampling distribution, and compare what you see to the theoretical expectations from Step 1. Choose a measure of center that matches the shape (mean for roughly symmetric distributions, median for skewed distributions). For spread, use standard deviation with the mean and IQR with the median.

Sidenote

Important to know!

One of the most common uses of sampling distribution simulations is to illustrate the central limit theorem by simulating the sampling distribution of the sample mean.

Practice problem

The following multi-part example walks through how choices like the sample size $n$ and the number of repetitions $N$ shape what a simulated sampling distribution actually looks like.

Example:

Consider a population that follows a uniform distribution between $0$ and $10$.

Part a.

Find the population mean and the population standard deviation.

Recall that the equation for the population mean of a uniform distribution is $\mu =\frac{a+b}{2}$ where $a$ and $b$ are the minimum and maximum values respectively.

Since $a=0$ and $b=10$,

$$\begin{array}{rl}\mu & =\frac{a+b}{2}\\ & =\frac{0+10}{2}\\ & =\frac{10}{2}\\ & =5\end{array}$$

Recall that the equation for the standard deviation of a uniform distribution is $\sigma =\sqrt{\frac{(b-a{)}^2}{12}}$ where $a$ and $b$ are the minimum and maximum values respectively.

$$\begin{array}{rl}\sigma & =\sqrt{\frac{(10-0{)}^2}{12}}\\ & =\sqrt{\frac{100}{12}}\\ & \approx 2.887\end{array}$$

Solution:

Now that we know the population’s true mean and standard deviation, we can think about how simulation choices affect how closely our results will reflect those true values.

Part b.

Suppose that James and Mark are both going to do a simulation. Both are using a sample size $n$ but James is doing the simulation $N=10$ times while Mark is doing the simulation $N=1000$ times. Who’s simulation is going to produce more accurate results and why?

Each repetition produces a sample statistic (such as a sample mean), and those statistics naturally vary from sample to sample. When you repeat the simulation more times, the overall simulated distribution becomes smoother and the average of the simulated statistics tends to settle closer to the population value.

At some point, increasing $N$ further gives diminishing returns:

Case	Value of $N$	Effect
Larger $N$	$1000\to 10000$	Little change in results
Small $N$	$10$	Less reliable output

Solution:

The next two parts shift focus from the number of repetitions $N$ to the sample size $n$, and ask what shape we should expect the simulated distribution to take.

Part c.

If a simulation of at least $N=1000$ and $n=30$ is performed, what would you expect the shape of the distribution to be?

Using a large number of repetitions (such as $N=1000$) won’t change the underlying theoretical shape, but it will make the simulated histogram look more clearly normal and less “bumpy” due to random simulation noise.

Solution:

Part d.

If a simulation of at least $N=1000$ is performed, but $n=3$, what would you expect the shape of the distribution to be?

For comparison, if $n=1$, the “sample mean” is just a single observation from the population, so the distribution would look uniform. With $n=3$, averaging pulls values toward the center, so the distribution becomes more concentrated around $5$, but not enough to be approximately normal.

Solution: