Experimental vs observational studies

Observational studies and experiments

Definitions

Observational study

Researchers observe and measure variables without imposing any treatments.

Experiment

Researchers actively impose treatments on subjects to measure their effects.

Treatment

A specific condition applied to experimental units.

Control group

A group that receives either no treatment, a placebo, or the standard treatment for comparison.

Placebo effect

Improvement caused by expectations rather than the treatment itself. A placebo helps researchers separate these effects from the treatment’s true impact.

Random assignment

Using chance to assign experimental units to treatment groups.

Observational studies

Observational studies involve observing and measuring data without influencing it.

Example:

Suppose a researcher wants to know whether there’s an association between smoking and life expectancy. An observational study makes sense for a few reasons:

• It would be unethical to randomly assign people to smoke or not smoke.
• Researchers can use existing records (for example, data from people who have already lived their lives) without disturbing the population being studied.
• Using existing data is often faster and less expensive than running a long-term experiment.

Experimental studies

Experimental studies involve the researcher actively manipulating one or more variables to investigate cause-and-effect relationships. Participants are typically divided into treatment groups and a control group so results can be compared.

Example:

Suppose a researcher wants to know whether a new exam preparation course (still in beta testing) increases students’ exam scores. The researcher could:

• assign one group of students to take the course (treatment group)
• assign another group not to take the course (control group)

Then the researcher compares exam results between the groups. In this situation, there may be no previous data available because the course is new, so an observational study may not be possible until the course is widely used.

Variables

Understanding the different types of variables helps you interpret what a study is actually measuring — and what might be getting in the way.

Definitions

Explanatory variable (independent variable)

The variable that is categorized to see how it affects the response variable.

Response variable (dependent variable)

The variable that is being measured and is dependent on the explanatory variable.

Confounding variable

A variable associated with the explanatory variable that affects the response variable.

Here’s how these ideas fit together in a common situation.

Example:

Suppose a researcher wants to know whether more hours of sleep leads to a higher GPA for post-secondary students.

• The explanatory variable is the number of hours of sleep per night.
• The response variable is GPA.
• A possible confounding variable is a parent’s income level.

Income could be a confounder because it is associated with sleep patterns and may also influence GPA:

• Students from lower-income families may need to work more hours, which can reduce sleep.
• Those same students may have fewer academic resources (such as tutoring), which can affect GPA.

Because income is associated with sleep and can influence GPA, it can make the relationship between sleep and GPA harder to interpret.

Additional concepts in experiments

• Placebo effect: Participants may respond differently simply because they believe they are receiving a treatment. A placebo helps researchers determine whether observed effects are due to the treatment itself rather than participants’ expectations.

Experimental design

Once you’ve identified your variables, the next step is designing the experiment itself. A well-designed experiment controls for factors that could distort results, including whether participants know what treatment they’re receiving.

Definitions

Blinding

The participants don’t know which treatment group they are a part of or if they are part of the control group.

Double-blinding

Neither the subjects nor the response evaluators know which treatment group or control group each participant is a part of.

Matched pairs design (also known as paired comparison design)

Two treatments are compared based on the responses of paired subjects, one of whom receives one treatment while the other receives another treatment.

Blocking

An experimental design where subjects are divided into representative groups called blocks. Subjects within blocks are randomly assigned different treatments.

Example 1:

$60$ people who have been diagnosed with the flu sign up to test a new flu medication. $20$ of them are given a full dose of the new medication, $20$ are given a half-dose, and the remaining $20$ are given a sugar pill that contains no medication at all.

Blinding occurs if the participants do not know whether they received the full dose, the half dose, or the sugar pill.

Double-blinding occurs if neither the participants nor the response evaluators know this information.

Example 2:

A teacher wants to know how effective a new studying tool is, so she divides the class into blocks based on grade point average. There are $10$ students with a GPA between $50$–$59\%$, $10$ with a GPA between $60$–$69\%$, $10$ with a GPA between $70$–$79\%$, $10$ with a GPA between $80$–$89\%$, and $10$ with a GPA between $90$–$100\%$. $5$ students from each block are given access to the new study tool and the other $5$ are not.

The purpose of blocking here is to account for previous academic history. In general, students who have previously scored higher or lower on tests are likely to continue to do so. So:

• If a student who usually scores around $85\%$ scores around $85\%$ after using the new studying tool, that doesn’t suggest the tool improved their performance.
• If a student who normally scores around $70\%$ scores $85\%$ after using the tool, that suggests a larger improvement.

Blocking helps ensure that students at different starting levels are represented in both the treatment and control conditions, making the comparison fairer.

Practice problem

Example:

The supervisor at a community swimming pool plans to introduce a new teaching method that she would like the swimming instructors to use moving forward. The supervisor is interested to know whether or not this new method will positively impact the students’ times for all of the different strokes within an $8$-week timeframe where students would participate in a coaching program before they are given a report card with their times during testing day.

Give an example of how the supervisor could conduct this study.

Solution:

Alternative solution using blocking: