Introduction to collecting data
There are several ways to collect data, and each method comes with trade-offs. The goal of data collection is to answer a statistical question — a question that expects variability in the answers.
Examples:
- A school board might want to learn about the quality of staff in its schools.
- A store manager might want to know which advertising methods are most effective at encouraging customers to spend money.
In theory, the best approach would be to survey the entire population and get a response from everyone. In practice, that’s rarely possible because it’s expensive, time-consuming, and there’s usually no way to require participation or guarantee honest answers. So instead, you choose a sampling method and try to make sure the people selected (the sample) represent the larger group you care about (the population).
Example:
If a study is conducted to find out about the political views of American adults, then the population is all American adults. If of those adults are surveyed, then those people are the sample.
Sampling methods
Sampling methods fall into two broad categories: biased and unbiased. Biased methods don’t give everyone in the population an equal chance of being selected, which can skew results and make the sample unrepresentative.
Example 1:
A school asks students what they think of spending an extra of the annual budget on the sports program. Students who feel strongly — either that sports are underfunded or that sports are a waste of resources — are more likely to respond. Students in the middle are less likely to participate. This is an example of voluntary response sampling.
Example 2:
A grocery store clerk wants to find out about the religious beliefs of the state, so he asks the first people who enter the grocery store one Monday morning. People who happen to live nearby and don’t have work that morning do not necessarily represent the full population of the state. This is an example of convenience sampling.
Unbiased methods give every member of the population a fair and equal chance of being selected. These are generally preferred over biased methods.
Example 1:
A principal wants to find out what students think about the atmosphere at their school. She puts all student names into a computer program that randomly selects students. This is simple random sampling.
Example 2:
The same principal obtains a full list of students, randomly selects a starting point using a randomizing program, then selects every th student from that point forward. This is systematic sampling.
Example 3:
A principal wants to know what seniors thought about the quality of their education. He randomly visits senior homerooms and surveys every student in each one. This is cluster sampling.
Example 4:
A news outlet wants to learn about the political beliefs of people in Indiana. The surveyor divides people into different income brackets and randomly selects participants from each income level to ensure all groups are represented. This is stratified sampling.
Choosing the right method
As a general rule, avoid voluntary response sampling and convenience sampling because they often produce biased samples.
A good choice depends on:
- Feasibility: Can you realistically carry out the method in this situation?
- Budget: Can you afford the time, labor, and tools needed?
- Representativeness: Does the method give all members of the population a fair chance to be included?
A few important details help you choose well:
- Cluster sampling works best when clusters are heterogeneous. For example, surveying everyone in a homeroom works well only if students were assigned to homerooms randomly. If some homerooms contain only advanced students, a few homerooms may not represent the whole student population.
- Make sure key subgroups appear in the sample. For example, if a survey asks about Americans’ views on the cost of living, it matters that different socioeconomic groups are represented. People with higher incomes may be less likely to see cost of living as a problem. In that situation, stratified sampling (done properly) is often a strong choice because it ensures representation from each income bracket.
Bias not relating to sampling
Choosing a good sampling method is a major step toward getting representative data — but bias can still appear after sampling, during the data-collection process itself. Here are two common sources and practical ways to reduce them.
For participants who are unable to respond, the key is to identify the barrier and adjust. For example, if a survey is normally done on paper and a participant is blind, providing a braille version can help. If a survey is given over the phone and a participant is deaf, offering a text-based option can address that barrier.
While you can’t always prevent refusals, you can often reduce them by understanding why people hesitate. If some elderly participants don’t feel comfortable answering online, offering a phone call can help. If participants are concerned about privacy, clearly explaining that responses will be kept confidential may increase participation.
To minimize response bias:
- Use simple, direct language rather than figurative language so that people with different literacy levels or those taking the survey in a second language can understand what’s being asked.
- Make multiple-choice options clear and non-overlapping. For example, if you ask for an age group, having both – and – makes it unclear where a -year-old belongs. It’s clearer to use – and –.
- Make sure the number of selections allowed matches the question. For age group, each person should select only one option. For racial background, it may be necessary to allow multiple selections because many individuals identify with more than one racial background.
Leading questions
Another important source of bias is the leading question — a question written in a way that nudges people toward a particular answer. Both examples below ask about teacher salaries, but each is written to push the reader in a specific direction.
Example: Leading question 1
Taxpayers spent over 10 billion dollars on teachers salaries last year and yet the students in this country are still behind on reading, mathematics, and science in comparison to other countries. Do you believe that it is appropriate for teachers to ask for even higher pay than they already have?
This is a leading question because it encourages the reader to believe that teachers are overpaid and not doing their jobs properly. That framing makes respondents more likely to say it is not appropriate for teachers to ask for higher pay.
The statistic is also vague and likely misleading. It gives a total national expenditure but does not provide an average salary, does not account for the number of teachers, and does not address how salaries vary.
Example: Leading question 2
Teachers have to go to post-secondary school for 6 years and many have a student loan debt of over when they graduate. Many of them are struggling to keep up with even their most basic necessities. Do you believe that the government should increase teachers salaries?
This is also a leading question because it encourages the reader to believe that teachers are underpaid and struggling to meet basic needs. That framing makes respondents more likely to say the government should increase teachers salaries.
The information provided is also vague and likely misleading. It does not compare teachers’ student loan debt to the overall population, and it does not define what “many” means or provide evidence for the claim about struggling with basic necessities.