Statistical analysis allows researchers to quantify a huge range of phenomena, allowing them to study topics as diverse as social behavior, political opinions, cellular biology and forest growth rates from an objective perspective. Statistical approaches to research are far from perfect, however, and can produce serious distortions and misleading conclusions.
Sampling Error
A statistical test is only as good as the data it analyzes. If researchers collect data using faulty or biased procedures, resulting statistical analysis will be misleading. The term "sampling error" denotes the gap between the sample population and the actual population. A highly representative sample produces very little error, but a big gap between sample and population creates misleading data. If a researcher surveys college students about their favorite type of movie by standing outside a football game with a questionnaire, for instance, she might include more men than women in the sample. That might lead her to conclude incorrectly that college students prefer action movies over romantic comedies.
Correlation Versus Causation
Another problem with statistical analysis is the tendency to jump to unjustified conclusions about causal relationships. Researchers often find evidence that two variables are highly correlated, but that doesn't prove that one variable causes another. For example, researchers at the New York Times found cities with higher gun ownership rates also had higher murder rates. It's tempting to conclude that to allow more guns into a city caused the murder rate to rise, but it's equally possible that people purchased guns because they felt threatened by already elevated violent crime rates. Statistical analysis alone is not capable of proving causal relationships between two variables.
Construct Validity
Statistical analysis is a means of using aggregated measurements to draw conclusions, but if researchers aren't measuring the right thing, the analysis will fail. Construct validity is the degree to which researchers' measurements actually reflect what they're trying to measure. For example, advertising researchers usually want to study how effectively an ad persuades people to buy a product. They estimate the persuasive effect through surveys that try to gauge consumers' "purchase intention" toward the product. The problem, however, is that while study participants might say an ad made them want to buy the product, their actual behaviors might not reflect that attitude. The construct validity of "purchase intention" may be questionable, because purchase intention doesn't always lead to actual purchases.
Simplified Solutions
A final problem with statistical analysis is its tendency to produce excessively simple answers to complex questions. A soft drink company might test the efficacy of its ad campaign, for example, and through a statistical analysis, determine that the new ads could boost sales by 10 percent. No matter how detailed their surveys, however, the company's statistical analysis can't reveal exactly why the ads work. Maybe viewers respond to an emotional component in the ad, or maybe they were just reminded of a brand name they already like. Statistical data can't capture the complexity of something like emotion very easily, so statistical analyses often miss important but subtle information.
Related Articles
References
Writer Bio
Nick Robinson is a writer, instructor and graduate student. Before deciding to pursue an advanced degree, he worked as a teacher and administrator at three different colleges and universities, and as an education coach for Inside Track. Most of Robinson's writing centers on education and travel.