Standard Deviations

1. Meaningless Patterns Can Be Misleading

We are too quick to assume that meaningless patterns are meaningful when they are presented as evidence of the consequences of a government policy, the power of a marketing plan, the success of an investment strategy, or the benefits of a food supplement.

Pattern recognition is innate. Humans are genetically predisposed to seek patterns, a trait that aided survival for our ancestors. However, this inclination can lead to misinterpretations in the modern world, where data is complex and not easily interpreted. We often assume that meaningless patterns are meaningful, leading to flawed conclusions.

Examples of misleading patterns:

• Paul the Octopus predicting World Cup winners based on flag colors.
• The belief that athletes on Sports Illustrated covers are "jinxed."
• Assuming a correlation between messy rooms and racism.

The need for skepticism. It's crucial to approach patterns with skepticism, demanding logical explanations and testing theories with fresh data. Extraordinary claims require extraordinary evidence. Don't be fooled into thinking that a pattern is proof.

2. Garbage In, Gospel Out: Data Quality Matters

I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

Computers are not infallible. The saying "garbage in, gospel out" highlights the danger of blindly trusting computer-generated results without considering the quality of the input data. Computers execute instructions accurately, but their output is only as good as the data they're fed.

Examples of data-driven errors:

• The Mars Climate Orbiter burning up due to a unit conversion error.
• JP Morgan's "London Whale" debacle caused by a faulty averaging calculation.
• The Joint Economic Committee's report on wealth concentration skewed by a single data entry error.

The importance of critical thinking. It's essential to critically evaluate the data and assumptions underlying any analysis, regardless of how sophisticated the methods used. Question the source, accuracy, and relevance of the data before accepting the conclusions.

3. Beware of Unfair Comparisons

Comparisons are the lifeblood of empirical studies.

Comparisons are essential. Empirical studies rely on comparisons to determine the effectiveness of treatments, policies, or strategies. However, not all comparisons are valid. Superficial or misleading comparisons can lead to incorrect conclusions.

Types of unfair comparisons:

• Comparing percentage changes in small numbers to those in large numbers (e.g., Wellfleet's murder rate).
• Attributing causation to correlated trends that are simply increasing with population (e.g., mine production and property values).
• Ignoring self-selection bias (e.g., assuming college graduates earn more solely because of their degree).

Focus on relevant metrics. Ensure that comparisons are based on relevant metrics and account for potential confounding factors. Avoid being distracted by irrelevant details or misleading statistics.

4. Oops! Even Experts Make Mistakes

This is personally quite embarrassing because I pride myself on being careful with data.

Human error is inevitable. Even highly educated and experienced researchers are prone to making mistakes. These errors can range from simple typos to flawed methodologies.

Examples of expert errors:

• Reinhart and Rogoff's spreadsheet error that influenced austerity policies.
• Steven Levitt's programming error in his abortion-crime study.

The need for transparency and replication. It's crucial for researchers to be transparent about their methods and data, allowing others to replicate their findings and identify potential errors. Scientific progress relies on the ability to scrutinize and challenge existing knowledge.

5. Graphs Can Lie: Visual Deception

We tried it with numbers and found it was very hard to read on television, so we took them off. We were just trying to get a point across.

Graphs can distort data. While graphs can be powerful tools for visualizing data, they can also be used to mislead and manipulate. Intentional or unintentional design choices can create a false impression of trends and relationships.

Common graphical gaffes:

• Omitting zero from the vertical axis to exaggerate changes.
• Using inconsistent intervals on the time axis.
• Employing two vertical axes to create misleading comparisons.
• Adding chartjunk to distract from the data.

Critical evaluation of graphs. Always examine graphs carefully, paying attention to the axes, scales, and data sources. Be wary of graphs that seem too good to be true or that support a particular agenda.

6. Common Sense is Essential

Probabilities are nothing but common sense reduced to calculation.

Calculations alone are not enough. While statistical calculations are valuable, they should always be grounded in common sense and logical reasoning. Blindly accepting numerical results without considering their plausibility can lead to absurd conclusions.

Examples of defying common sense:

• The Monty Hall problem, where intuition suggests a 50/50 chance but switching doors doubles your odds.
• The two-child paradox, where flawed logic leads to incorrect probability calculations.

The importance of critical thinking. Always question the assumptions and implications of statistical analyses. If the results defy common sense, investigate further to identify potential flaws in the methodology or reasoning.

7. Confounding Factors Can Distort Results

Although proof of discriminatory motive is generally required in disparate treatment cases, the evidence of subjective, standard-less decision-making by company officials, which is a convenient mechanism for discrimination, satisfies this requirement.

Confounding factors can mislead. When analyzing observational data, it's crucial to consider potential confounding factors that may be influencing the observed relationships. Ignoring these factors can lead to incorrect conclusions about cause and effect.

Examples of confounding factors:

• The claim that arrests reduce voting, ignoring that those arrested may be less likely to vote anyway.
• The assertion that pitcher beer consumption increases drinking, ignoring that those who order pitchers intend to drink more.
• The belief that the French are unfriendly, ignoring that those who return to France likely had a good experience.

The need for careful analysis. Always consider potential confounding factors and attempt to control for them in your analysis. Be wary of drawing causal conclusions from observational data without accounting for these influences.

8. The Hot Hand Fallacy: When Luck is Mistaken for Skill

You’re in a world all your own. It’s hard to describe. But the basket seems to be so wide. No matter what you do, you know the ball is going to go in.

Streaks can be coincidental. People often perceive "hot streaks" in sports and other activities, believing that past success increases the likelihood of future success. However, these streaks may simply be due to random chance.

The law of small numbers. We tend to overestimate the significance of patterns in small samples, failing to recognize that randomness can generate seemingly meaningful sequences.

The importance of objectivity. Avoid being swayed by anecdotal evidence or personal feelings. Rely on objective data and statistical analysis to determine whether a hot streak is real or simply a statistical illusion.

9. Regression to the Mean: The Inevitable Pull

On two occasions I have been asked [by members of parliament], “Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?” … I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

Extreme values tend to regress. When a trait is measured imperfectly, extreme values tend to regress toward the mean. This means that individuals or entities that perform exceptionally well or poorly in one period are likely to perform closer to average in the next.

Examples of regression to the mean:

• Children of unusually tall parents tend to be shorter than their parents.
• Students who score highest on a test tend to score lower on a subsequent test.
• Companies with the highest profits in one year tend to have lower profits in the following year.

Avoid overreacting to extreme results. Recognize that regression to the mean is a natural phenomenon and avoid drawing unwarranted conclusions based on extreme performances.

10. Theory Without Data is Just Speculation

Data! Data! Data! I can’t make bricks without clay.

Data and theory are both essential. While data can be misleading, theory without data is just speculation. A compelling theory should be supported by empirical evidence.

Examples of theory without data:

• Malthus's predictions of overpopulation and resource depletion.
• The belief that the Super Bowl outcome predicts the stock market.
• The assumption that the world's resources are fixed.

The need for empirical testing. Always demand evidence to support theoretical claims. Be wary of arguments that rely solely on logic or intuition without any empirical backing.

Last updated: April 13, 2025