Damned Lies and Statistics

1. Statistics are socially constructed, not objective facts

We think of statistics as facts that we discover, not as numbers we create.

Statistics reflect choices. Every statistic involves decisions about what to count and how to count it. These choices shape the resulting numbers and our understanding of social issues. For example:

• Defining "homelessness" impacts estimates:
  • Only counting people on streets vs. including those in shelters
  • Including or excluding people staying with friends/family
  • Time frame: one night vs. any time in past year

Social context matters. Statistics don't exist in a vacuum - they're created and used by people with particular goals and perspectives. Key factors include:

• Who is producing the statistic and why
• How the media reports and frames numbers
• Which statistics gain traction and get repeated
• How different groups interpret and use the same figures

2. Bad statistics often stem from guessing and poor definitions

Certainly, activists' numbers aren't much good from the start, because they are based on nothing more than guesses or dubious data.

Guesstimates abound. When hard data is lacking, advocates often resort to educated guesses or ballpark figures. These tend to:

• Err on the side of exaggeration
• Use round numbers (e.g. "one million victims")
• Get repeated and treated as fact

Definitions are crucial. How a problem is defined determines what gets counted. Broad definitions lead to bigger numbers. For example, defining "child abuse" could include:

• Physical abuse only
• Physical and emotional abuse
• Neglect
• Witnessing domestic violence

The broader the definition, the larger the resulting statistic will be. Activists often prefer inclusive definitions to highlight the scale of problems they care about.

3. Mutant statistics arise from misunderstanding and manipulation

Bad statistics live on; they take on lives of their own.

Mangled meanings. Statistics often get distorted as they're repeated, like a game of telephone. Common transformations include:

• Confusing "having a condition" with "dying from it"
• Mistaking "new cases" for "total cases"
• Applying a statistic for one group to the whole population

Motivated misinterpretation. People may deliberately twist statistics to support their views. This can involve:

• Cherry-picking favorable numbers
• Using misleading comparisons
• Presenting estimates as definitive facts
• Ignoring important context or caveats

Once a mutant statistic is created, it often gets repeated without scrutiny, taking on a life of its own.

4. Inappropriate comparisons lead to misleading conclusions

The lesson should be clear: bad statistics live on; they survive and even thrive.

Apples to oranges. Comparing unlike things can produce distorted impressions. Common pitfalls include:

• Comparing different time periods without accounting for population growth
• Contrasting countries with different definitions or measurement systems
• Juxtaposing raw numbers for groups of very different sizes

Change over time. Comparing statistics from different eras is tricky because:

• Definitions and measurement methods evolve
• Social attitudes and reporting practices shift
• The underlying reality being measured may have changed

For instance, rising child abuse reports may reflect increased awareness and reporting rather than more actual abuse.

5. Competing interests fuel debates over social statistics

Statistics, then, can become weapons in political struggles over social problems and social policy.

Stakes shape statistics. Different groups have vested interests in particular numbers:

• Activists want big numbers to highlight problems
• Officials may prefer smaller figures showing progress
• Businesses push stats favorable to their industry

Stat wars erupt. When issues are contentious, competing sides often:

• Cherry-pick supportive statistics
• Attack opponents' numbers as flawed
• Produce dueling studies with different methods
• Frame the same data in contradictory ways

These battles reflect underlying disagreements about values and priorities, not just technical disputes over numbers.

6. Critical thinking is essential for evaluating statistical claims

The Critical attempt to evaluate numbers, to distinguish between good statistics and bad statistics.

Beyond face value. A critical approach involves asking questions like:

• How was this number produced?
• What's being counted and what's left out?
• Who created this statistic and why?
• What assumptions underlie the calculation?
• How does it compare to other estimates?

Balancing skepticism and utility. The goal isn't to dismiss all statistics, but to:

• Recognize limitations of all numbers
• Assess relative strengths and weaknesses
• Use statistics as tools, not magic facts
• Seek multiple sources and perspectives

Critical thinking allows us to navigate between naive acceptance and cynical rejection of statistical claims.

7. Good statistics require clear definitions, measures, and samples

Good statistics are based on clear, reasonable definitions.

Key elements of quality. Robust statistics typically involve:

• Precise, transparent definitions of what's being counted
• Valid, consistent measurement methods
• Representative samples of adequate size
• Clear explanations of limitations and margins of error

Red flags. Warning signs of potentially problematic statistics include:

• Vague or shifting definitions
• Unexplained methodological choices
• Convenience samples or tiny data sets
• Lack of information about how numbers were produced
• Implausibly precise or dramatically round figures

Good statistics don't just present a number, but explain how it was derived.

8. Social context shapes the creation and interpretation of statistics

Our numbers are undoubtedly good numbers, while our opponents' figures are questionable at best.

Beyond the numbers. Understanding statistics requires considering:

• Historical context (e.g. changing social norms)
• Institutional practices (e.g. how police classify crimes)
• Cultural assumptions (e.g. what counts as "poverty")
• Political climate (e.g. which issues get measured)

Motivated reasoning. People tend to:

• Accept statistics that confirm their views
• Scrutinize numbers that challenge their beliefs
• Interpret ambiguous data in ways that fit their perspective

Recognizing these tendencies in ourselves and others is crucial for fair evaluation of statistical claims.

9. Statistical literacy is crucial for informed citizenship

Being Critical requires more thought, but failing to adopt a Critical mind-set makes us powerless to evaluate what others tell us.

Ubiquitous numbers. Modern society is awash in statistics about:

• Social problems
• Economic trends
• Public opinion
• Government policies
• Health risks

Empowered engagement. Statistical literacy allows citizens to:

• Critically assess claims by media, politicians, and activists
• Understand the limitations of data-driven arguments
• Participate more effectively in public debates
• Make more informed personal and political choices

Developing these skills is an ongoing process, but essential for navigating our data-rich world.

Last updated: August 14, 2024