Superforecasting

1. Superforecasters exist and outperform experts and prediction markets

Superforecasters looking out three hundred days were more accurate than regular forecasters looking out one hundred days.

Superforecasters are real. The Good Judgment Project, a forecasting tournament sponsored by IARPA, discovered that some individuals consistently outperform others in predicting geopolitical events. These "superforecasters" beat not only average forecasters but also prediction markets and even professional intelligence analysts with access to classified information.

Characteristics of superforecasters:

• High cognitive abilities and knowledge
• Active open-mindedness
• Granular thinking
• Regular belief updating
• Strong teamwork skills
• Probabilistic thinking
• Ability to break down complex problems

Superforecasters are not infallible, but they demonstrate that accurate forecasting is a skill that can be cultivated and improved with practice and the right mindset.

2. Cognitive abilities and knowledge are important, but not sufficient

Regular forecasters scored higher on intelligence and knowledge tests than about 70% of the population. Superforecasters did better, placing higher than about 80% of the population.

Intelligence matters, but only to a point. While superforecasters tend to score higher on intelligence and knowledge tests, these factors alone do not explain their superior performance. Many highly intelligent and knowledgeable individuals fail to make accurate predictions.

The key difference lies in how superforecasters use their cognitive abilities:

• They actively seek out diverse perspectives
• They constantly question their own assumptions
• They focus on process rather than outcomes
• They are willing to admit mistakes and change their minds

This suggests that while a certain level of intelligence and knowledge is necessary for good forecasting, what truly sets superforecasters apart is their approach to thinking and decision-making.

3. Active open-mindedness is crucial for accurate forecasting

For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded.

Embracing uncertainty is key. Superforecasters cultivate a mindset of active open-mindedness, which involves:

• Constantly seeking out information that might prove their beliefs wrong
• Welcoming criticism and alternative viewpoints
• Adjusting their beliefs in light of new evidence
• Avoiding ideological commitments that might blind them to contrary information

This approach helps superforecasters avoid common cognitive biases such as confirmation bias and overconfidence. By treating their beliefs as provisional hypotheses rather than fixed truths, they remain flexible and responsive to changing circumstances.

4. Granular thinking improves forecast accuracy

Superforecasters often think in terms of precise probability estimates, like 63% versus 65%, rather than vague terms like "likely" or "unlikely."

Precision enhances accuracy. Superforecasters distinguish between fine degrees of uncertainty, often using probability estimates down to single percentage points. This granular thinking forces them to:

• Carefully consider the strength of their evidence
• Avoid vague language that can mask uncertainty
• Make more precise and testable predictions

Benefits of granular thinking:

• Allows for more nuanced updates as new information emerges
• Facilitates clearer communication about levels of certainty
• Enables more accurate aggregation of multiple forecasts

By thinking in precise probabilities, superforecasters develop a more calibrated sense of uncertainty and improve their overall accuracy.

5. Regular belief updating is key to superforecasting

Superforecasters update much more frequently, on average, than regular forecasters.

Constant refinement improves predictions. Superforecasters don't just make a forecast and stick to it; they continually revise their predictions as new information becomes available. This process of regular updating:

• Allows for quick incorporation of new evidence
• Helps correct initial errors or biases
• Improves calibration over time

Effective updating techniques:

• Set reminders to review forecasts regularly
• Actively seek out new information relevant to the forecast
• Make small, incremental adjustments rather than dramatic swings
• Track the reasons for updates to identify patterns and improve future forecasts

By treating forecasting as an ongoing process rather than a one-time event, superforecasters maintain more accurate predictions over time.

6. Teamwork enhances forecasting performance

On average, teams were 23% more accurate than individuals.

Collaboration boosts accuracy. The Good Judgment Project found that teams of forecasters consistently outperformed individuals, even when those individuals were superforecasters. This is because teams:

• Bring together diverse perspectives and knowledge
• Allow for constructive criticism and debate
• Aggregate individual judgments to reduce errors

Effective forecasting teams:

• Foster an environment of respectful disagreement
• Encourage members to explain their reasoning clearly
• Use structured techniques to combine individual forecasts
• Maintain a balance between cohesion and diversity of thought

By leveraging the power of collective intelligence, teams can achieve levels of accuracy that surpass even the best individual forecasters.

7. Overconfidence is the enemy of good forecasting

Confidence and accuracy are positively correlated. But research shows we exaggerate the size of the correlation.

Beware of certainty. One of the most common pitfalls in forecasting is overconfidence – the tendency to be more certain in our predictions than the evidence warrants. Superforecasters combat this by:

• Constantly questioning their own judgments
• Seeking out disconfirming evidence
• Expressing uncertainty even when they feel confident

Techniques to reduce overconfidence:

• Consider alternative scenarios that could lead to different outcomes
• Explicitly list reasons why your forecast might be wrong
• Use calibration training to improve your sense of true probabilities
• Avoid making extreme probability estimates (0% or 100%) except in rare cases

By maintaining appropriate humility about their forecasts, superforecasters avoid the trap of false certainty that often leads to poor predictions.

8. Probabilistic thinking is essential for accurate predictions

Probabilistic thinking demands teasing subtle signals from noisy news flows—all the while resisting the lure of wishful thinking.

Embrace uncertainty. Superforecasters think in terms of probabilities rather than certainties. This mindset allows them to:

• Express nuanced degrees of confidence in their predictions
• Incorporate new information smoothly without dramatic reversals
• Avoid the trap of binary thinking (either it will happen or it won't)

Key aspects of probabilistic thinking:

• Use precise numerical probabilities (e.g., 60% chance) instead of vague terms
• Consider multiple possible outcomes and assign probabilities to each
• Update probabilities incrementally as new information arrives
• Recognize that even low-probability events sometimes occur

By embracing uncertainty and thinking probabilistically, superforecasters can make more accurate and nuanced predictions about complex events.

9. Breaking down complex problems improves forecasting

Channel the playful but disciplined spirit of Enrico Fermi who—when he wasn't designing the world's first atomic reactor—loved ballparking answers to head-scratchers such as "How many extraterrestrial civilizations exist in the universe?"

Decompose to conquer. Superforecasters excel at breaking down complex problems into smaller, more manageable components. This approach, often called "Fermi-izing" after physicist Enrico Fermi, involves:

• Identifying key factors that influence the outcome
• Estimating probabilities for each factor
• Combining these estimates to reach an overall forecast

Benefits of problem decomposition:

• Makes seemingly intractable problems more approachable
• Reveals hidden assumptions and knowledge gaps
• Allows for more precise updating as new information emerges
• Facilitates clearer reasoning and communication about the forecast

By systematically breaking down complex issues, superforecasters can tackle even the most daunting prediction challenges with greater accuracy and confidence.

10. Good judgment requires balancing opposing errors

Implementing each commandment requires balancing opposing errors.

Navigate trade-offs wisely. Effective forecasting often involves finding the right balance between competing principles or potential errors. Some key trade-offs include:

• Confidence vs. humility
• Quick updates vs. stable predictions
• Reliance on data vs. incorporation of intuition
• Individual judgment vs. team consensus
• Granular thinking vs. big-picture perspective

Superforecasters excel at:

• Recognizing when they're at risk of falling into one extreme
• Adjusting their approach to maintain an optimal balance
• Being flexible in their strategies depending on the specific forecasting challenge

By skillfully navigating these trade-offs, superforecasters can avoid common pitfalls and maintain good judgment across a wide range of forecasting situations.

Last updated: August 8, 2024