Noise

1. Noise: The hidden variability in human judgment

Wherever there is judgment, there is noise, and more of it than you think.

Noise defined. Noise is the unwanted variability in judgments that should be identical. It occurs when different professionals, given the same information, reach different conclusions. This variability is often invisible and underestimated, yet it has significant consequences in various fields, including:

• Medicine: Doctors making different diagnoses for the same patient
• Criminal justice: Judges giving vastly different sentences for similar crimes
• Business: Underwriters setting widely varying premiums for the same risk

Impact of noise. The presence of noise leads to unfairness, inconsistency, and costly errors in decision-making processes. Unlike bias, which causes systematic errors in a particular direction, noise creates random scatter in judgments. This randomness can be just as detrimental as bias, if not more so, because:

• It undermines the credibility of professional judgment
• It results in unequal treatment of similar cases
• It can lead to significant financial losses in business settings

2. Bias and noise: Two distinct components of error

Bias and noise—systematic deviation and random scatter—are different components of error.

Understanding error. Error in judgment consists of two components: bias and noise. While both contribute to inaccuracy, they do so in different ways:

• Bias: The average error or systematic deviation from the target
• Noise: The variability or scatter of judgments around the average

Error equation. The relationship between bias and noise in determining overall error can be expressed mathematically:

Total Error = Bias² + Noise²

This equation highlights that:

• Reducing either bias or noise will improve accuracy
• In many cases, noise contributes more to error than bias
• Efforts to reduce error should address both components

3. The pervasive nature of noise across various fields

System noise is a serious problem: it costs us hundreds of millions.

Widespread occurrence. Noise is not limited to a few specific domains but is pervasive across various fields and professions. Examples include:

• Medicine: Variability in diagnoses, treatment recommendations, and prognoses
• Law: Inconsistencies in sentencing, bail decisions, and asylum approvals
• Business: Differences in hiring decisions, performance evaluations, and financial forecasts
• Government: Disparities in policy implementation and regulatory decisions

Hidden costs. The financial and social costs of noise are often underestimated or entirely overlooked. These costs manifest as:

• Inefficiencies in resource allocation
• Unfair treatment of individuals
• Reduced public trust in institutions
• Missed opportunities and suboptimal decisions

4. Measuring and auditing noise in organizations

To understand error in judgment, we must understand both bias and noise.

Noise audit. A noise audit is a systematic approach to measuring the level of noise within an organization or decision-making system. The process typically involves:

1. Presenting multiple judges with the same cases or scenarios
2. Collecting their independent judgments
3. Analyzing the variability in these judgments

Types of noise. A noise audit can reveal different types of noise:

• Level noise: Consistent differences in judgment levels between individuals
• Pattern noise: Differences in how individuals respond to various case features
• Occasion noise: Variability in an individual's judgments of the same case over time

Benefits of auditing. Conducting a noise audit can:

• Raise awareness of the extent of noise in an organization
• Identify specific areas or processes prone to high noise levels
• Provide a baseline for measuring the effectiveness of noise reduction efforts

5. The psychology behind noise: Why it occurs and persists

Judgment is like a free throw: however hard we try to repeat it precisely, it is never exactly identical.

Cognitive factors. Several psychological mechanisms contribute to the generation and persistence of noise:

• Excessive coherence: The tendency to form overly consistent impressions
• Confirmation bias: Seeking information that supports initial impressions
• Availability bias: Overweighting easily recalled information
• Anchoring: Being influenced by irrelevant initial information

Illusion of agreement. People often overestimate the extent to which others would agree with their judgments. This illusion stems from:

• The difficulty in imagining alternative perspectives
• The tendency to seek confirmation of one's views
• The lack of regular feedback on judgment accuracy

Objective ignorance. The inherent unpredictability of many situations contributes to noise. People often underestimate their level of uncertainty, leading to:

• Overconfidence in predictions
• Failure to account for unknown factors
• Difficulty in distinguishing skill from luck in judgment

6. Decision hygiene: Strategies to reduce noise and improve judgments

Decision hygiene will prevent errors without knowing what they are.

Principles of decision hygiene. To reduce noise and improve judgment quality, organizations can implement several strategies:

1. Favor accuracy over individual expression in judgments
2. Adopt an outside view and think statistically
3. Structure complex judgments into independent tasks
4. Resist premature intuitions
5. Obtain independent judgments before group discussion
6. Use relative rather than absolute judgments when possible

Specific techniques. Various methods can be employed to implement decision hygiene:

• Guidelines and checklists: Provide structured approaches to decision-making
• The mediating assessments protocol: Break down complex decisions into separate evaluations
• Aggregation of multiple independent judgments: Leverage the wisdom of crowds
• Calibrated use of algorithms: Complement human judgment with data-driven insights

Overcoming resistance. Implementing decision hygiene often faces resistance due to:

• Belief in the superiority of intuitive judgment
• Discomfort with perceived loss of autonomy
• Underestimation of the prevalence and impact of noise

7. Balancing rules and standards: The quest for optimal noise reduction

Rules simplify life, and reduce noise. But standards allow people to adjust to the particulars of the situations.

Rules vs. standards. The choice between rules and standards represents a fundamental trade-off in noise reduction:

• Rules: Precise, inflexible guidelines that minimize discretion
• Standards: General principles that allow for case-by-case interpretation

Considerations. When deciding between rules and standards, organizations should weigh:

• The costs of decision-making: Rules typically require less effort to apply
• The costs of errors: Standards may allow for better tailoring to unique situations
• The need for consistency: Rules promote uniformity but may sacrifice adaptability
• The complexity of the domain: Standards may be more suitable for highly variable contexts

Optimal noise. The goal is not to eliminate all noise but to find the right balance that:

• Reduces harmful variability in judgments
• Maintains flexibility to address unique circumstances
• Preserves human dignity and the sense of fair treatment
• Allows for the evolution of values and practices over time

Last updated: August 9, 2024