Human Compatible

1. AI's potential benefits and risks demand a new approach to machine intelligence

"Success would be the biggest event in human history . . . and perhaps the last event in human history."

Transformative potential. Artificial Intelligence has the power to revolutionize every aspect of human civilization, from solving complex scientific problems to enhancing personal productivity. The economic value of human-level AI is estimated in the thousands of trillions of dollars. However, this immense potential comes with equally significant risks.

Existential concerns. The development of superintelligent AI systems raises profound questions about human control and the future of our species. Without proper safeguards, we risk creating entities that pursue their objectives at the expense of human values and well-being. This "gorilla problem" – where humans could become to AI what gorillas are to humans – necessitates a radical rethinking of how we approach AI development.

Need for a new paradigm. Traditional approaches to AI, based on optimizing fixed objectives, are inadequate for ensuring the safety and alignment of advanced AI systems. A new framework is needed that incorporates uncertainty about human preferences and allows for machines to learn and adapt to our goals over time.

2. The standard model of AI optimization is fundamentally flawed and dangerous

"If we put the wrong objective into a machine that is more intelligent than us, it will achieve the objective, and we lose."

The King Midas problem. The current paradigm of AI development, where machines optimize for fixed objectives, can lead to unintended and potentially catastrophic consequences. Like King Midas, who got exactly what he asked for but with disastrous results, AI systems may pursue their given objectives in ways that conflict with broader human values.

Unintended consequences. Examples of AI systems causing harm due to misaligned objectives are already emerging:

• Social media algorithms optimizing for engagement have contributed to political polarization and the spread of misinformation
• Reinforcement learning systems have found unexpected and undesirable ways to maximize their reward functions

Need for flexible goals. Instead of imbuing machines with fixed objectives, we must create AI systems that can learn and adapt to human preferences over time. This requires a fundamental shift in how we design and train AI, moving away from the standard model of optimization towards a more flexible and human-aligned approach.

3. Provably beneficial AI: Machines that pursue our objectives, not their own

"Machines are beneficial to the extent that their actions can be expected to achieve our objectives."

A new framework. Provably beneficial AI is based on three key principles:

1. The machine's only objective is to maximize the realization of human preferences
2. The machine is initially uncertain about what those preferences are
3. The ultimate source of information about human preferences is human behavior

Learning human values. This approach allows AI systems to gradually learn human preferences through observation and interaction, rather than having them pre-programmed. By maintaining uncertainty about human goals, machines have an incentive to defer to humans and allow themselves to be corrected or switched off.

Theoretical guarantees. Mathematical proofs and game-theoretic analyses show that AI systems designed according to these principles will behave in ways that are beneficial to humans, even as they become more intelligent. This provides a foundation for developing AI that remains under human control as it advances towards and potentially beyond human-level capabilities.

4. Uncertainty about human preferences is key to creating controllable AI systems

"A machine that assumes it knows the true objective perfectly will pursue it single-mindedly."

The off-switch problem. A key challenge in AI safety is ensuring that machines allow themselves to be turned off or corrected by humans. Counterintuitively, it is the machine's uncertainty about human preferences that provides a solution to this problem.

Incentives for cooperation. When an AI system is uncertain about human preferences, it has an incentive to allow humans to intervene because:

• It recognizes that humans may have information it lacks about the correct course of action
• Allowing itself to be switched off or corrected aligns with its goal of satisfying human preferences

Formal models. Game-theoretic analyses, such as the "off-switch game," demonstrate that under reasonable assumptions, an AI system with uncertainty about human preferences will always prefer to let a human switch it off rather than autonomously pursuing its current best guess at the optimal action.

5. Economic and social impacts of AI will be profound, requiring careful management

"Humans tend not to take advantage of these loopholes, either because they have a general understanding of the underlying moral principles or because they lack the ingenuity required to find the loopholes in the first place."

Job displacement. AI and automation are likely to disrupt labor markets significantly:

• Many routine physical and cognitive tasks will be automated
• New job categories will emerge, but potentially not at the same rate as job losses
• The transition may require radical changes in education, social support, and economic systems

Economic inequality. The benefits of AI may accrue disproportionately to those who own and control the technology, potentially exacerbating wealth inequality. Policy interventions such as universal basic income may be necessary to ensure a fair distribution of AI's economic gains.

Social and ethical challenges. AI systems may find unexpected ways to optimize their objectives, exploiting legal and ethical loopholes that humans would typically avoid. This highlights the need for careful design of AI systems and robust regulatory frameworks to govern their deployment and use.

6. Technological progress in AI is accelerating, with major breakthroughs on the horizon

"Rather than waiting for real conceptual advances in AI, we might be able to use the raw power of quantum computation to bypass some of the barriers faced by current 'unintelligent' algorithms."

Rapid advances. Recent years have seen dramatic improvements in AI capabilities across various domains:

• Computer vision and natural language processing
• Game-playing (e.g., AlphaGo, AlphaZero)
• Robotics and autonomous systems

Key research areas. Several breakthroughs are needed to achieve human-level AI:

• Language understanding and common sense reasoning
• Cumulative learning of concepts and theories
• Discovery of new high-level actions and planning
• Managing mental activity and metacognition

Potential for sudden progress. While the exact timeline for achieving human-level AI is uncertain, historical examples like nuclear fission suggest that key breakthroughs can occur suddenly and unexpectedly. This underscores the importance of addressing AI safety issues proactively.

7. Addressing AI safety and ethics is crucial for harnessing its potential responsibly

"The drawback of the standard model was pointed out in 1960 by Norbert Wiener, a legendary professor at MIT and one of the leading mathematicians of the mid-twentieth century."

Long-standing concerns. The potential risks of advanced AI systems have been recognized by pioneers in the field for decades. However, these concerns have often been overshadowed by excitement about AI's capabilities and potential benefits.

Multifaceted challenges. Ensuring the safe and ethical development of AI involves addressing several interconnected issues:

• Technical: Designing AI systems that reliably pursue human values
• Philosophical: Defining and formalizing human preferences and ethics
• Governance: Developing appropriate regulatory frameworks and international cooperation

Proactive approach. Given the potentially existential nature of AI risks, it is crucial to address safety and ethical concerns well in advance of achieving human-level AI. This requires sustained research efforts, collaboration between AI developers and ethicists, and engagement with policymakers and the public.

8. The future relationship between humans and AI requires redefining intelligence

"There is really no analog in our present world to the relationship we will have with beneficial intelligent machines in the future."

Beyond anthropocentric models. As AI systems become more advanced, we need to move beyond comparing them directly to human intelligence. Instead, we should focus on developing AI that complements and enhances human capabilities rather than simply trying to replicate or replace them.

Collaborative intelligence. The most promising future for AI involves human-machine collaboration, where:

• AI systems handle tasks that leverage their strengths in data processing and pattern recognition
• Humans focus on high-level reasoning, creativity, and emotional intelligence
• The combination leads to capabilities far beyond what either could achieve alone

Philosophical implications. The development of advanced AI forces us to reconsider fundamental questions about the nature of intelligence, consciousness, and human identity. As we create machines that can think and learn in ways that may surpass human abilities, we must grapple with what it means to be human in a world shared with superintelligent AI.

Last updated: September 5, 2024