AI Snake Oil

1. AI Snake Oil: Separating Reality from Hype

AI snake oil is AI that does not and cannot work as advertised.

Defining AI's Scope. Artificial Intelligence (AI) is a broad term encompassing diverse technologies, from generative models like ChatGPT to predictive algorithms used in finance. It's crucial to distinguish between these different forms of AI, as their capabilities, applications, and potential for failure vary significantly.

The Rise of Generative AI. Generative AI, exemplified by chatbots and image generators, has captured public attention with its ability to create realistic content. However, it's essential to recognize that this technology is still immature, unreliable, and prone to misuse, often accompanied by hype and misinformation.

The Perils of Predictive AI. Predictive AI, used to forecast future outcomes and guide decision-making in areas like policing, hiring, and healthcare, is often oversold and ineffective. AI snake oil refers to AI that doesn't and can't function as advertised, posing a societal problem that requires critical evaluation and discernment.

2. Predictive AI: Flawed Logic and Harmful Outcomes

Even if AI can make accurate predictions based on past data, we can’t know how good the resulting decisions will be before AI is deployed on a new dataset or in a new setting.

Automated Decision-Making. Predictive AI is increasingly used to automate consequential decisions about individuals, often without their knowledge or consent. These systems, employed in areas like healthcare, hiring, and criminal justice, can have profound impacts on people's lives and opportunities.

Recurring Shortcomings. Despite claims of accuracy and fairness, predictive AI systems are plagued by recurring shortcomings, including:

• Making good predictions that lead to bad decisions
• Incentivizing gaming and strategic manipulation
• Over-reliance on AI without adequate human oversight
• Using data from one population to make predictions about another
• Exacerbating existing inequalities

Embracing Unpredictability. The pervasiveness of predictive logic stems from a deep discomfort with randomness. However, accepting the inherent uncertainty in many outcomes can lead to better decisions and institutions, fostering a world genuinely open to the unpredictability of the future.

3. The Illusion of Predictability: Why the Future Remains Unwritten

The same fundamental roadblocks seemed to come up over and over, but since researchers in different disciplines rarely talk to each other, many scientific fields had independently rediscovered these limits.

Limits to Prediction. Accurately predicting people's social behavior is not a solvable technology problem, and determining people's life chances on the basis of inherently faulty predictions will always be morally problematic. The challenges are ultimately not about AI, but rather the nature of social processes.

The Fragile Families Challenge. The Fragile Families Challenge, a large-scale study that tried to predict children’s outcomes using AI and lots of data, found that the best models were only slightly better than a coin flip, highlighting the difficulty of predicting life outcomes.

The Meme Lottery. The social media equivalent of a blockbuster or a bestseller is the viral hit; the main difference is that a social media post’s success or failure is determined on an accelerated timescale compared to a book or movie. A tiny fraction of videos or tweets go viral while the rest get little engagement.

4. Generative AI: Demystifying the Technology and Its Double-Edged Sword

The technology is remarkably capable, yet it struggles with many things a toddler can do.

Understanding Generative AI. Generative AI, encompassing technologies like ChatGPT and image generators, is built on a long series of innovations dating back eighty years. Understanding how these systems work is crucial for assessing their capabilities and limitations.

Harms and Misuses. Generative AI presents various harms, including:

• Software that claims to detect AI-generated essays doesn't work, leading to false accusations of cheating.
• Image generators are putting stock photographers out of jobs even as AI companies use their work without compensation to build the technology.
• News websites have been caught publishing error-filled AI-generated stories on important topics such as financial advice.

The Power of Data. The success of generative AI depends on the availability of vast amounts of data, often scraped from the internet without consent or compensation to the creators. This raises ethical questions about the appropriation of creative labor and the potential for misuse.

5. Existential AI Risk: A Grounded Perspective

We don’t have to speculate about the future but can instead learn from history.

The Ladder of Generality. The fear that advanced AI systems will become uncontrollable rests on a binary notion of AI crossing a critical threshold of autonomy or superhuman intelligence. However, the history of AI reveals a gradual increase in flexibility and capability, which can be understood through the concept of a "ladder of generality."

Rogue AI? Claims of out-of-control AI rest on a series of flawed premises. A more grounded analysis shows that we already have the means to address risks concerning powerful AI calmly and collectively.

A Better Approach. Instead of focusing on hypothetical existential threats, we should prioritize defending against specific, real-world harms caused by AI, such as misuse by bad actors, bias, and labor exploitation.

6. Social Media's Content Moderation Conundrum: AI's Limited Role

The central question we examine is whether AI has the potential to remove harmful content such as hate speech from social media without curbing free expression, as tech companies have often promised.

The Promise and Peril of AI in Content Moderation. Social media platforms have long promised that AI can effectively remove harmful content, such as hate speech, without curbing free expression. However, the reality is far more complex.

Shortcomings of AI for Content Moderation. AI struggles with:

• Contextual understanding
• Cultural nuances
• Evolving language and tactics
• Adversarial manipulation
• Balancing free expression and safety

A Problem of Their Own Making. The problems with social media are inherent in their design and cannot be fixed by the whack-a-mole approach of content moderation. The focus on engagement and ad revenue incentivizes the amplification of harmful content, making it difficult to achieve a balance between free speech and safety.

7. The AI Hype Vortex: Unmasking the Sources of Misinformation

Every day we are bombarded with stories about purported AI breakthroughs.

The AI Hype Machine. Misinformation, misunderstanding, and mythology about AI persist because researchers, companies, and the media all contribute to it. Overhyped research misleads the public, while overhyped products lead to direct harm.

The Role of Researchers. Textbook errors in machine learning papers are shockingly common, especially when machine learning is used as an off-the-shelf tool by researchers not trained in computer science. Systematic reviews of published research in many areas have found that the majority of machine-learning-based research that was re-examined turned out to be flawed.

The Media's Contribution. The media fans the flames of AI hype by publishing stories about purported breakthroughs, often reworded press releases laundered as news. Many AI reporters practice what's called access journalism, relying on maintaining good relationships with AI companies so that they can get access to interview subjects and advance product releases.

8. Charting a New Course: Regulation, Responsibility, and the Future with AI

We must urgently figure out how to strengthen existing safety nets and develop new ones so that we can better absorb the shocks caused by rapid technological progress and reap its benefits.

Addressing the Demand for AI Snake Oil. AI snake oil is appealing because those buying it are in broken institutions and are desperate for a quick fix. We can't fix these problems by fixing AI. If anything, AI snake oil does us a favor by shining a spotlight on these underlying problems.

Regulation and Responsibility. Setting ground rules for companies to govern how they build and advertise their products is essential. Regulation has an important role here, while we acknowledge that regulation shouldn’t go overboard.

AI and the Future of Work. We must urgently figure out how to strengthen existing safety nets and develop new ones so that we can better absorb the shocks caused by rapid technological progress and reap its benefits.

Last updated: March 24, 2025