The Loop

1. We are on a "Generation Ship," unknowingly reprogramming our future with technology.

The Loop is a downward tailspin of shrinking choices, supercharged by capitalist efficiency, in which human agency is under threat from irresistible systems packaged for our unconscious acceptance.

Humanity's journey. The author introduces the metaphor of a "generation ship" to describe humanity's trajectory, suggesting that our current technological advancements are fundamentally reshaping human behavior for generations to come. We are no longer just passengers but are now at the helm, building systems that will alter our progeny's actions without their conscious awareness. This process forms "The Loop," a cycle of behavioral and technological conditioning.

Three interconnected loops. The book outlines three loops that govern our existence. The innermost loop comprises our inherited, unconscious human behaviors, such as biases and autopilot functions. The second loop involves modern forces like consumer technology, capitalism, and politics, which sample and reflect these innate patterns back to us. The outermost loop, "The Loop" itself, represents a future where automated pattern-recognition systems (AI) study, amplify, and sell back these tendencies, leading to a collapse of individual choice.

Long-term consequences. The most profound and lasting effects of this technological reprogramming will unfold over at least a generation. Without long-term sensibilities in our policies and programming, we risk becoming a species that is distracted, obedient, and helpless to resist technologies that make choices for us, even when those choices are detrimental. The challenge lies in recognizing and correcting this trajectory before it's too late.

2. Our brains construct a simplified reality, making us vulnerable to manipulation.

Reality is the simplest story our brains can cobble together from an overwhelming flood of raw sensory information.

The reality gap. Our perception of reality is not a direct, unfiltered stream of information but a curated, simplified story assembled by our brains. Pioneering work by Otto Pötzl on "delayed piecemeal delivery into consciousness" and later research on "vision for perception and vision for action" (Goodale and Milner) demonstrates that much of what we experience is unconsciously processed and ordered. This "abstracting process" prevents us from being overwhelmed by sensory chaos.

Unconscious processing. Studies reveal that our senses are constantly receiving and assembling information outside of conscious awareness.

• Vision: Our hands can react accurately to illusions that fool our conscious eyes.
• Memory: Time perception can "slow down" in emergencies due to richer memory formation, not actual temporal distortion (David Eagleman).
• Emotion: "Blindsight" patients can unconsciously perceive and mirror emotions, suggesting an evolutionary pathway for rapid social communication (Beatrice de Gelder).

Fertile ground for influence. This mediating layer of curated reality makes us highly susceptible to external influences. Our brains are built to accept what they are told, especially if it conforms to expectations and saves mental effort. This fundamental vulnerability creates an enormous opportunity for those seeking to take advantage of and direct human behavior.

3. Two mental systems drive our choices: one fast and instinctive, the other slow and rational.

System 1, they wrote, “is characterized as automatic, largely unconscious, and relatively undemanding of computational capacity,” whereas System 2 handles “controlled processing” and “analytic intelligence.”

System 1 vs. System 2. Psychologists Keith Stanovich and Richard West formalized the concept of dual-process theories, identifying System 1 as our automatic, largely unconscious, and efficient decision-making system, and System 2 as our controlled, analytical, and energy-intensive system. System 1 handles most daily tasks, like climbing a ladder, while System 2 is reserved for complex, deliberate thought.

Efficiency and its costs. Our brains are "shortcut machines," constantly seeking to offload difficult cognitive tasks to conserve energy.

• Energy consumption: The brain, despite being 2% of body weight, consumes over 20% of our energy.
• Stress effects: Under stress, System 1 excels at instinctive tasks, while System 2's abilities (attention, memory, problem-solving) are impaired (Yerkes-Dodson law).
• Thin slice judgments: We make quick, often acceptable, judgments about people and situations without deep analysis, even when accuracy is not guaranteed (Ambady, Todorov).

System 2's limitations. While System 2 is meant to be a watchful error-correction system, it is easily distracted, tired, or simply fails to recognize when System 1 is making a mistake. Often, System 2 merely rationalizes the instinctive choices made by System 1, leading us to believe we've made a well-considered decision when we haven't. This reliance on an ancient, survival-oriented system for complex modern choices creates significant vulnerabilities.

4. Our ancient tribal instincts and biases are easily amplified by modern technology.

Our natural state is terrible, horrible. You’d never want to go back to that.

The "ugliness" of our nature. Contrary to romanticized notions of ancient life, our evolutionary programming instilled powerful, often undesirable, biases. Mahzarin Banaji's Implicit Association Test (IAT) reveals deep-seated, unconscious biases related to race, gender, and other categories that persist despite conscious efforts to be fair-minded. These biases, while once adaptive for survival in small, insular groups, are detrimental in our diverse modern world.

The power of group identity. Humans are predisposed to tribalism, quickly favoring in-group members over outsiders, even based on arbitrary distinctions (Henri Tajfel's minimal group paradigm).

• Childhood biases: Yarrow Dunham's experiments show children forming strong in-group/out-group biases within minutes, reinforcing them with "evidence."
• Sacred values: Research by Artis International (Wilson, Atran, Hamid) on "devoted actors" (e.g., fighters) reveals that abstract, non-negotiable "sacred values" trigger instinctive, rule-based brain systems, overriding rational cost-benefit analysis.

Technological amplification. Social media and online platforms exploit these ancient instincts. Mak Kapetanovic's story of radicalization illustrates how algorithms can funnel individuals toward extreme content, fostering powerful group identities and "sacred values" that override critical thinking. This process, often camouflaged as objective inquiry, empowers primitive instincts with disastrous effects, as seen in events like the January 6th Capitol riot.

5. Businesses exploit our unconscious tendencies, creating addictive "guidance systems."

Our experiential system is extremely sophisticated, and accurate. It’s extraordinary. When you think about human societies and cultures interacting in complex ways with technology and weapons and the rest, and the fact that most of the time we’re doing this with our experiential system, and it basically works pretty well.

The Autopia effect. We often feel in control while being guided by external systems, much like driving a car on a track with a central spine. This illusion of free will is a powerful tool for manipulation. Robert Cialdini's six principles of persuasion (reciprocity, commitment, social proof, authority, liking, scarcity) demonstrate how marketers leverage these psychological triggers to influence behavior.

Exploiting cognitive frailties. Businesses capitalize on our inherent cognitive shortcomings:

• Risk perception: We consistently underestimate risks (e.g., smoking) and are terrible at grasping probabilities, making us susceptible to "dark nudges" (Richard Thaler).
• Emotional decision-making: The "affect heuristic" shows emotions guide decisions, overriding rational analysis (Paul Slovic). Marketing often focuses on positive emotional associations to sell products.
• Habit formation: Our brains outsource decision-making to environmental cues, creating "rip currents" of habit that are hard to resist (Wendy Wood).

The "machine zone" and addiction. Technologies like social casino apps are designed to create a "machine zone" – an immersive state where time and money dissolve, fostering addiction not to winning, but to the escape itself (Natasha Dow Schüll). Companies like Dopamine Labs (now Boundless Mind) explicitly sell "behavior design" to make apps addictive, often rationalizing that only individuals with "deep-seated pain" are vulnerable. This industry profits by identifying and exploiting "whales" – users most susceptible to compulsive spending.

6. Artificial intelligence operates as an opaque "black box," perpetuating hidden biases.

Modern machine learning gives us answers but doesn’t show its work.

The black box problem. AI, while powerful at pattern recognition, often operates as an opaque "black box," providing answers without revealing the complex, improvisational computational process behind them. This lack of "explainability" makes it difficult to understand how decisions are reached, even for the specialists who build the systems. This opacity is often intentional, driven by desires for efficiency, competitive advantage, and avoidance of legal scrutiny.

Objective functions and ruthless efficiency. AI systems are designed with a specific "objective function" – a human-defined goal they pursue with ruthless efficiency. However, this efficiency can lead to unforeseen consequences:

• Simplistic correlations: AI may latch onto "random correlations" (e.g., swimming pool ownership predicting lower health risk) without understanding underlying causality.
• Reused algorithms: For cost reasons, companies often adapt existing algorithms, built for one objective function, to entirely different tasks, potentially porting over algorithmic habits and biases.
• "Aether" illusion: We naively assume AI measures against a simple, universal standard, much like physicists once believed in an "aether," when reality is far more complex and values are not universal.

The challenge of human values. Unlike the "Cold War rationalists" who naively sought to reduce geopolitical complexity to predictable patterns, modern AI faces the challenge that human values are not logical or coherent. As John Patty and Elizabeth Penn argue, groups often cannot achieve coherent collective preferences; legitimacy comes from explaining decisions, not universal consensus. AI's inability to grasp these nuances means it can easily perpetuate and amplify existing societal biases, rather than correct them.

7. AI-driven curation is narrowing our choices in culture, justice, and daily life.

The Loop is growing and growing.

Cultural curation. AI is increasingly curating our cultural consumption, from books and music to art. Platforms like Wattpad use machine learning to identify successful stories by analyzing audience reactions and content patterns, leading to publishing and film deals. While this can diversify content, it risks creating a "survivorship bias," where future creations are merely minor variations of past hits that statistically resonate with audiences, leading to "invasive mediocrity" (e.g., TikTok influencers).

The "performativity" problem. AI's predictions don't just reflect reality; they actively shape it. This "performativity" means that when AI evaluates what art we'll enjoy, what jobs we'll do well, or what crimes will occur, its predictions influence those outcomes.

• Journalism: Brand-safety algorithms, by blacklisting keywords like "crime" or "violence," inadvertently defund critical journalism, leading to a less informed public.
• Art creation: AI is now generating music and graphic novels, raising questions about artistic originality and whether audiences will distinguish human creativity from algorithmic imitation.

Reinforcing inequality. AI's pattern recognition, when applied to social data, can reinforce existing systemic biases. Jesus Hernandez's work on Sacramento housing shows how historical redlining patterns, when fed into AI, can predict and perpetuate inequality in home loans and life outcomes, absolving humans of moral responsibility. This creates a feedback loop where past discrimination dictates future opportunities.

8. Automated systems are making critical decisions in war and policing, reducing human agency.

We have engineered the human frailty out of this sort of automated system. But how are we going to engineer human morality into it?

Automating conflict. Militaries worldwide are automating critical functions, from fighter jets to missile defense, because human frailty (e.g., blacking out under G-forces) and slow reaction times are no longer sufficient for modern combat. Systems like Raytheon's SeaRAM and Israel's Iron Dome operate autonomously, making life-or-death decisions in milliseconds without human intervention. This fundamentally changes the role of humans in conflict, making war "logistically and morally and politically easier."

Predictive policing's feedback loop. AI is also transforming law enforcement. Predictive policing software like PredPol uses historical crime data to forecast where and when crimes are likely to occur.

• Bias amplification: Studies show PredPol reinforces existing biases, disproportionately targeting Black communities even when drug use is evenly distributed across races.
• Accountability shift: This "black-box machinery" lends unwarranted legitimacy to biased policing strategies, shifting accountability from human decision-makers to opaque algorithms.
• Reduced discretion: Officers are guided by software, potentially eroding their independent judgment and discretion, leading to a cycle where predictions of crime in certain areas lead to increased policing, which leads to more arrests, reinforcing the initial prediction.

The moral vacuum. The increasing reliance on AI in these critical domains raises profound ethical questions. While automation removes human weaknesses, it also removes human morality. The convenience of offloading difficult, morally fraught decisions to algorithms risks desensitizing us to the consequences of conflict and injustice, creating a future where we lose our clear sense of dealing with death and human rights.

9. "Weak perfection" prioritizes human values and deliberate thought over ruthless efficiency.

The legal process puts people through an intentionally slow decision-making system when they decide on their plea. It forces slow thinking. We call it ‘weak perfection,’ the idea that the system remains difficult to help them make better decisions.

Inefficiency as a safeguard. Not all systems should be optimized for ruthless efficiency. "Weak perfection" describes intentionally slow, difficult, or fragile systems that force humans to engage in mentally taxing System 2 processing, especially in critical domains like law. Forcing deliberate thought, even if inconvenient, can be a hidden human safeguard against rash or biased decisions.

AI's threat to due process. AI's drive for efficiency can undermine fundamental rights and values:

• Opaque decisions: Algorithms make life-changing decisions (e.g., credit, public benefits, housing) without transparency or direct consultation with affected individuals.
• Reinforcing bias: Pretrial risk assessment tools (RATs), intended to reduce bias, often channel and deepen existing structural racism by relying on historically biased data.
• Automated judgment: The California EDD's use of off-the-shelf AI to freeze 1.4 million unemployment claims based on "high risk" criteria demonstrates how automated systems can make sweeping, flawed judgments without human oversight or accountability.

Beyond data-driven decisions. Statistics and data, while powerful, do not always reflect who we are or want to be as a society. The decision to mandate backup cameras in all new vehicles, despite preventing a relatively small number of deaths, shows that human emotions and values can (and sometimes should) override purely data-driven arguments. We must prioritize these unquantifiable human considerations over AI's cold efficiency.

10. We must create new legal and ethical frameworks to harness AI for collective benefit.

If we are going to maintain our capacity to make these human choices, we have to incorporate the unquantifiable but unmistakable effects of The Loop into some sort of regulatory framework.

Harnessing AI for good. AI's pattern-recognition capabilities can be used beneficially, for example, in wildfire risk assessment (Zesty.ai) to encourage mitigation, or in analyzing art history (Metropolitan Museum) to uncover hidden trends. Similarly, AI could be used to identify patterns of inequality in policing (Andrew Papachristos) or the justice system (Donald Black's "relational distance" theory), acting as a mirror to reveal systemic biases rather than perpetuate them.

New regulatory models. Existing legal frameworks are inadequate for addressing the unique challenges posed by AI. We need new models that account for:

• "Public goods" problem: The Loop creates a "public goods" problem where individuals cannot easily "exit" or "voice" objections to pervasive AI systems (Albert O. Hirschman).
• "Weak perfection": Mandating intentionally inefficient processes in critical areas to preserve human deliberation.
• "Vaccine courts" for AI: Establishing no-fault compensation programs for harms caused by AI, similar to vaccine injury compensation, to encourage innovation while protecting individuals.

The Hand Rule for AI. Justice Learned Hand's rule (B < PL) provides a framework for assessing liability: the burden of precaution (B) should be less than the probability (P) of harm multiplied by the gravity of loss (L). This rule can be adapted to AI, recognizing that the probability and gravity of loss from unchecked AI (e.g., digital addiction, algorithmic bias) may far outweigh the burden on technology companies to implement precautions. Lawsuits against social casino platforms are beginning to prove that "real-world harm" from digital addiction can be quantified, forcing platforms to account for their role in enabling it.

Prioritizing human agency. Ultimately, we must decide what we want for our future and push AI to deliver that, rather than passively accepting its default, profit-driven trajectory. This requires updating our legal and ethical frameworks to recognize the unquantifiable but unmistakable effects of The Loop on human agency, ensuring that technology serves our values, not the other way around.