Optimal Illusions

1. Optimization's Promise and Paradox: Efficiency's Unintended Consequences

Optimization was invented, over time, with the idea of making a system, or a society, more perfect. And yet, with a world steeped in optimization, we often see the opposite happening.

The Allure of Perfection. Optimization, born from the desire to improve systems and societies, has paradoxically led to unforeseen negative consequences. While aiming for perfection, we often create imbalances and vulnerabilities, undermining the very goals we sought to achieve. The book opens with the construction of the Amazon Air Hub in Kentucky, a symbol of optimized logistics, juxtaposed against the hidden ruins of America, suggesting that progress often obscures what has been lost.

The Trade-offs of Efficiency. The pursuit of efficiency has come at the cost of slack (redundancy), place (local knowledge and connection), and scale (human-relatable proportions). These losses manifest in various ways, from fragile supply chains to the erosion of community bonds and a disconnect from the sources of our sustenance. The story of Bob, the North Dakota beet farmer, illustrates this point, as his resistance to GMOs highlights the trade-offs between yield and autonomy.

A Call for Reassessment. The book challenges the unquestioned dominance of optimization as a guiding principle, urging us to consider its limitations and the value of what has been sacrificed in its name. It sets the stage for exploring alternative approaches that prioritize resilience, connection, and a more holistic understanding of progress. The introduction serves as a roadmap for the journey ahead, promising to delve into the origins, successes, failures, and potential future of optimization.

2. The Rise of Optimization: Atomization, Abstraction, and Automation

When we become good at controlling nature with our ever more miraculous inventions, we start to assume we can solve anything.

Conceptual Shifts. The rise of optimization is rooted in three key conceptual shifts: atomization (breaking down reality into indivisible units), abstraction (representing the world through increasingly complex models), and automation (decoupling output from human-scale input). These shifts have transformed our understanding of the world and enabled unprecedented levels of efficiency. The story of Zappos, with its focus on metrics and customer service, exemplifies these shifts in action.

Mathematical and Infrastructural Foundations. These conceptual shifts have spurred the development of new mathematical techniques and infrastructure, from calculus and linear programming to production lines and server farms. These advancements have accelerated progress and solidified optimization as a dominant worldview. The work of Isaac Newton, with his focus on quantifiable laws of nature, laid the groundwork for this transformation.

Unintended Consequences. While these shifts have brought about immense growth and innovation, they have also led to a neglect of important considerations, such as the value of human connection and the long-term sustainability of resources. The World Series of Poker, with its reliance on probabilities and player modeling, illustrates the increasing abstraction and complexity of modern systems. The chapter sets the stage for exploring the cultural shifts that have further cemented optimization's dominance.

3. Cultural Shifts Cementing Optimization: Individualism, Perfectibility, and Codification

The man dwelling in a democracy . . . is aware of beings about him who are virtually similar; he cannot, therefore, think of any part of the human species without his thought expanding and widening to embrace the whole.

Individualism and Self-Sufficiency. A key cultural shift has been the rise of individualism and the belief in self-sufficiency, particularly in America. This has led to a focus on personal responsibility and the pursuit of individual goals, often framed in terms of efficiency and productivity. Marie Kondo's tidying philosophy, with its emphasis on individual action and control, reflects this cultural trend.

The Pursuit of a Perfect World. The belief that humanity has the power and duty to create a more perfect world on Earth has also fueled the rise of optimization. This has led to a focus on engineering solutions and a faith in the ability of technology to solve societal problems. The story of Stan Ulam and the Manhattan Project illustrates this drive to harness scientific knowledge for the betterment of society.

Codifying Individual Action. The codification of individual action in service of a better world has further cemented optimization as a dominant mindset. This has led to the development of systems and processes designed to measure and improve performance, often at the expense of other values. The chapter explores the religious undertones of optimization, suggesting that it has become a kind of secular gospel.

4. The Malaise of Modernity: Recognizing Optimization's Limits

Two ways. First gradually, then suddenly.

The Cracks in the Facade. Despite its promises, optimization has not delivered on its promise of a perfect society. Instead, we are faced with a growing sense of unease, a feeling that something is amiss. The Texas power grid failure serves as a stark example of the fragility of over-optimized systems.

The Gradual Buildup of Fragility. The Texas blackout was not an isolated event but the culmination of decades of decisions that prioritized efficiency over resilience. This gradual erosion of slack and redundancy created a system vulnerable to unforeseen shocks. The story of Allen Gilmer and the oil industry illustrates how the pursuit of optimization can lead to a disconnect from the material world.

The Sudden Breakdown. The Texas blackout, like many other crises, hit suddenly, exposing the underlying vulnerabilities that had been building for years. This sudden breakdown serves as a wake-up call, forcing us to confront the limits of optimization and the need for a more balanced approach. The chapter sets the stage for exploring alternative responses to the malaise of modernity.

5. Beyond Optimization: Reclaiming Slack, Place, and Scale

We must cultivate our garden.

The Need for a New Vision. In the face of optimization's failures, there is a growing need for a new vision of progress, one that goes beyond the narrow focus on efficiency and embraces a more holistic understanding of human well-being. This requires reclaiming the values that have been sacrificed in the pursuit of optimization: slack, place, and scale. The story of Candide, with his call to cultivate his garden, offers a metaphor for this shift in perspective.

Reclaiming Slack. Slack, or redundancy, is essential for resilience and adaptability. By building slack into our systems, we can better withstand unforeseen shocks and adapt to changing circumstances. This might involve diversifying supply chains, investing in backup power sources, or simply allowing for more downtime in our schedules.

Reclaiming Place. Place, or local knowledge and connection, is crucial for building strong communities and fostering a sense of belonging. This involves supporting local businesses, preserving cultural traditions, and engaging in civic life. The chapter sets the stage for exploring concrete strategies for reclaiming these values.

6. The Treachery of Optimals: Unwinding the Engineered World

Ceci n’est pas une pipe.

The Illusion of Control. Optimization often creates the illusion of control, leading us to believe that we can engineer our way to a perfect world. However, this belief can blind us to the unintended consequences of our actions and the inherent complexity of the systems we are trying to manage. The painting "The Treachery of Images" by René Magritte serves as a metaphor for this deception.

The Asymmetry of Optimization. While optimization is often a straightforward process, unwinding an over-optimized system is far more complex. There is no simple formula for reversing the effects of decades of decisions that prioritized efficiency over other values. The story of Jason Baldes and the bison restoration project illustrates the challenges of decolonizing land and restoring ecological balance.

The Need for Humility. Unwinding an engineered world requires humility and a recognition of the limits of our knowledge and control. It also requires a willingness to embrace uncertainty and to experiment with new approaches. The chapter sets the stage for exploring concrete strategies for navigating this complex terrain.

7. Reconciling Myth and Reality: Charting a Course Beyond Optimization

It is still an unending source of surprise for me to see how a few scribbles on a blackboard . . . could change the course of human affairs.

The Power and Peril of Mathematics. Mathematical models have played a crucial role in shaping our world, enabling us to understand and manipulate complex systems. However, these models are also abstractions, and they can lead us astray if we forget their limitations. The story of Stan Ulam, with his mixed feelings about the impact of his work, serves as a cautionary tale.

The Need for a New Narrative. To move beyond the limitations of optimization, we need a new narrative that embraces both the power of human ingenuity and the importance of ecological balance. This narrative must recognize the interconnectedness of all things and the need for a more holistic approach to progress. The chapter sets the stage for exploring concrete strategies for charting a course beyond optimization.

The Importance of Human Connection. The story of Allen Gilmer, with his appreciation for the "real" music of TLKF, highlights the importance of human connection and authenticity in a world increasingly dominated by technology. This connection is essential for building strong communities and fostering a sense of belonging. The chapter sets the stage for exploring concrete strategies for reclaiming human scale and understanding.

8. Islands of Knowing: Reclaiming Human Scale and Understanding

Nations are like travelers.

The Value of Local Knowledge. Islands, both literal and metaphorical, offer a space for cultivating local knowledge and developing a deeper understanding of the interconnectedness of things. By focusing on smaller, more manageable systems, we can gain a greater appreciation for the complexities of the world around us. The story of Nathan and Sage, with their island bakery, illustrates the value of local production and community connection.

The Tension Between Interconnection and Insularity. While islands offer a refuge from the homogenizing forces of optimization, they must also remain connected to the mainland. The key is to find a balance between interconnection and insularity, allowing for the exchange of ideas and resources while preserving local autonomy and diversity. The chapter concludes by emphasizing the need for a new way of seeing, one that embraces both the power of human ingenuity and the importance of ecological balance.

A Call to Action. The book ends with a call to action, urging readers to reclaim their agency and to create a more sustainable and fulfilling future. This requires a willingness to challenge the dominant narratives of optimization and to embrace a more holistic and human-centered approach to progress. The final image of the Amtrak train, traversing the continent and connecting diverse communities, serves as a reminder of the potential for both connection and disconnection in our increasingly interconnected world.

Last updated: April 15, 2025