Why Data Science Projects Fail

1. AI Hype Masks Complex Realities

More than anything, we want to counter the ADSAI hype with a dose of realism.

Erosion of trust. The field of AI, data science, and analytics is "crippling itself" with exaggerated promises and marketing hype, leading to an erosion of trust in data-driven decision-making. A stunning 80% (or more) of analytics projects continue to fail, costing enterprises and society hundreds of billions of dollars. This widespread failure is often due to false expectations and a lack of understanding of what truly goes into these complex projects.

Learning from failure. To fix this, we must embrace realism and learn from failures, which is an "extraordinarily effective form of feedback." Research shows individuals learn more from the failures of others than from others' successes. By illustrating the "harsh realities" of implementing AI and analytics through personal stories, we can gain insights into what truly makes a data science project successful and avoid common pitfalls.

The Epic Sepsis Model. The case of the Epic Sepsis Model (ESM) exemplifies this complexity. Despite initial claims of reducing sepsis mortality by one-fifth, independent studies revealed lower accuracy and significant "alert fatigue." This highlights how real-world deployment introduces challenges like:

• Data drift and changing "coding of illnesses"
• Lack of independent, peer-reviewed assessment
• The need for model calibration to specific patient populations

2. Strategic Misalignment Dooms Projects

If you try to tell someone else how to do their job better using sophisticated mathematics and computers without thoroughly understanding how they do their job today, including all of the problems and challenges they encounter, then you sir/madam are a fraud.

Lack of clarity. Many data science projects fail because business leaders and data scientists, individually and collectively, "don’t understand the real business problem at hand, and as a result, fail to develop a plausible business case." This often leads to a technically sound solution for the wrong problem, or one that doesn't align with core business priorities. Without a clear vision and strategy, projects become "solutions looking for a problem."

RetailCo's nightmare. The RetailCo personalization project illustrates strategic failure. Despite a clear objective to leverage data for customer relationships, the project unraveled due to:

• Scope creep: IT pushing for a holistic Customer Data Platform (CDP) instead of focusing on the immediate project needs.
• Decision-making by committee: Unqualified executives interfering with technical and resource allocation decisions.
• Lack of leadership buy-in: Executives not fully comprehending the effort required for an AI system in production, leading to under-resourcing and competing KPIs.

Economics in the driver's seat. Projects must be ruthlessly prioritized based on potential business value and economic impact. An AI system should only be adopted if its marginal benefit (additional profit, cost reduction, service improvement) is greater than its marginal cost (installation, operation, maintenance) compared to the next best alternative. This economic mindset helps avoid investing in "pipe dreams" or "curiosities" that lack a clear return, as seen in Mike the farmer's precision farming dilemma or WayBlazer's failure to monetize its innovative travel AI.

3. Data is the Unsung Hero (and Common Villain)

“Everyone wants to do the model work, not the data work.”

Undervalued foundation. Data quality, availability, and management are consistently the second most common reason data science projects fail, yet they are "the most undervalued and least exciting aspect" of AI. Google's research highlights that data-related failures are often avoidable but occur over 90% of the time, leading to insidious and delayed "failure cascades." Practitioners often prioritize model work for prestige and career advancement, neglecting the arduous, invisible work of data.

The data hunt. Often, the data presented by a client doesn't tell the whole story, requiring data scientists to "broaden the data aperture" and seek additional sources. Doug's healthcare HR project, investigating "no notice" terminations, initially hit a "brick wall" with internal data. The breakthrough came from integrating external macroeconomic data (mortgage defaults, foreclosures) to uncover the true cause. This detective work underscores the need for tenacity and perseverance in data exploration.

Bridging the gap. Data issues manifest in myriad ways:

• Lack of suitable data: Many companies lack automated, clean, or consistent data.
• Data silos: Data scattered across dozens of systems, databases, and spreadsheets.
• Data integration: The challenge of cleaning, organizing, and integrating data into a single accessible workspace.
• Data governance: Ensuring one version of the truth, consistent definitions, and usage patterns.
  These challenges necessitate collaboration between data scientists, IT, and data governance teams, focusing on value-based prioritization of data capture to avoid overwhelming "data tsunamis."

4. Unrealistic Expectations Undermine Success

“Under Promise, Over Deliver.”

The expectation continuum. Setting realistic expectations for project scope, timing, resources, budget, and business value is crucial. Over-promising and under-delivering ("epic fail") can be "politically irredeemable and career jeopardizing," while "sandbagging" (under-promising and over-delivering too much) can erode customer confidence. The ideal is a "Target Zone" that balances optimism and pessimism, with ambitious but achievable "Big Hairy Audacious Goals" (BHAGs) as stretch targets.

Beyond the hype. Executives often demand unreasonable delivery timelines, fueled by AI hype. Data scientists must temper these expectations, recognizing that building models is relatively easy, but transforming them into "enterprise-grade production systems is difficult, can take years, and cost tens of millions of dollars." The true value of AI often lies in augmentation, where humans interact iteratively with models, rather than full automation, which is rarely a realistic short-term goal for complex decision-making.

Finding true value. To set realistic targets, organizations should:

• Analyze financial statements: Understand the firm's economic performance.
• Identify key business areas: Focus on labor, inventory, assets, manufacturing, pricing, or revenue management where significant economic inefficiencies exist.
• Quantify maximum potential benefit: Then rigorously estimate a realistic 5%-25% of that maximum, as real-world factors always reduce the ideal.
• Prioritize impactful problems: Target problems currently solved manually with rules of thumb, where AI can provide significant leverage.

5. Communication Bridges the Technical-Business Divide

A manager would rather live with a problem they cannot solve than accept a solution they cannot understand.

Language barrier. Clear, concise communication is a significant challenge in data science because business people and data scientists "rarely speak the same 'language.'" Data scientists speak in models and code, while managers speak in KPIs and business jargon. This disconnect can lead to mistrust and rejection of solutions, even if technically sound.

Effective communication strategies:

• Seek first to understand: Data scientists must listen two-thirds of the time, asking exploratory and clarifying questions, especially in early project stages.
• Tailor the message: Adapt communication style and content to the audience (e.g., executives need business value, not technical jargon).
• Storytelling: Frame the project by describing "what life was like before the model was developed and implemented, and how life changed (hopefully for the better) afterwards."
• "What's in it for me?": Clearly articulate the benefits for individuals, teams, and the company (cost savings, revenue increase, improved satisfaction).
• Avoid email for critical topics: Face-to-face or video calls are essential for nuanced, complex information.

Breaking down silos. Poor communication and organizational silos can derail projects, even with capable teams. The credit default risk project failed partly because the data science team didn't adequately engage the existing actuarial team or collaborate effectively with IT. This led to resistance, delays, and ultimately, the abandonment of a promising solution. Effective communication fosters mutual understanding and agreement, preventing projects from becoming "pet projects" or falling victim to internal politics.

6. People Skills Trump Technical Prowess

All history is biography. Business, like history, is all about people.

The human factor. Most data science failures stem not from technical issues, but from a need for greater "soft skills" and understanding of human factors. While technical acumen is a mandatory prerequisite, emotional intelligence (EQ) and interpersonal skills are equally, if not more, critical for success. The story of ReliableCo, an analytically mature company, highlights how an insular culture and leadership's failure to recognize and support its award-winning Advanced Analytics team led to missed opportunities and significant business disruptions.

The Analytics Translator. A key factor separating successful AI adopters from failures is the "Analytics translator"—an individual with both domain knowledge and analytical skills who can effectively communicate between business and AI teams. These translators are crucial for:

• Troubleshooting inevitable AI engine problems.
• Interacting with vendors on requirements.
• Ensuring consistent best practices and knowledge sharing.
  Universities are beginning to recognize this need, offering programs that blend technical skills with business acumen, but a significant gap remains in providing "real-world data literacy" and soft skills training.

Analytically driven leadership. Visionary leadership, like that of Robert Crandall at American Airlines, is essential. Crandall's foresight and humility led him to invest heavily in Operations Research, transforming AA into a technologically advanced airline. He surrounded himself with experts, fostering a data-driven culture that optimized complex operations and generated billions in value. This demonstrates that true leadership involves:

• Recognizing one's own limitations and hiring experts.
• Having the "mental acuity (brains) and fortitude (guts)" to invest strategically.
• Empowering ADSAI leaders to execute while holding them accountable.

7. Technology is a Means, Not an End

A model is a means to an end, not an end itself.

Misapplication of models. A common technical failure is "misapplying a model," either by incorrectly implementing the right model or using the wrong model altogether. This often stems from a lack of detailed understanding of the business problem's nuances and the principles of experimental design. For example, using a simple A/B test to measure revenue (a complex, multivariate quantity) instead of a hit rate, or applying logistic regression to predict a percentage (On-Time Performance) rather than a probability.

Overemphasis on tech. Data scientists, especially recent graduates, can become "excessively focused and too infatuated with the model and the mathematics, algorithms, and technology." This "technology for technology's sake" approach often ignores the Pareto principle (80/20 rule), where achieving 80% of the business value with 20% of the effort is often sufficient. The airline carry-on baggage computer vision project, while technically sound, failed because the "incremental business value and operational and/or economic impact over and above the status quo incumbent solution" did not justify the significant investment.

Business value first. In business, mathematics and models are a means to an end: "contribute to the betterment of a corporation’s economic and financial performance." Executives care about "recurring physical dollars in the bank," not arcane technical details. Status reports should lead with business value, not technical jargon. While impressive AI technologies exist (robotics, computer vision, NLP), their application must be driven by a strong business case that demonstrates substantial improvement over existing solutions.

8. Production Deployment is a Different Beast

Building models is easy, you can do that in a day... Building a model into an enterprise-grade production system is difficult, can take years, and cost tens of millions of dollars...

Exponential complexity. The transition from a "sandbox model" to an "enterprise-grade production system" is exponentially more complex, often requiring 10x to 100x more effort and resources than the pilot. Key factors influencing this complexity include:

• Dynamism: Static data vs. real-time streaming data.
• Integration: Stand-alone vs. heavy reliance on numerous other systems.
• Mission Criticality: Low inconvenience vs. entire company grinding to a halt.
• Problem/Model Complexity: Simple heuristics vs. sophisticated algorithms.
  Projects like UPS's ORION (delivery routing) and American Airlines' DINAMO (yield management) took years and hundreds of millions to build due to these factors, but delivered billions in value.

The FastFoodCo debacle. Even analytically mature organizations stumble, as seen with FastFoodCo. A successful pilot model failed in production because:

• Feature mismatch: Separate teams built batch and real-time data pipelines, leading to subtly different feature definitions.
• Communication gap: Lack of direct, regular communication between the two internal data teams.
• Resource reassignment: The real-time data team moved to other projects before the issue was discovered, leaving the consultants scrambling.
  This highlights that technical problems often have deeper "people problems" at their root, even in capable organizations.

Robust infrastructure. Successful deployment requires a comprehensive approach:

• Data Organization: Data Engineering and Governance for timely, high-integrity data pipelines.
• Technology Organization: Software engineering for interfaces, cloud services for compute/storage, and test/QA for validation.
• Change Management: Orchestrating the transition for business users from old to new processes.
• Reusability: Building models as microservices with APIs for integration into larger ecosystems.
  This holistic view ensures that the model, once deployed, is reliable, performant, and truly delivers value.

9. Even Mature Organizations Stumble

The outcomes for companies undergoing one or two major projects may be obvious and an A/B test may not be needed; however, monitoring is still necessary and important for all tools.

Subtle pitfalls. While analytically mature organizations generally avoid the foundational failures of their immature counterparts, they still face a significant 40% failure rate. These failures often stem from more nuanced issues that can trip up even the best teams. One top reason for mature organizations is a "lack of the right resources," not in terms of skill, but in volume, as high demand stretches capable teams across too many projects.

Beyond the model. Booking.com's lessons from 150 ML models highlight that even after extensive testing, "offline model performance is [not always] a reliable indicator of online (or deployed) model performance." Key insights for mature organizations include:

• Experimental design: Rigorous A/B testing is crucial to unequivocally prove the benefit of new technology in a live environment.
• Continuous monitoring: Beyond outputs (model drift), continuously monitor input data (data drift) to detect issues like new circumstances (e.g., COVID-19) that deviate from training data, preventing erroneous predictions.

Outside influences. Projects in mature organizations can be derailed by factors outside the analytics team's control:

• Stakeholder turnover: Key project champions leaving can jeopardize continuity.
• Changing priorities: High-level strategic shifts can deprioritize ongoing projects.
• Internal politics: Silos and competing agendas can undermine cross-functional collaboration.
• Communication gaps: As seen in the FastFoodCo case, even subtle miscommunications between specialized internal teams can lead to critical errors in data pipelines.
  These challenges underscore the need for robust organizational processes and strong interpersonal skills to navigate complex corporate environments.

10. Embrace Learning from Failure with Humility

Yes, excessive automation at Tesla was a mistake. To be precise, my mistake. Humans are underrated.

The virtue of humility. Humility is a critical virtue for ADSAI success, enabling learning, growth, and effective teamwork. As Patrick Lencioni notes, humility, hunger, and social intelligence are key for ideal team players. Humble leadership has been shown to improve follower and team performance. A lack of humility can lead to complacency, causing teams to skip crucial steps or inflate their role, as seen in Tesla's "excessive automation" mistake.

Empathy and managing expectations. Data scientists must cultivate empathy for end-users, understanding their comfort zones, KPIs, and what truly matters to them. Projects can be "sabotaged by the demands of the end users to 'stay within brand guidelines,'" even if it reduces the optimal ROI. The Max Kelsen SAVI project, developing computer vision for surgical kits, stumbled when a commercial team member over-promised "detect other things" functionality without understanding its technical complexity, leading to scope creep and a rushed, less scalable solution.

The journey to perfection. Achieving a 0% failure rate in ADSAI projects is unrealistic and potentially counterproductive, as it might indicate a lack of risk-taking and learning. As Jeff Bezos famously said, "I didn't think I'd regret trying and failing, and I suspected I would always be haunted by a decision to not try at all." ADSAI projects are inherently complex, involving human factors, organizational dynamics, and external influences that cannot be fully controlled. The goal is not perfection, but continuous improvement through learning from both successes and failures, fostering a culture where "To Analyze is Human."