The AI Playbook

1. Machine Learning Deployment is About Business Transformation, Not Just Technology

Never sell AI. Instead, pitch operational improvements, with no more than a footnote to mention machine learning as part of the solution.

Reframe ML projects. Instead of focusing on the technology itself, successful ML initiatives should be framed as business transformation projects that use ML as a tool. This shift in perspective is crucial for gaining buy-in from stakeholders and ensuring that the project aligns with organizational goals.

Value proposition first. When pitching an ML project, lead with the business value proposition expressed in terms of key performance indicators (KPIs) such as increased revenue, reduced costs, or improved efficiency. Only after establishing the potential business impact should you introduce ML as the means to achieve these goals.

Example elevator pitch:

• Current problem: "99.5% of our direct mail is ineffective."
• Potential improvement: "Increasing response rate to 1.5% would mean $500,000 more revenue."
• Solution: "ML can target customers more likely to respond, tripling marketing ROI."

2. The Six Steps of BizML: A Framework for Successful ML Projects

BizML's six steps are universal—they always work, regardless of your org chart.

BizML framework. The book introduces a six-step framework called BizML for successfully deploying machine learning projects:

1. Value: Establish the deployment goal
2. Target: Establish the prediction goal
3. Performance: Establish the evaluation metrics
4. Fuel: Prepare the data
5. Algorithm: Train the model
6. Launch: Deploy the model

Iterative process. While these steps are presented sequentially, it's important to note that ML projects often require backtracking and iteration. Each step informs the others, and insights gained later in the process may necessitate revisiting earlier decisions.

Key principles of BizML:

• Deep collaboration with business stakeholders at every step
• Business professionals need semi-technical understanding of ML
• Focus on deployment and business value from the start

3. Data Preparation is the Most Critical and Time-Consuming Step

Data trumps the algorithm. Machine learning algorithms may be the fun, sexy part—everyone wants to crash that party—but improving the data is where you usually get the greatest payoff.

Data is king. While many focus on the excitement of advanced algorithms, the quality and preparation of data are often the most critical factors in an ML project's success. Data preparation typically consumes 80% of a project's technical efforts and is frequently underestimated.

Challenges in data prep. Preparing data for ML involves more than just organizing it into a table. It requires careful consideration of temporal aspects, derived variables, and potential biases or errors in the data. Common challenges include:

• Aligning input variables with the time of prediction
• Creating informative derived variables through feature engineering
• Ensuring data quality and representativeness
• Addressing class imbalance in the target variable

4. Model Performance Should Be Measured by Business Metrics, Not Just Accuracy

Accuracy is a blunt instrument. It's one thing to know a model is wrong, say, 12 percent of the time. That's the same as saying it is correct 88 percent of the time; that is, it's 88 percent accurate. But it's another thing, a much more helpful thing, to separately break down how often it's wrong for positive cases and how often it's wrong for negative cases.

Beyond accuracy. While accuracy is a commonly reported metric, it can be misleading, especially for imbalanced datasets. More meaningful metrics include lift, which measures how much better the model performs compared to random guessing, and business-specific KPIs.

Translating to business impact. The true value of an ML model lies in its ability to improve business outcomes. This requires translating model performance metrics into tangible business impacts:

• Example: Fraud detection
  • Model performance: Lift of 300 for top 0.2% of transactions
  • Business impact: $16 million annual cost savings

Key considerations:

• False positive vs. false negative costs
• Operational constraints and thresholds
• Alignment with business objectives

5. Deployment Requires Cross-Functional Collaboration and Change Management

Large-scale change requires advancing an inspirational vision, building relationship capital, and maintaining organizational alignment . . . leadership simultaneously embraces unifying and disruptive ideals.

Beyond technical challenges. Successful ML deployment often hinges more on organizational and human factors than on technical ones. It requires buy-in and collaboration across multiple departments and levels of the organization.

Change management strategies. To overcome resistance and ensure successful adoption:

• Involve stakeholders early and throughout the project
• Provide training and support for affected employees
• Use balanced scorecards to reward adoption of new processes
• Start with pilot deployments or A/B tests to demonstrate value
• Communicate the vision and benefits clearly to all levels of the organization

6. Real-Time Scoring Presents Challenges but Offers the Greatest Opportunities

The greatest opportunities are the hardest to tap.

High-stakes, high-reward. Real-time ML deployment, such as for fraud detection or ad targeting, often presents the greatest opportunities for business impact. However, it also comes with increased complexity and risk.

Challenges and solutions. Key considerations for real-time deployment include:

• Speed requirements: Models often need to score in milliseconds

• Data pipeline optimization: Ensuring input data is available quickly

• Infrastructure: Choosing between cloud, on-premises, or hybrid solutions

• Risk mitigation: Using control groups and gradual rollouts

• Example: FICO Falcon fraud detection

  • Scores transactions in 10-30 milliseconds
  • Processes billions of transactions globally
  • Achieves real-time fraud prevention at scale

7. Ethical Considerations are Paramount in ML Deployment

Algorithmic bias. And when one race, ethnicity, or other protected group more often experiences injustice by way of a model—that is, when the model commits FPs more for one group than another—it's called algorithmic bias.

Ethical imperative. As ML systems increasingly influence important decisions, it's crucial to consider their ethical implications and potential for bias or unintended consequences.

Key ethical considerations:

• Fairness: Ensuring models don't discriminate against protected groups
• Transparency: Making model decisions interpretable and explainable
• Privacy: Protecting individual data used in model training and deployment
• Accountability: Establishing clear responsibility for model outcomes

Strategies for addressing ethical concerns:

• Diverse teams in model development and review
• Regular audits for bias and unintended consequences
• Clear documentation of model limitations and assumptions
• Ongoing monitoring and adjustment of deployed models

Last updated: July 30, 2024