The Site Reliability Workbook

1. SLOs are the Compass for Reliability Decisions

Once you’re equipped with a few guidelines, setting up initial SLOs and a process for refining them can be straightforward.

SLOs guide priorities. Service Level Objectives (SLOs) are fundamental to SRE because they provide a data-driven framework for deciding where to invest limited engineering resources. Instead of aiming for an impossible 100% reliability, SLOs set realistic targets based on user needs and business goals, allowing teams to balance feature development with reliability work. The error budget, derived from the SLO, quantifies acceptable downtime or performance degradation, acting as a clear signal for when reliability must take precedence over new features.

Start simple, iterate often. Implementing SLOs doesn't require perfection from day one. Begin by identifying a few key Service Level Indicators (SLIs) that reflect critical user journeys, such as availability or latency, and measure them. Use these measurements to set initial SLO targets, even if they are based on current performance. The most important step is to get stakeholders to agree on these targets and commit to using the error budget for decision-making.

SLOs empower teams. Well-defined SLOs and a clear error budget policy provide SRE and development teams with the objective data needed to push back on unrealistic demands or justify investing time in reliability projects. They transform subjective debates about "how reliable is reliable enough" into concrete discussions based on user impact and business value. This shared understanding fosters better collaboration and ensures that reliability work is prioritized appropriately.

2. Measure User Experience, Not Just System Metrics

Your Users, Not Your Monitoring, Decide Your Reliability

Focus on user happiness. The ultimate goal of SRE is to ensure user satisfaction by providing a reliable service. This means that the most important metrics are those that directly reflect the user's experience, not just internal system health indicators. While CPU usage or disk space are useful for debugging, they don't tell you if users can actually use your service effectively.

SLIs capture experience. Service Level Indicators (SLIs) should be chosen to measure aspects of the service that matter most to users. Common examples include:

• Availability (successful requests / total requests)
• Latency (requests faster than X ms / total requests)
• Correctness (correct results / total results)
• Freshness (data updated recently / total data)

Measure close to the user. To accurately capture user experience, measure SLIs as close to the user as possible. Client-side instrumentation or load balancer logs are often better sources than application server logs, as they include network effects and frontend issues. Regularly compare your SLI measurements with user feedback channels like support tickets or social media to ensure your metrics align with perceived reliability.

3. Ruthlessly Eliminate Toil Through Engineering

For SRE, any manual, structurally mandated operational task is abhorrent.

Toil hinders progress. Toil is defined as manual, repetitive, automatable, tactical work that lacks enduring value and scales at least as fast as the service it supports. While some operational work is necessary, excessive toil prevents SREs from doing the engineering work required to improve systems and reduce future toil. Google's 50% operational work cap (including toil) is a mechanism to ensure time for strategic projects.

Identify, measure, automate. The first step to eliminating toil is to identify what constitutes toil for your team and measure the time spent on it. This provides objective data to prioritize automation efforts based on potential time savings and return on investment. Don't just automate the task; engineer the toil out of the system by fixing the root cause that necessitates the manual work.

Strategies for toil reduction:

• Reject toil: Analyze the cost of doing the work versus not doing it.
• Automate response: Build tools to handle repetitive tasks programmatically.
• Provide self-service: Empower users to perform tasks themselves via APIs or UIs.
• Increase uniformity: Standardize systems and processes to make automation easier.
• Use SLOs: Let error budgets guide when manual intervention is necessary.

Eliminating toil is a continuous process that requires support from management and a culture that values automation as a feature.

4. Design for Simplicity to Enhance Reliability

A complex system that works is invariably found to have evolved from a simple system that worked.

Simplicity reduces failure. Simple systems are inherently more reliable because they have fewer components, fewer interactions, and are easier to understand, maintain, and debug. Complexity, on the other hand, introduces more potential failure modes and makes incidents harder to resolve.

Simplicity is end-to-end. Strive for simplicity not just in code, but also in system architecture, dependencies, configuration, and operational processes. SREs are uniquely positioned to champion end-to-end simplicity due to their holistic view of the system in production. Encourage SREs to participate in design reviews early to identify and mitigate complexity risks.

Strategies for regaining simplicity:

• Remove unnecessary components or features.
• Standardize technologies and processes across the organization.
• Refactor complex parts of the system incrementally.
• Prioritize simplification projects and celebrate code removal.
• Diagram the system to identify complex interactions like amplification or cyclic dependencies.

Complexity is an externality; its cost is often borne by those who operate the system, not those who introduce it. Actively fighting complexity is crucial for long-term system health and sustainability.

5. Master Incident Response and Learn from Every Failure

Everyone wants their services to run smoothly all the time, but we live in an imperfect world in which outages do occur.

Structure reduces chaos. Incidents are inevitable. Having a well-defined incident response process, often based on frameworks like the Incident Command System (ICS), is crucial for coordinating efforts, communicating effectively, and maintaining control during a crisis. Clear roles (Incident Commander, Communications Lead, Operations Lead) and communication channels reduce confusion.

Prioritize mitigation. During an incident, the primary goal is to stop the impact on users as quickly as possible (mitigation), even if the root cause isn't fully understood. Generic mitigation tools (like rollbacks or draining traffic) should be prepared beforehand. Root cause analysis and permanent fixes happen after the incident is resolved.

Postmortems drive learning. Every incident, regardless of size, is an opportunity to learn. A blameless postmortem culture is essential for fostering trust and ensuring that teams identify systemic issues rather than blaming individuals. Good postmortems are:

• Factual and objective
• Detailed with quantifiable impact
• Include concrete, prioritized, owned action items
• Shared widely for organizational learning

Regular incident response training and drills build muscle memory and prepare teams for real emergencies, reducing Mean Time To Respond (MTTR) and Mean Time To Detect (MTTD).

6. Automate Changes and Rollouts Safely (Canarying)

Canarying is a partial and time-limited deployment of a change in a service and its evaluation.

Change is the primary risk. While necessary for progress, changes (code, config, data) are the most common trigger for incidents. Automating the release process (CI/CD) is the first step, ensuring reproducible, tested builds and automated deployments. However, testing environments can't perfectly replicate production.

Canarying mitigates risk. Canarying exposes a small subset of production traffic to a new change and evaluates its impact before a full rollout. This allows detection of defects in a controlled environment, minimizing the blast radius and conserving error budget. The size and duration of the canary should be representative of traffic patterns and allow sufficient time for metrics to stabilize.

Evaluate relevant metrics. Canary evaluation relies on comparing metrics from the canary population to a control group. Choose metrics that are strong indicators of user-perceivable problems (like SLIs) and are attributable to the change being tested. Avoid metrics easily influenced by external factors or those that don't clearly signal user impact.

• HTTP return codes (excluding client errors)
• Latency percentiles
• Application-specific correctness checks

Integrate canary evaluation into your automated release pipeline, allowing for automatic rollback if the canary fails.

7. Manage Load Holistically for Scalable Systems

No service is 100% available 100% of the time: clients can be inconsiderate, demand can grow fifty-fold, a service might crash in response to a traffic spike, or an anchor might pull up a transatlantic cable.

Load management is multifaceted. Ensuring a service remains available and performant under varying and unexpected load requires a combination of strategies, not just one tool. Load balancing, autoscaling, and load shedding are key components that must work together harmoniously. Misconfiguring their interactions can lead to cascading failures.

Load balancing directs traffic. Systems like Google Cloud Load Balancing (GCLB) use techniques like anycast and sophisticated routing (Maglev, GFE) to direct user requests to the nearest healthy backend with available capacity. This minimizes latency and routes around failures transparently to the user.

Autoscaling adjusts capacity. Autoscaling dynamically increases or decreases the number of instances based on load metrics (like CPU utilization or requests per second). This optimizes resource usage and helps absorb traffic spikes. Proper configuration requires setting limits, handling unhealthy instances, and considering the impact on downstream dependencies.

Load shedding protects from overload. When systems are pushed beyond their capacity, load shedding allows them to gracefully reject excess traffic rather than crashing entirely. This protects the system's core functionality for the users it can still serve. It's crucial that load shedding signals (like error responses) are correctly interpreted by load balancers and autoscalers to avoid perverse outcomes.

8. Configuration Design Matters for Operational Health

The quality of the human-computer interface of a system’s configuration impacts an organization’s ability to run that system reliably.

Configuration is a critical interface. Configuration allows rapid changes to system behavior without code deployments. Its design significantly impacts operational toil, reliability, and the ability to respond to incidents under pressure. Poorly designed configuration leads to errors, confusion, and wasted effort.

Separate philosophy and mechanics. Focus on the philosophy of configuration first:

• Configuration asks users questions; minimize mandatory questions.
• Questions should be close to user goals, not infrastructure details.
• Provide sensible defaults (static or dynamic) that work for most users.
• Allow "escape hatches" for power users to override defaults.

The mechanics (language, format, tooling) should support this philosophy. Separate the configuration language (how users write config) from the configuration data (the static representation the application consumes).

Tooling is essential. Good configuration systems provide tooling for:

• Semantic validation (checking if config makes sense).
• Syntax highlighting, linting, and auto-formatting.
• Versioning, ownership tracking, and change logging.

Apply configuration changes safely through gradual rollouts and ensure easy, reliable rollback capabilities.

9. Build Practical Systems with Non-Abstract Design

All systems will eventually have to run on real computers in real datacenters using real networks.

Design must be grounded. Non-Abstract Large System Design (NALSD) is an iterative process for designing large-scale distributed systems by constantly grounding abstract ideas in concrete reality. It forces designers to consider real-world constraints like hardware limits, network latency, and failure domains from the outset.

Iterative design process:

1. Is it possible? Design a system that works in principle, ignoring practical limits initially.
2. Can we do better? Optimize the basic design for efficiency.
3. Is it feasible? Scale the design considering real-world constraints (cost, hardware, etc.), potentially requiring a distributed architecture.
4. Is it resilient? Design for graceful degradation and resilience to component or datacenter failures.
5. Can we do better? Refine the scaled, resilient design.

Quantify resources early. At each step, estimate the required resources (CPU, RAM, disk, network) based on realistic assumptions about workload and component performance. This helps identify bottlenecks early and guides architectural decisions, preventing costly redesigns later. The process is more about the reasoning and assumption-making than the exact final numbers.

NALSD is a critical skill for SREs, enabling them to translate business requirements into practical, scalable, and reliable system architectures.

10. Prioritize Team Health and Combat Overload

When operational load outstrips a team’s ability to manage it, the team ends up in a state of operational overload (also called work overload).

Overload cripples productivity. Operational overload, whether real (too much work) or perceived (feeling overwhelmed), severely impacts team health, morale, and productivity. It leads to burnout, increased errors, and prevents teams from doing the strategic project work needed to improve systems and reduce future load.

Recognize the symptoms:

• Decreased team morale (complaints, frustration).
• Team members working long hours or when sick.
• Increased frequency of illness.
• Unhealthy task queues (backlogs, missed deadlines).
• Imbalanced metrics (long MTTR, high toil percentage).

Strategies for recovery:

• Identify and alleviate psychosocial stressors (lack of control, poor communication).
• Prioritize and triage workload ruthlessly, dropping non-essential tasks.
• Implement lightweight, regular triage processes to prevent backlog accumulation.
• Protect project time by limiting operational work to on-call shifts where possible.
• Invest in automation and root cause fixes to permanently reduce toil.

Empowering team members, fostering psychological safety, and ensuring transparent decision-making are crucial for restoring trust and enabling the team to work together effectively to manage their workload.

11. SRE is a Journey of Continuous Improvement and Collaboration

SRE is a journey as much as it is a discipline.

SRE is adaptable. The principles of SRE are applicable to organizations of any size and culture, not just Google. You can start implementing SRE practices (like SLOs) even without dedicated SRE staff. The journey involves maturing through stages, from initial adoption to building cohesive, potentially distributed teams.

Principles guide evolution:

1. SLOs with consequences: Drive decisions based on user-centric reliability targets.
2. Time for improvement: Ensure engineers have capacity for strategic project work.
3. Workload regulation: Empower teams to manage their operational burden.

Collaboration is key. SRE success relies heavily on strong partnerships with product development teams, product managers, and even external customers. This involves aligning goals, maintaining open communication, conducting joint reviews, and sharing knowledge. SREs act as champions for reliability throughout the service lifecycle.

Scaling requires structure. As the organization grows, structure SRE teams logically (by product or technology), standardize practices and tooling, and invest in training and knowledge sharing. Ending SRE engagements should be a deliberate decision based on value proposition, not just workload. The SRE journey is one of continuous learning, adaptation, and a relentless focus on improving the reliability of systems and the health of the teams that run them.

Last updated: May 12, 2025