How do you monitor, alert, and respond with SLOs and error budgets?

Long Answer

A comprehensive approach to monitoring, alerting, and incident response is central to the Site Reliability Engineer role. The philosophy of SRE emphasizes that reliability is a feature, just as important as functionality, and must be explicitly measured and enforced. The foundation is built on defining Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets, which then drive how monitoring, alerting, and on-call processes are structured.

1) Defining SLIs and SLOs

SLIs are quantitative measures that reflect user experience, such as request latency, error rate, throughput, and availability. For example, an e-commerce platform might track the percentage of checkout requests completing within 500 ms. SLOs are the agreed reliability targets for those SLIs, such as “99.9% of checkout requests under 500 ms over a rolling 30-day window.” By tying SLOs to user journeys, engineers avoid vanity metrics and instead focus on customer impact.

2) Error budgets and trade-offs

Error budgets quantify how much unreliability is acceptable before development velocity must slow down. For instance, if an SLO allows 0.1% error rate over a month, that tolerance is the error budget. If the budget is exhausted, feature launches are paused until reliability improves. This mechanism balances innovation with stability and makes discussions between product and engineering data-driven rather than subjective.

3) Monitoring and observability stack

Effective monitoring captures system health across infrastructure, application, and user experience layers. Metrics-based systems like Prometheus and Datadog provide quantitative insights, while tracing (OpenTelemetry, Jaeger) exposes distributed system bottlenecks, and logs (ELK, Splunk) provide detailed forensic data. Dashboards in Grafana or New Relic visualize SLIs and SLO compliance. Observability is broader than monitoring: it is about answering unknown questions with the data you have, enabling faster debugging of complex incidents.

4) Alerting philosophy

Alerting must be actionable, precise, and free of noise. SREs avoid triggering alerts on raw infrastructure metrics like CPU utilization alone, unless they directly correlate to degraded SLIs. Instead, they focus on user-facing impact: elevated error rates, breached latency targets, or availability drops. Alerts must be routed via paging systems (PagerDuty, Opsgenie) with clear severity levels. Runbooks should accompany every alert to reduce cognitive load during triage.

5) On-call rotations and escalation

On-call procedures require well-defined schedules, equitable rotation, and automated escalation paths. Typically, a primary engineer receives the first alert, with backup layers if they cannot respond. Escalation policies ensure that incidents do not stall. To reduce burnout, SRE teams balance frequency of pages, enforce “follow the sun” global coverage when possible, and conduct blameless postmortems to fix systemic issues rather than punishing individuals.

6) Incident response lifecycle

Incident response follows structured phases: detection, triage, mitigation, resolution, and postmortem. Detection comes from automated monitoring. Triage assesses severity and assigns ownership. Mitigation applies stopgap fixes (e.g., feature flag rollback, scaling resources). Resolution involves permanent remediation. Postmortems analyze root causes, prevent recurrence, and track systemic improvements. A culture of blamelessness encourages open learning.

7) Case studies and industry scenarios

In fintech, strict SLIs around transaction latency and accuracy are critical, as regulatory compliance depends on them. In SaaS, uptime and feature responsiveness drive customer satisfaction and churn. In e-commerce, high availability during peak events (Black Friday) requires predictive alerting and well-drilled on-call readiness. Across industries, error budgets provide a governance mechanism to pause feature rollout if reliability risk rises.

By combining clearly defined SLOs with robust monitoring, precise alerting, and resilient on-call practices, SREs enable systems to scale reliably while maintaining developer agility. This alignment of engineering practices with business outcomes is what makes Site Reliability Engineering a cornerstone discipline in modern operations.

‍

Table

‍

Common Mistakes

• Defining SLIs on infrastructure metrics instead of user experience (e.g., CPU vs checkout latency).
  
• Creating too many alerts, leading to alert fatigue and ignored pages.
  
• Failing to define or enforce error budgets, leaving reliability as an afterthought.
  
• Relying on static dashboards instead of building an observability culture.
  
• Not rotating on-call fairly, causing burnout and resentment.
  
• Skipping blameless postmortems, leading to repeat incidents.
  
• Escalating manually without automation, increasing response delays.
  
• Ignoring global time zones in scheduling, leading to uneven workloads.
  
  

Sample Answers

Junior:
“I would set up SLIs like error rate and latency, then monitor them with Prometheus and Grafana. Alerts should trigger only when an SLO is breached. For incidents, I would follow runbooks and escalate if I cannot resolve it.”

Mid:
“I define SLIs tied to user journeys and set clear SLOs. I enforce error budgets so reliability issues stop feature releases if needed. Monitoring spans metrics, logs, and traces. Alerts are routed via PagerDuty with runbooks. On-call schedules rotate fairly, and I contribute to postmortems.”

Senior:
“I build an observability platform integrating metrics, logs, and distributed tracing. SLIs map to business goals, SLOs are agreed with product, and error budgets govern release velocity. Alerts are actionable, tied to customer impact, and routed via Opsgenie. I establish global on-call coverage, blameless postmortems, and systemic reliability improvements. Reliability is treated as a feature.”

‍

Evaluation Criteria

Interviewers look for structured thinking that covers SLIs, SLOs, error budgets, monitoring, alerting, and on-call. Strong answers emphasize user-centric SLIs (latency, error rate) over infrastructure vanity metrics. They show understanding of error budgets as a mechanism to balance velocity and reliability. Monitoring is multi-layered, spanning metrics, logs, and traces. Alerts are actionable and reduce noise. On-call rotations include fairness, escalation, and burnout mitigation. Red flags include: vague answers without metrics, ignoring error budgets, proposing too many alerts, relying on manual escalation, or treating postmortems as optional.

‍

Preparation Tips

• Review Google SRE Workbook chapters on SLIs, SLOs, and error budgets.
  
• Practice designing SLIs for a sample service (e.g., login API, checkout flow).
  
• Set up a small Prometheus + Grafana lab to visualize latency/error metrics.
  
• Create sample alerts that fire only on meaningful thresholds.
  
• Explore tools like PagerDuty for on-call rotations and escalation.
  
• Study real postmortems from Google, Amazon, or GitHub to understand systemic fixes.
  
• Practice explaining error budgets as a negotiation tool between product and engineering.
  
• Run through a mock incident: define detection, triage, mitigation, resolution, and postmortem.
  
  

Real-world Context

At Google, SRE teams pioneered error budgets to align engineering speed with reliability. For example, a Gmail team enforced a 99.9% SLO for availability; exceeding the error budget forced a feature freeze until stability improved. At Netflix, observability systems monitor stream start latency and error rates; on-call rotations are global to provide seamless coverage. In fintech, payment systems set strict SLIs on transaction success rates; alerts escalate within seconds if thresholds are breached. In e-commerce, Amazon builds runbooks for peak shopping events like Prime Day, ensuring every alert has a clear triage path. Across industries, these practices ensure reliability at scale.

‍

Key Takeaways

• Define SLIs and SLOs around user-facing impact, not infra vanity metrics.
  
• Enforce error budgets to balance velocity with reliability.
  
• Build a layered observability stack: metrics, logs, traces, dashboards.
  
• Alerts must be actionable, routed with clear escalation and runbooks.
  
• On-call rotations and blameless postmortems prevent burnout and enable systemic improvement.
  
  

Practice Exercise

Scenario:
You are the SRE responsible for a global SaaS platform’s authentication service. Users complain about intermittent login failures. Your task is to design a monitoring and incident response strategy.

Tasks:

1. Define at least two SLIs (e.g., login success rate, p95 login latency) and corresponding SLOs.
   
2. Establish an error budget for the month; decide how much unreliability is acceptable.
   
3. Set up monitoring with Prometheus/Grafana to track SLIs in real time.
   
4. Create alert rules: one for error budget burn rate (fast exhaustion), one for sustained latency above SLO.
   
5. Document a runbook for the on-call engineer that includes immediate mitigation steps (e.g., scaling, failover) and escalation contacts.
   
6. Propose a fair on-call rotation schedule and escalation policy (primary, secondary, management).
   
7. After a simulated incident, write a blameless postmortem identifying the root cause and proposing preventive actions.
   
   

Deliverable:
A documented monitoring and incident response plan that shows SLIs, SLOs, error budget policy, alert configuration, on-call schedule, and a sample postmortem outline.