Cruxly › Cloud and DevOps › Cloud & DevOps: Observability

SLOs, Error Budgets, and Alerting on What Users Actually Experience

the Google SRE discipline for deciding what's worth a page

Prompt: design an alerting strategy for a service currently paged on CPU%, memory%, and disk I/O thresholds — alerts that fire constantly, train the on-call rotation to ignore pages, and have repeatedly turned out not to correlate with any actual user-facing problem. Replace it with something that pages only when users are genuinely affected.

30 minutes first. The naive fix tunes the existing thresholds tighter or looser. The actual fix is alerting on a different kind of signal entirely.

SLIs and SLOs turn 'is this bad' into a defined, measured question

An SLI (Service Level Indicator) is a specific, measured quantity that reflects real user experience — the fraction of requests completing successfully, the fraction completing under some latency threshold — chosen specifically because it correlates with whether a real user was affected, unlike an internal resource metric that might correlate with nothing at all. An SLO is a target for that SLI over a defined window: "99.9% of requests succeed, measured over a trailing 28-day window" is a real SLO; "keep CPU under 80%" is not an SLO at all, it's an implementation detail with no stated connection to user experience.

The error budget reframes "how much unreliability is acceptable" as an explicit, spendable number instead of an implicit, unstated expectation of perfection. The error budget is 1 minus the SLO: a 99.9% SLO leaves a 0.1% error budget over that window — a concrete allowance the team can spend on things that carry real reliability risk (a risky migration, a faster release cadence) without every such decision being an unstated gamble against an undefined bar.

Google's own SRE practice ties real policy to the budget, not just a dashboard number: when a service has exhausted its error budget for the measurement window, feature work and non-critical releases pause until reliability recovers, with an explicit exception carved out for critical fixes and security patches — turning "we should probably slow down and focus on reliability" from a debate that happens after an incident into a pre-agreed, automatic policy that doesn't require re-litigating in the moment.

Why alerting on the SLI, not the internal cause, is the actual fix

An internal resource metric (CPU%, memory%, queue depth) is a plausible-sounding proxy for user impact, but it's a proxy — it can be wrong in both directions. A service can run at 90% CPU while serving every request within its latency target (over-provisioned headroom being used efficiently, not a problem), and a service can be well under every resource threshold while still failing requests for a completely unrelated reason (a downstream dependency timing out, a bad deploy). Paging on the proxy means paging on a guess about user impact instead of a direct measurement of it.

Alerting on the SLI — specifically on how fast the error budget is burning, as the next section shows — pages only when users are actually, measurably affected, which is the one property that resource-threshold alerting cannot guarantee no matter how carefully the thresholds are tuned, because tuning a proxy metric's threshold never fixes the fact that it's still a proxy, several inferential steps removed from the thing that actually matters.

This is also what fixes alert fatigue at its root, not just its symptom. A rotation that gets paged on CPU%, memory%, and disk I/O independently, several times a shift, for events that don't correlate with real impact, rationally learns to under-react to pages — which is dangerous specifically because it also dulls the reaction to the rare page that does matter. Reducing the number of alert sources down to the ones that measure actual user impact is a more durable fix than tuning the thresholds on alerts that were measuring the wrong thing to begin with. The old CPU, memory and disk signals don't have to disappear — move them to dashboards and tickets, and keep paging only for the few that predict certain failure (next section).

Burn-rate alerts: paging on how fast the budget is going

"Page when the error rate goes above 0.1%" is the naive SLO alert, and it's bad in both directions. A 5-minute blip over 0.1% pages someone for a problem that used a tiny slice of the month's budget; a steady 0.09% error rate never pages while slowly using most of it. The Google SRE Workbook's answer is to alert on the burn rate — how many times faster than "evenly over the window" the budget is being used.

Multi-window, multi-burn-rate alerting, with the Workbook's numbers for a 30-day, 99.9% SLO:

• The long window decides whether the problem is significant.
• The short window checks it's still happening, so the alert resolves quickly once the problem stops instead of firing for an hour after a fix.
• Low-traffic services need different handling: with a few requests a minute, one failure can look like a huge burn rate. Use longer windows, synthetic traffic, or a minimum-request-count condition.

Some cause-based alerts are still worth keeping. A disk that will be full in four hours (predict_linear on free space) should page before it causes errors — the SLO only starts burning once it's too late. Certificate expiry is the same. The rule isn't "delete every resource alert"; it's "page on user impact and on imminent, certain failures, and send the rest of the resource signals to tickets and dashboards."

How to say it in the interview

"CPU at 85% might mean nothing - the service could be handling every request fine, just running warm. Alerting on internal resource thresholds pages someone based on a guess about user impact instead of a direct measurement of it. An SLO defined on what users actually experience - request success rate, latency - is the signal that actually correlates with whether anyone is affected, which is the entire point of paging someone in the first place."

Checklist

• Defined SLIs as signals that directly reflect user experience (success rate, latency), not internal resource usage
• Set an SLO as a target over a defined measurement window, not an implicit expectation of perfection
• Calculated the error budget explicitly (1 - SLO) and used it as a real, spendable allowance
• Tied a real, pre-agreed policy to error-budget exhaustion (halting non-critical releases) rather than deciding reactively after each incident
• Alerted on the SLI rather than on internal resource thresholds as a proxy for user impact
• Used multi-window, multi-burn-rate alerts (e.g. 14.4× over 1h with a 5m confirm) instead of a raw error-rate threshold
• Kept paging cause-based alerts for imminent certain failures (disk full, cert expiry) and handled low-traffic services differently

Related lessons

• Metrics, Logs, and Traces: Picking the Pillar That Actually Answers Your Question