Cruxly › System design › Reliability and Observability

SLAs, Availability Math and Error Budgets as Design Inputs

a target is a design input: it decides how many dependencies you may call, how many copies you run, and when releases stop

Reliable, Scalable, Maintainable: Measuring a System gave you the nines, the SLI and SLO vocabulary and the error-budget idea, and SLOs, Error Budgets, and Alerting on What Users Actually Experience and Designing for Observability cover the mechanics of measuring them. This lesson uses the numbers to design: how availability composes across the dependencies you call, why the SLO you run must be tighter than the SLA you sign, how multi-window burn-rate alerts are parameterised, what an error-budget policy must say, and which tier of a product deserves how many nines.

SLI, SLO and SLA: three documents for three audiences

An SLI is a measurement, an SLO is an engineering target, and an SLA is a contract with consequences; they have different owners and are allowed to disagree. The SRE book's Service Level Objectives chapter defines the SLA as an explicit or implicit contract with users that includes consequences of meeting or missing the SLOs it contains, and gives a test: ask what happens if the SLO is missed. With no explicit consequence, you are almost certainly looking at an SLO. The workbook's Implementing SLOs chapter puts the same point in business terms: an SLA is a business contract that takes effect when users are so unhappy that you have to compensate them.

The workbook recommends writing an SLI as the number of good events divided by the total number of events, for example successful HTTP requests over all HTTP requests. That form has a useful property: the error budget is 100% minus the SLO, so it is a count you can spend. Its worked case: a 99.9% SLO over 3 million requests in four weeks leaves 3,000 allowed errors, and one outage that causes 1,500 errors uses 50% of the budget.

The design consequence of the difference. Many internal services should have an SLO and no SLA, because nobody will pay you for missing it and the SLO is only a tool for deciding how to spend engineering time. An SLA is a liability: it should be signed only for the few request classes where a customer's money depends on you, and it should always sit behind a stricter SLO (section 5). Teams that skip the distinction either sign contracts that match their best month, or treat an internal target as if it were a legal promise and refuse to ship.

Why an SLO is a shared decision. The SRE book's Embracing Risk chapter describes the arrangement: product management sets the SLO, a neutral monitoring system measures actual availability, and the difference is the remaining budget. The same chapter states the economics. An incremental improvement in reliability may cost 100 times more than the previous increment, and the chapter's own example values a move from 99.9% to 99.99% on a service earning $1M at $900 (a 0.09% availability increase), so the improvement is worth doing only if it costs less than that. The number is a business input, so engineering cannot choose it alone.

What experts say in the room. "Who gets paid if we miss this?" splits SLO from SLA in one question. What a senior checks: that every SLA clause maps to a measured SLI with the same definition of a good event, the same measurement point and the same window, because a contract measured differently from your dashboard is a dispute waiting to happen. For monitoring mechanics see Designing for Observability.

Choosing SLIs per system type

The right SLI is the one that fails when a user of that kind of system is unhappy, so it depends on what the system does. The workbook's Implementing SLOs chapter groups them by system type, and the same list is a good template for a design review.

The SRE book's Service Level Objectives chapter says the same by listing user-facing serving systems (availability, latency, throughput), storage systems (latency, availability, durability) and big-data systems (throughput, end-to-end latency), and adds that all systems should track correctness, though correctness is often a property of the data and not of the infrastructure.

Three habits that separate good SLIs from dashboards.

• Latency is a proportion, not a mean. "99% of requests under 900 ms" is a good-events-over-total SLI. The SRE book warns that averaging latencies hides the slow requests behind a fast mean. Reliable, Scalable, Maintainable: Measuring a System covers percentiles and fan-out; here the point is that the percentile form is what feeds an error budget.
• Split by request class. The workbook's tiered examples set premium customers at 99.99% availability and free customers at 99.9%, and separate interactive requests (90% completing in 100 ms) from CSV downloads (90% starting within 5 seconds). One blended SLO lets a cheap, noisy class hide a failure in the valuable one.
• Measure close to the user. Measuring at the load balancer or the client captures failures that the application never logs, such as a dropped connection or a timeout in a proxy. Measuring inside the process only counts requests that arrived.

Pipelines need freshness, not uptime. A nightly job that always runs but loads yesterday's data twice is 100% available and wrong. Pick freshness and correctness for a pipeline, coverage for a batch job that can silently process only part of its input, and durability for a store whose failure mode is a read that returns nothing.

:::warn Counting what cannot happen A request that never reaches you cannot be counted as an error by you. If your SLI is computed from your own server logs, a total outage upstream of the logs reads as 100% good. Pair the server-side SLI with a probe or client-side measurement. :::

Do not set the target from today's performance. The SRE book says adopting current values without reflection can lock you into a system that needs heroic effort to meet its targets. Start loose and tighten, keep aggregations simple, and keep as few SLOs as possible: drop any SLO you could never use to win an argument about priorities.

What experts say in the room. "What is a good event for this request class, and where do we count it?" What a senior checks: one SLI per class that matters, a latency SLI with a threshold in the same sentence, a freshness or correctness SLI on every pipeline, and a second vantage point so the SLI cannot read perfect during an outage.

Serial dependencies multiply: the ceiling on what you can promise

If a request needs every one of its dependencies to succeed, its availability is the product of theirs, so the ceiling on your SLO is below your weakest dependency and gets lower with every extra hard dependency. This is arithmetic, not a source's figure: availabilities of independent components in series multiply.

Worked arithmetic. A checkout handler that hard-depends on five services, each at 99.9%, is available 0.999 to the fifth, which is 0.99501, or 99.5%. Over a 30-day month (43,200 minutes) that is 0.00499 x 43,200 = about 215.6 minutes of expected unavailability, nearly 3.6 hours, from services that each look excellent alone. Ten hard dependencies at 99.99% each give 0.9999 to the tenth = 99.90%, so a four-nines target has become three nines before your own code has failed once. Your own service sits in the product too: a 99.99% service in front of a 99.9% database cannot exceed 0.9999 x 0.999 = 99.89%.

Computed, the diagram's product is 0.9999 x 0.999 x 0.9999 x 0.999 x 0.999 = 0.9968 (worked arithmetic).

The ways to beat the product are design moves, not operations.

• Remove the dependency from the critical path. Move the call after the response (publish an event and let a worker finish the job), so a failure becomes delay and not an error. The queue lessons in this track cover the hop.
• Make it soft. A dependency with a cached fallback, a default or a degraded response does not multiply into the product; it moves from availability to the quality SLI.
• Cut the number of hops. Each synchronous layer removed takes a factor out of the product. A chatty chain of eight services at 99.9% is 0.992, 99.2%, before any of them is the bad one.
• Make your dependency better than you promise. If you sign 99.9% and a hard dependency offers 99.9%, you have no margin at all: the product of the two is below the number you signed.

Dependency SLAs are not guarantees. A cloud provider's published SLA is a promise of compensation, and it is typically per service and per region. It tells you the number the provider is willing to be paid against, not what your composed system will achieve. Use measured availability of the dependency for your own maths, and the published number for your risk register.

What experts say in the room. "Draw the request path, circle every hard dependency, and multiply." The argument is whether a call may be taken off the path; the number is the product. What a senior checks: the dependency count on the critical path, which dependencies have a fallback, and whether the SLO is above the product of the hard ones, because if it is, the promise cannot be kept by operations alone.

Redundancy raises availability, until failures stop being independent

Parallel redundancy multiplies the unavailabilities, which is why it is the main tool, and why the independence assumption under it is the part that fails. The AWS whitepaper Availability and Beyond, in its section on availability with redundancy, gives the formula: a workload with two redundant subsystems is down only when both are down, so the probability of failure is the product of the two failure probabilities. Its example: two subsystems at 99% each fail together with probability 1% x 1% = 0.01%, giving 99.99%.

:::key Verified November 2026 AWS whitepaper, Availability and Beyond, "Availability with redundancy": a workload with s spares fails only when s + 1 subsystems fail; each spare costs as much as the original module, so cost grows at least linearly; with a 99.5% subsystem, two spares give about 99.9999875% (about 3.94 seconds of downtime a year) and ten spares are effectively zero downtime, "a 5X increase in the cost of sparing to achieve four seconds less downtime a year". The paper's conclusion: use the fewest spares needed, and beyond three spares the returns are negligible for almost all workloads. :::

The unit of failure decides what a spare is. The whitepaper's example: a workload needs 10 instances at peak. In one availability zone, adding more instances does not help against a zone failure, because the whole zone fails together. Two zones with 10 each, 20 instances, give one spare zone. Three zones with 5 each give 15 instances and still serve peak load after losing one zone, which is cheaper and tolerates the same event. So you count spares in units of the isolation boundary: instance, cell, zone, region.

The independence trap. The formula assumes failures happen independently. Real incidents are often common-mode: the same bad deploy goes to both copies, the same configuration push, the same expired certificate, the same shared control plane, the same traffic spike that overloads both. The AWS paper states the assumption directly (independent and redundant subsystems), and sparing applies to dependencies that fail independently, such as instances in different zones.

Worked arithmetic, not a source's figure. Take two replicas at 99% each. Assume 10% of each replica's downtime comes from a cause that hits both at once (a shared release), and the rest is independent. Joint unavailability is about 0.1 x 1% + (0.9 x 1%)^2 = 0.001 + 0.000081 = 0.001081, so the pair is 99.89%, not the 99.99% of the naive formula. One tenth of correlated failure took away a whole nine. Even at 99.9% replicas (the same split) the pair gives about 99.990% against the naive 99.9999%. The independent part is almost irrelevant; the correlated tenth dominates.

Moves that restore independence. Stagger rollouts across cells or zones so a bad release reaches one copy first. Keep a manual or separate control path for failover. Avoid putting the failover decision behind the thing that failed. Test the failover on a schedule, because an untested spare has an unknown probability of working, and the formula silently assumes it works.

What experts say in the room. "Two zones gives me the independent part of the maths; the shared deploy is what I need to design around." What a senior checks: which causes are common to both copies, how many spares are in units of the failure boundary, and when failover was last exercised.

Why your SLO must be tighter than the SLA you sign

The SLA is the line where you pay, the SLO is the line where you act, and the gap between them is the time you have to notice and fix a problem before it costs money. The SRE book's Service Level Objectives chapter lists a tighter internal SLO than the advertised one as a recommended tactic, so you can respond to chronic problems before users notice. The Embracing Risk chapter's example is a contract of 99.9% on a quarterly basis for an external product with a stronger internal target behind it.

Worked arithmetic, not a source's figure. Suppose a customer SLA promises 99.9% a month, which allows 43.2 minutes over 30 days. If the internal SLO is also 99.9%, a single 30-minute incident leaves 13 minutes, and an ordinary bad month breaches the contract. With an internal SLO of 99.95% the budget is 21.6 minutes, so the operating rule triggers (freeze, reliability work) after 21.6 minutes spent, which leaves 21.6 minutes between the SLO line and the SLA line. That slack pays for detection delay, a rollback that takes longer than planned, and a second incident in the same month.

How to set both.

• Start from what the dependencies allow. The SLA you sign cannot exceed the product of your hard dependencies (section 3) plus what your fallbacks add back. If three hard dependencies are at 99.9% each, 0.999^3 = 99.7%, and you cannot responsibly sign 99.9% on the strength of hope.
• Set the SLO from user pain. Pick where users start to notice, informed by measured history, then loosen or tighten. The SRE book says not to pick from current performance, and to start loose.
• Set the SLA below the SLO, and below what you have actually delivered for several windows. Customers rarely ask for more than the previous best months; the contract should not.
• Define the window and the exclusions in the contract. A calendar month, a quarter and a rolling 28 days give very different budgets for the same percentage. Planned maintenance, customer-caused errors and force majeure belong in the text.
• Match the measurement. The contract's definition of a good request must be reproducible from your SLI, or credits become a negotiation.

Do not overshoot either. The SRE book's Chubby example is the counter-argument to over-delivering: users come to depend on what a service delivers, not what it promises, so SRE ensures that global Chubby meets but does not significantly exceed its SLO, and in a quarter without real failure synthesises a controlled outage to expose unreasonable dependencies early. If you are running far above your SLO you are either overbuilt or accumulating hidden dependants.

:::warn A dependency SLA does not cover your SLA If you sign 99.9% to customers on top of a 99.9% provider, you have no margin: the product of the two is below what you signed, and the provider's remedy is usually a credit against its fee, not compensation for your customer's loss. :::

What experts say in the room. "Sign the SLA at the number we can defend, run the SLO above it, and alert on the SLO." What a senior checks: the gap between SLO and SLA in minutes (not percent), the window and exclusions in the contract text, and that the SLA is not above the composed ceiling.

Multi-window burn-rate alerts: the numbers and why each exists

A burn rate says how fast the error budget is being spent relative to the SLO, and alerting on it ties each page to a fixed share of the budget. The workbook's Alerting on SLOs chapter defines burn rate as how fast, relative to the SLO, a service consumes its error budget: a burn rate of 1 uses exactly the whole budget by the end of the window. The budget consumed by the time an alert fires is the burn rate times the alert window divided by the period. SLOs, Error Budgets, and Alerting on What Users Actually Experience shows the final recommendation; this section explains how the workbook gets there.

:::key Verified November 2026 Workbook, Alerting on SLOs, recommended parameters for a 99.9% SLO over 30 days: page at burn rate 14.4 with a 1-hour long window and a 5-minute short window (2% of budget consumed); page at 6 with 6 hours and 30 minutes (5%); ticket at 1 with 3 days and 6 hours (10%). Guideline: make the short window 1/12 of the long window. :::

Worked arithmetic on the table. A 30-day period is 720 hours. Budget consumed = burn rate x window / period: 14.4 x 1/720 = 2%; 6 x 6/720 = 5%; 1 x 72/720 = 10%. Time to empty the whole budget at that burn rate is 720 hours / burn rate: 50 hours at 14.4, 120 hours (5 days) at 6, and 30 days at 1. So the first page means "we will be out of budget in about two days", the second "in five days", the ticket "at the end of the month".

The strategies, in the order the chapter tries them, and the flaw of each.

Why two windows. The long window answers is this significant, because a burst of one minute at a huge rate is a rounding error against a month. The short window answers is it still happening, so the page clears about when the errors stop and not an hour later. The 1/12 guideline: the chapter's example is 15% errors for 10 minutes, where, with a 14.4 threshold (a 1.44% average over the hour), the long window crosses after about 6 minutes (1.44 / 15 x 60 = 5.8) and the alert clears about 5 minutes after the errors stop, when the short window drains.

Worked arithmetic on detection time. A total outage (error ratio 1) at a 14.4 threshold means the 1-hour window must average 14.4 x 0.001 = 1.44% errors. That takes 0.0144 x 3600 = about 52 seconds of full outage, which is the page latency for the worst case. A partial failure at 1.44% takes the whole hour to cross the line, and a 0.1% failure never pages and only reaches the ticket tier.

Where it breaks: low traffic. The workbook's example: with 10 requests an hour, one failed request is a 10% error rate for that hour. Its suggestions: generate artificial traffic for signal, combine related services into one monitored group, change the product or client (retries, fallbacks) so a single failure matters less, or lower the SLO or lengthen the window to match the real impact of one failed request. Arithmetic: against a 0.1% budget rate, a 10% hourly error is 100x, so one request would page if you did nothing.

What experts say in the room. "Page on budget velocity, not on a rate, and keep the ticket tier so slow burns do not wait for the month's end." What a senior checks: that each alert maps to a share of budget, that a 35x style gap does not exist, that a short window exists on every page, and that low-traffic services have a floor of events or synthetic probes. Cause-based alerts that predict exhaustion (a disk filling) remain, as the DevOps lesson says.

Error-budget policy: what happens when the budget is spent

An error budget has no force until a written policy says what changes when it runs out, agreed in advance by the people who can stop a release. The SRE book's Embracing Risk chapter describes the arrangement: the product and SRE teams jointly define a quarterly budget from the SLO, a neutral system measures real availability, and new releases may ship while measured availability stays above the SLO. When the budget runs low, releases can slow or halt until reliability improves. Its example: for a 99.999% SLO the budget is 0.001% of queries, and a problem failing 0.0002% of queries spends 20% of the quarterly budget.

What the policy must contain. The workbook's Implementing SLOs chapter lists the actions that can follow exhaustion: the development team gives top priority to reliability bugs from the past four weeks; the team focuses exclusively on reliability until the service is back within SLO; and a production freeze halts certain changes until there is enough budget to resume. A usable policy turns this into a table with owners and thresholds:

Worked incidents from the workbook. On a four-week budget of 109,897 errors (a 97% availability SLO over 3,663,253 requests), a bad API release causing 14,066 errors used about 13% of the budget (14,066 / 109,897 = 12.8%). A database server failure with 20 hours of restore time, causing about 72,000 errors, used about 65% (72,000 / 109,897 = 65.5%). The point of the numbers: one operational event can cost most of a period, so recovery time (restore, rollback) is a design input that sets how many such events the SLO can absorb.

The negotiation with product. Without a budget the argument is reliability against features with no shared unit. With one, the question becomes: do we want to spend the remaining minutes on this launch? The SRE book frames the budget as something to spend on innovation, and the rule that product sets the SLO means product has a lever: a looser SLO buys more change, and a tighter one costs release speed. Both sides see the same neutral measurement, so the debate is about the target, not about whose dashboard is right.

Failure modes of a policy.

• A policy nobody can enforce. If the person who can freeze releases did not sign it, the first conflict will cancel it.
• A budget that is always full. Per the Chubby example (section 5), a service far above its SLO is a signal to loosen the SLO, ship faster or take a planned risk, not a source of pride.
• A budget that is always empty. The SLO is wrong, or the system needs investment; a permanent freeze is a cost to the business that should be visible.
• Counting the wrong things. Planned maintenance, client errors and load-test traffic need written rules on whether they count. The Embracing Risk chapter counted scheduled maintenance windows in one product as planned downtime and not unplanned downtime; whatever you choose, write it down.

What experts say in the room. "We spend the budget on purpose, and when it is gone, features stop." What a senior checks: that the policy is signed by product and engineering leaders, that the freeze condition is a number, and that the budget remaining is visible on the same dashboard as the release calendar.

Designing to a target: which tier gets how many nines, and what it costs

Targets belong per tier, not per company: assign each request class the fewest nines its failure actually costs, then pay for exactly that in architecture. The Embracing Risk chapter's cost rule (an increment may cost 100 times more than the last) and the AWS paper's rule (use the fewest spares needed) point at the same practice. The workbook's tiered examples set premium customers at 99.99% and free customers at 99.9%.

A way to assign tiers (a suggested starting point, not a sourced table). Ask what a failed request costs:

Worked arithmetic, not a source's figure. Allowed unavailability per 30 days: 99% = 432 minutes; 99.9% = 43.2; 99.95% = 21.6; 99.99% = 4.32; 99.999% = 0.43 (26 seconds). At 4.32 minutes a month, a single human-triggered rollback that takes 10 minutes breaches the target alone, so the design cannot rely on a person in the loop. Automated failover, canary and automatic rollback are not optional extras at that tier; they are what the budget physically requires. At 99.9%, a 43-minute budget tolerates one detected incident with a manual rollback.

What each extra nine adds to the bill.

• From 99% to 99.9%: automate deploys and rollbacks, add health checks and a second instance, put the data behind a replica with a promoted standby. Mostly engineering process.
• From 99.9% to 99.99%: spread across zones (see the unit-of-failure rule in section 4), remove hard dependencies from the path (section 3), canary every release, practise failover. Capacity cost rises because spares are real machines, and so does the cost of the complexity that finds and routes around failures.
• Beyond: regional redundancy and active-active data. Here consistency (see the consistency lessons in this track) and cost both rise, and the independence trap bites hardest because the remaining outages are the shared ones: global config, control planes, the deploy pipeline itself.

:::warn The user's weakest link may not be you The Embracing Risk chapter's smartphone example: a user on a phone with 99% reliability cannot tell the difference between 99.99% and 99.999% service. If the client path, the radio network or a third-party SDK is worse than your target, spending on more nines is invisible. :::

Describing the trade-off in an interview. Start with the user-visible request classes, give each an SLI and a target, compose the ceilings from the hard dependencies, show the one or two design moves that lift the critical class (async hop, fallback, extra zone), and state what you are deliberately leaving lower. Then say how you would know: the SLI, the burn-rate alerts and the policy. That one chain, from target to architecture to alert to policy, is the shape of a staff-level answer.

What experts say in the room. "Four nines for checkout, three for browse, and the extra nine costs us an automatic rollback and a spare zone, which I will pay for." What a senior checks: per-tier targets with the minutes in the budget, whether any single human action fits inside the budget, the unit of failure for each spare, and whether the SLA signed to customers is below the composed ceiling.

Further reading. Site Reliability Engineering: How Google Runs Production Systems, edited by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (free at sre.google), is this phase's book, and the SRE workbook that goes with it carries the alerting and policy material used here.

How to say it in the interview

"I would not pick a number first, I would pick the SLI per request class and then compose. If checkout hard-depends on five services at 99.9% each, the best I can promise is about 99.5%, roughly 3.6 hours a month, unless I mask a failure with a fallback or an async hop. Redundancy only helps for failures that are independent, and a shared deploy or config is not, so I give the SLA a margin: SLO 99.95% behind a 99.9% contract, with burn-rate alerts at 14.4 over an hour and 6 over six hours, and a written policy that freezes feature releases when the budget is gone."

Checklist

• Each request class has its own SLI written as good events over total events, with the definition of a good event stated in one sentence
• The SLI is also measured from a second vantage point (probe, load balancer or client) outside the service's own logs
• The critical path is drawn with every hard dependency circled, and the product of their measured availabilities is below the SLO by a stated margin
• Dependencies that can be soft (cache, default, async hop) are made soft and moved to the quality SLI
• Spares are counted in units of the failure boundary (instance, cell, zone, region), with capacity to survive losing one unit
• The common-mode causes (shared release, config push, certificate, control plane) are listed and rollouts are staggered across copies
• The SLA is below the SLO, the gap is expressed in minutes, and the window, exclusions and measurement point are in the contract
• Page alerts use the multi-window burn rates (14.4 over 1 hour with 5 minutes, 6 over 6 hours with 30 minutes) and a ticket tier at 1x over 3 days with 6 hours
• A signed error-budget policy names the freeze condition as a number and the person who can enforce it
• Every tier's budget is converted to minutes and checked against the longest single recovery action it must absorb

Related lessons

• Designing for Observability
• Metrics Pipeline
• Distributed Tracing System
• Overload, Load Shedding and Cascading Failure
• Disaster Recovery: Backup, Pilot Light, Warm Standby, Active-Active