Cruxly › System design › Foundations of Data Systems

Reliable, Scalable, Maintainable: Measuring a System

three adjectives that mean nothing until each one has a number, a distribution and an owner

Reliable, scalable and maintainable are the three words every design review starts with and almost none ends with a measurement. This lesson turns each into something you can write on a whiteboard: a fault budget, a load parameter, a latency percentile, a fan-out calculation and a list of the work that will page someone at 3am. It builds on Estimating the Load and Designing for Observability and does not repeat them.

Three words that each need a number

An adjective in a design review is a request for a measurement that nobody has made yet. "It must be reliable" can be argued for an hour. "99.9% of requests succeed in under 300 ms, measured at the edge, over 28 days" can be tested on Monday.

Two points keep this honest.

First, the three pull against each other. Redundancy raises reliability and adds moving parts to maintain. Sharding raises scalability and removes the ability to run a transaction anywhere. A simpler system is easier to run and often worse at both of the others. Your job in the room is to say which one you are spending, not to claim all three are free.

Second, 100% is not a target. The Google SRE book's chapter Embracing Risk says 100% is probably never the right reliability target, because it is impossible to reach and usually more than users want or can notice. It adds that an incremental improvement in reliability may cost 100 times more than the previous increment, and that a user on a smartphone with 99% reliability cannot tell the difference between 99.99% and 99.999% service. Other components dominate what that user sees.

:::key Verified November 2026 Google SRE book, Embracing Risk: "100% is probably never the right reliability target", an incremental improvement "may cost 100x more than the previous increment", and the smartphone example above. The same chapter works a request-based example: a service handling 2.5 million requests a day against a 99.99% target can return up to 250 errors and still meet it. :::

Worked arithmetic, not a source's figure. The downtime a target allows is just the miss rate times the window. 99.9% over 28 days leaves 0.001 x 28 x 24 x 60 = 40.3 minutes. 99.99% over a 365-day year leaves 0.0001 x 525,600 = 52.6 minutes. 99.9% over the same year leaves 525.6 minutes, about 8.8 hours. Each extra nine divides the allowance by ten, and by the SRE book's account it can multiply the cost far more than that.

What a staff engineer says in the room. "Which of the three are we optimising for this quarter, and what are we willing to make worse?" What they check before agreeing to a number: whether it is measured at the edge where the user is, whether the window is rolling, and whether anyone has the authority to stop releases when the budget is gone. Designing for Observability covers the SLI, SLO and error-budget mechanics; this lesson uses them as inputs.

Faults versus failures, and why you design for faults

A fault is one component deviating from its spec; a failure is the system as a whole no longer delivering its service. The goal of reliable design is to keep faults from becoming failures, which means you cannot get there by preventing every fault. Disks fail, processes get killed by out-of-memory handlers, deploys ship bugs, operators mistype, a cable gets unplugged. At enough scale something is always broken somewhere, so the design question is what the system does while it is.

Dean and Barroso make the same move for latency in The Tail at Scale: just as fault-tolerant design builds a reliable whole from unreliable parts, they propose "latency tail-tolerant" services that give predictable responses from less predictable components. The idea carries across the whole lesson. You do not eliminate variability in the parts; you build a whole that absorbs it.

Three families of fault, with different defences

• Hardware and infrastructure faults (disk, node, rack, network link, availability zone). Random and mostly independent, so redundancy works: replicas on different failure domains, as in Replication and Consistency. The defence is cheap and well understood. The trap is assuming independence. Two replicas on the same rack share a power feed and a top-of-rack switch.
• Software faults (a bug, a memory leak, a bad config, a poison message). These are correlated: every replica runs the same code, so they fail together. Redundancy does not help. What helps is staged rollout, a canary, a fast rollback, bulkheads and a kill switch.
• Human faults (a wrong command, a bad migration, a mis-scoped delete). Usually the largest source of outages in practice, but this is a general observation and this lesson does not give a percentage because none of the sources it uses states one. Defences: make the safe path the easy one, dry runs, small blast radius, reversible changes, and backups you have restored at least once.

Correlation is the number that matters. Two independent replicas each 99.9% available give, in the simplest model, 1 - 0.001 x 0.001 = 99.9999% for the pair (worked arithmetic, assuming independence). If a shared dependency has its own 99.9%, the pair is capped by it: the system cannot be more available than the thing both replicas need. This is why a design with "three nines, three times" can still be a three-nines system.

Chaos and drills are how you learn what the fault actually does. A failover path that has never run is a hypothesis. A staff engineer asks when each safety mechanism last fired for real or in a rehearsal, because untested recovery code is itself a likely source of the next incident.

What experts say in the room. "What are the faults we are designing for, and which of them are correlated?" The failure story is nearly always a correlated one. What a senior checks: the list of shared dependencies behind the "redundant" components (DNS, config service, certificate expiry, a single deploy pipeline), and the last time failover was exercised.

Load parameters: what 'scale' means for this system

Scalability is not a property of a system; it is a property of a system under a named kind of growth. "Does it scale?" has no answer until you say which number grows. The number is a load parameter, and the right one depends on the design: requests per second to a web tier, the read-to-write ratio on a database, simultaneous open connections for a chat gateway, the size of the largest fan-out for a feed, the hit rate for a cache, the number of keys in the hottest partition.

The same product has several candidates, and they stress different parts:

The distribution matters as much as the total. An average of 100 followers per account hides the account with 50 million. Two systems with the same requests per second can behave completely differently if one has uniform keys and the other a hot key. Always ask for the shape of the load, not just the sum. The SRE book's Service Level Objectives chapter makes the same point about throughput: a system serving 200 requests a second in alternate seconds and nothing in between has the same average as one serving a steady 100, but twice the instantaneous load at peak.

Two ways to answer growth

• Scale up (bigger machine): the least change to the design, and a hard ceiling. Good until the largest instance you can buy, and often the right first answer.
• Scale out (more machines): no ceiling in principle, but you pay in partitioning, coordination and operations. Partitioning and Sharding and Scaling Postgres and MySQL before you leave them cover the moves.

Linear scaling is a claim you test, not assume. If doubling machines doubles throughput, you have linear scaling for that load parameter. Shared bottlenecks (a single leader, a coordination service, a lock, a shared cache) bend the curve. The honest statement is "this design scales out on reads up to the leader's write capacity, about X writes a second, then we need to partition". That sentence names the load parameter, the bottleneck and the next move.

Capacity has a safety margin. The SRE book's Monitoring Distributed Systems chapter lists saturation, how "full" the service is, among its four golden signals, next to latency, traffic and errors. Many services degrade sharply past some load, so you plan for peak plus the loss of a failure domain, not for the average.

What experts say in the room. "What are the one or two load parameters, what is their growth over the next 12 months, and what is the headroom?" What a senior checks: the saturation signal for each tier (CPU, connections, queue depth, disk), and which tier saturates first under the forecast mix. Estimating the Load gives the arithmetic for turning users into requests per second; this section is about picking which number to estimate.

Latency percentiles and why the mean lies

The mean is the one summary of a latency distribution that no user ever experiences. Latency distributions are skewed: most requests are fast and a few are very slow, so the mean sits somewhere between and describes neither. The SRE book's Service Level Objectives chapter says plainly that averaging request latencies may seem attractive but obscures an important detail, and that you cannot assume the mean and the median are the same, or even close.

Worked arithmetic, not a source's figure. Take 100 requests, 99 at 50 ms and 1 at 5,000 ms. The mean is (99 x 50 + 5,000) / 100 = 99.5 ms, double the typical request, and no request took 99.5 ms. The median is 50 ms and the slowest 1% is 100 times slower. Whichever you alert on tells a different story.

The vocabulary

• p50 (median): half of requests are faster. This is the typical experience.
• p95, p99, p99.9: the latency that 95%, 99% or 99.9% of requests beat. The SRE book describes a high percentile such as the 99th or 99.9th as showing a plausible worst case.
• Tail: the slow end of the distribution. The Monitoring chapter's own example: a service with a 100 ms average at 1,000 requests per second can easily have 1% of requests, ten a second, taking 5 seconds.

:::key Verified November 2026 Google SRE book, Monitoring Distributed Systems: "If you run a web service with an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds." The same chapter says to collect request counts bucketed by latency, suitable for a histogram, with boundaries spaced roughly exponentially (it uses factors of about 3), instead of storing every latency. Service Level Objectives gives an illustrative percentile SLO: 90% of Get calls under 1 ms, 99% under 10 ms, 99.9% under 100 ms. :::

Rules a senior holds to

• Never average percentiles. The p99 of the whole service is not the mean of each host's p99. Aggregate the histogram buckets across hosts, then read the percentile. Storing only a per-host p99 makes a correct fleet-wide p99 impossible to compute afterwards.
• Measure error latency separately. The Monitoring chapter says to distinguish the latency of successful requests from that of failed ones, and adds that a slow error is worse than a fast error. If you filter errors out of the latency chart, a system that times out slowly looks healthy.
• Know the ceiling. A timeout creates a cap: the SRE book notes that with a 1,000 ms timeout there can be no successful response slower than that, so the tail of successes is clipped and the rest appears as errors.
• Use different targets for different workloads. The SLO chapter's own example sets separate objectives for throughput-oriented clients and latency-sensitive ones, instead of one number for everything.
• Pick the percentile from the user, not from habit. p99 means 1 in 100 requests. A page that makes 30 backend calls per view will, at p99 per call, give most users at least one slow call (see the next section). Heavy users, who make the most requests, also hit the tail most often, and they are often your most valuable users.
• Do not set a target from today's performance. The SRE chapter recommends starting loose and tightening, and keeping the number of SLOs small.

Coordinated omission is the measurement bug interviewers like. If a load generator waits for each response before sending the next request, a stall removes all the requests that would have been sent during the stall, and the recorded tail looks far better than users would have seen. Send on a schedule, record from the intended send time. The term is Gil Tene's, and this is load-testing practice rather than a claim from the sources above.

What experts say in the room. "p99 under 300 ms at the edge, p50 around 40, and I want the histogram, not the mean." What a senior checks: that percentiles are computed from merged histograms, that error latency is plotted, and that the SLO percentile matches what a user session actually experiences.

Tail latency under fan-out

Fan-out turns a rare slow response into the common case. This is the central result of Dean and Barroso's The Tail at Scale (Communications of the ACM, February 2013), and it is the reason percentile targets at the leaf are far stricter than percentile targets at the user.

The paper's example. Suppose each server typically answers in 10 ms but has a 99th-percentile latency of one second. If a request touches one such server, one user request in 100 is slow. If it must collect answers from 100 such servers in parallel, the paper states that 63% of user requests take more than one second. It adds a second case: with only one in 10,000 requests slower than a second at each server, a service of 2,000 such servers sees almost one in five user requests exceed a second.

The formula behind it (worked arithmetic). If each of N servers is slow with probability p, independently, the chance that a request waiting for all of them is slow is 1 - (1 - p)^N.

The 63% and the roughly 18% match the paper's figures. To keep the user-level p99 healthy with a fan-out of 100, the per-server slow rate has to be about 0.01% (a p99.99 target), because 1 - 0.99^(1/100) is about 0.0001. That is where the leaf's budget comes from.

:::key Verified November 2026 Dean and Barroso, Table 1, measured on a real Google service with a fan-out tree. The 99th-percentile latency for one random leaf to finish, measured at the root, is 10 ms. Waiting for 95% of leaf requests to finish: 70 ms at the 99th percentile. Waiting for all of them: 140 ms. The paper observes that waiting for the slowest 5% of leaf requests accounts for half of the total 99th-percentile latency. :::

The mitigations, and what each one costs

• Hedged requests. Send the request to a second replica if the first has not answered within roughly the 95th-percentile latency for that class. The paper says this limits extra load to about 5%. In a BigTable benchmark reading 1,000 keys spread across 100 servers, a hedge after 10 ms cut the 99.9th-percentile latency to retrieve all 1,000 values from 1,800 ms to 74 ms while sending just 2% more requests. Cost: more load, and the operation must be safe to run twice. Tag the hedge as lower priority to reduce its overhead.
• Tied requests. Send to two servers at once, each told about the other, and cancel the duplicate as soon as one starts running. The paper's Table 2 benchmark on a mostly idle cluster shows the 99.9th percentile falling from 98 ms to 61 ms (about 38%), and from 159 ms to 108 ms (about 32%) with a concurrent terasort job. Cost: cross-server cancellation messages and replicas that can accept the work.
• Micro-partitions. Create many more partitions than machines, and move small ones to rebalance. With about 20 partitions per machine, the paper says, load can be shed in roughly 5% steps, in about one twentieth of the time a one-to-one mapping would take. Cost: more metadata and more places to look.
• Latency-induced probation. An intermediate server watches response times and puts a slow machine on probation, because the slowness is often temporary interference. Cost: it can make a loaded system smaller at the moment it needs capacity, so it needs a limit and a way back.

Where the arithmetic bites in your own design. A product page that calls 30 services serially and 6 in parallel, an API gateway that scatters a search to 200 shards, a cache cluster where a multi-get touches every node, all multiply tails. The cheap defences are fewer calls per request, a deadline propagated down the call chain, timeouts shorter than the caller's deadline, and returning partial results rather than waiting for the slowest. Rate Limiting and the load-shedding work in the reliability phase are the other half: hedging under overload adds load exactly when the system can least take it.

What experts say in the room. "With fan-out 100 the user p99 is the leaf p99.99, so I will hedge after the p95 and keep the extra load near 5%." What a senior checks: the fan-out per request in the busiest path, whether the operation is idempotent before hedging it, and whether a retry storm is possible when the slow component is also the overloaded one.

Throughput versus latency, and where they trade

Throughput is how much work per second; latency is how long one piece of work takes; and you cannot raise one forever without paying in the other. The SRE book's Service Level Objectives chapter notes that they are linked: higher request rates often increase latency, and many services have a performance cliff past some load threshold.

Little's law ties the two together. In a stable system, the average number of requests in flight equals the arrival rate times the average time each spends in the system: L = lambda x W. Worked arithmetic: a service taking 200 ms per request at 500 requests per second has, on average, 500 x 0.2 = 100 requests in flight. If each uses a worker thread, you need at least 100 threads, and if the pool is 64 the excess queues. This is the cheapest capacity calculation in the field and it works for connection pools, thread pools and database connections alike.

The queueing curve is the cliff. Utilisation is the share of capacity in use. With random arrivals, queues grow explosively as utilisation approaches 100%. In the simplest single-queue model (M/M/1: random arrivals, random service times, one server), average time in the system is the service time divided by (1 - utilisation). Worked arithmetic under that model:

Real systems are not M/M/1, so the exact multiples differ, but the shape is general: latency is nearly flat until the knee and then climbs steeply. That is why running a tier at 90% CPU "because it is efficient" is a latency decision, not a cost decision, and why the tail suffers first. Dean and Barroso argue the opposite direction as well: tail-tolerance techniques are what let utilisation rise without lengthening the latency tail, so you avoid wasteful over-provisioning.

Where design choices trade one for the other

• Batching. Group 100 writes into one disk flush or one network call and throughput rises sharply, but each item waits for the batch to fill. A batch window of 10 ms adds up to 10 ms to every request. Choose the window from the latency budget.
• Queues. A queue lets bursts through without rejecting them and keeps throughput high, but it converts overload into delay. A queue without a bound or an age limit is a latency bug waiting for a spike. Drop requests older than the caller's deadline, since the caller has already given up.
• Caching. Improves both until the hit rate drops; the miss path then has to carry the full load alone, which is the thundering-herd case in Caching.
• Concurrency limits. Capping work in flight lowers peak throughput slightly and protects latency for the work that is admitted. Excess is rejected quickly instead of failing slowly.
• Replication and durability. Waiting for more replicas to acknowledge improves safety and adds latency, the same trade as Replication and Consistency.

Goodput, not throughput. Throughput counts requests served; goodput counts requests served within their deadline that the user still wants. Past the knee, throughput can stay flat while goodput falls toward zero, because every response arrives too late to be used. Load shedding exists to protect goodput.

What experts say in the room. "I would not plan above roughly 70% utilisation on a latency-sensitive tier, and I would put a bound on every queue." The 70% is a common planning rule, not a number from the sources here; the table above is why a team would pick some number below the knee. What a senior checks: latency versus utilisation on a load test, the knee's location, queue depth and age, and what happens to goodput when the offered load is 150% of capacity.

Maintainability: operability, simplicity, evolvability

Most of a system's cost arrives after launch, as operations and change, which is why maintainability is a design property and not a coding-style one. It is the quality people skip because it has no obvious metric. It does have measurable proxies, and a staff engineer asks for them.

Three parts, each with something you can count

Operability: count the toil. The SRE book's Eliminating Toil chapter defines toil as operational work that is manual, repetitive, automatable, tactical, without lasting value, and scaling linearly with the service. It states a target of keeping toil below 50% of each SRE's time, and warns that toil grows if unchecked and becomes toxic in large quantities. As a design review question: for each new component, which recurring manual task does it create (rotating certificates, resharding, rebuilding an index, replaying a dead-letter queue), and is there an owner and a plan to automate it? A design that needs a human every Friday to rebalance is one that has linear operating cost hidden inside it.

Monitoring is part of operability, and it should be simple. The Monitoring chapter says to design the monitoring system with an eye toward simplicity, that the rules catching real incidents most often should be as simple, predictable and reliable as possible, and that monitoring can become so complex it is fragile. It also says to spend much more effort catching symptoms than causes. Its framing question is what is broken, and why. The four golden signals (latency, traffic, errors, saturation) are the shortest answer for a user-facing service. See Designing for Observability for how to wire them.

Simplicity: essential versus accidental complexity. Some complexity comes from the problem, and some comes from choices. Each additional store, queue or service adds failure modes, a deployment, a dashboard, an upgrade path and a learning curve. The test is whether you can name what each component is for in one sentence and what you would lose if you deleted it. Good abstractions hide detail behind a stable interface so a change stays local; bad ones leak, and the leak is where incidents live. Boundaries Are Where Guarantees Die covers where to cut.

Evolvability: assume the requirements are wrong. They will change, so the design should make the common changes cheap.

• Keep schema changes backward compatible so old and new code can both run during a rollout, the expand-then-contract pattern.
• Make deploys reversible and small. The ability to roll back in minutes is worth more than a perfect pre-release review.
• Feature flags decouple deploying code from enabling it.
• Prefer data stores whose operations you can do online (add a column, add a replica, split a shard) over ones that need downtime.

Three questions to put on the whiteboard

1. What is the cost to the team of running this for a year, in people and pages, not just in servers?
2. What is the first thing a new on-call engineer would be confused by?
3. Which decision here is hardest to reverse, and have we deferred it as long as we can?

A boring technology rule. Every novel component is a bet on the team's ability to operate it at 3am. Spend that novelty on the one place the product needs it, and use well-understood parts elsewhere.

Further reading. The framing of reliable, scalable and maintainable used here is the one Martin Kleppmann's Designing Data-Intensive Applications uses to open its first chapter; this lesson's explanations, examples and numbers are our own and come from the Google papers and SRE book chapters cited above. For the operational side in depth, read the free Google SRE book at sre.google, and for latency at scale, Dean and Barroso's The Tail at Scale.

What experts say in the room. "Who gets paged for this, what do they do first, and how many manual steps is that?" What a senior checks: the toil budget, rollback time, the number of data stores and queues versus the number of engineers who understand them, and whether a schema change can ship without downtime.

How to say it in the interview

"When someone says the system must scale, I ask scale in what: which load parameter, and at which percentile do we promise latency? The mean hides the slow requests, so I would commit to something like p99 under 300 ms and watch p99.9 as well. And I would do the fan-out arithmetic early: if one backend is slow on 1 request in 100 and a page calls 100 of them in parallel, 63% of pages hit a slow backend. Reliable then means tolerating faults, not preventing them, and maintainable means I can name the toil this design creates."

Checklist

• Every quality target in the design is written as a number, a window and a place of measurement (for example p99 under 300 ms at the edge over 28 days)
• The fault list names which faults are designed for and which are correlated (same code, same rack, shared config or DNS)
• Each availability figure for a composed system is computed from its dependencies, including shared ones
• One or two load parameters are named with a 12-month growth forecast and the tier that saturates first
• Latency is reported as percentiles from merged histograms, never as a mean or as an average of per-host percentiles
• Error latency is plotted separately from success latency
• The largest fan-out per request is known, and the leaf percentile target is derived from 1 - (1 - p)^N for it
• Hedging and retries are used only on idempotent operations and are capped to a small share of extra load, with deadlines propagated down the chain
• Every tier has a utilisation ceiling below its latency knee, every queue has a bound and an age limit, and goodput is measured under 150% offered load
• Each new component lists the recurring manual tasks it creates, with an owner and an automation plan, against a toil cap

Related lessons

• Clocks, Ordering and Partial Failure
• Choosing a Datastore
• How Indexes Actually Work
• Replication and Consistency
• Partitioning and Sharding
• Transaction Isolation Levels