Cruxly › System design › Reliability and Observability

Overload, Load Shedding and Cascading Failure

a system that does everything it is asked at the top of the curve ends up doing nothing

Most large outages are not one component breaking. They are a small fault, a slow dependency or a restart, that turns into more load on the survivors, which then fail too. This lesson is the defensive stack against that: why goodput collapses, how retries and timeouts multiply across layers, what to drop and when, and how bulkheads, breakers and degraded modes limit the blast radius. It builds on Rate Limiting (reject versus delay, priorities), Scaling Up, Scaling Out and Splitting (autoscaling lag) and Boundaries Are Where Guarantees Die (idempotent retries), and does not repeat them.

Why systems fail at the top of the load curve

Claim: past a threshold, a loaded server does less useful work as you give it more, so overload is a feedback loop and not just a big number.

Plot goodput, the work finished in time to be useful, against offered load. It rises with load, flattens at capacity, and then in an unprotected system falls. The AWS Builders' Library load-shedding article describes the shape: response time climbs until an inflection point and then deteriorates quickly. When a request's latency equals the client's timeout, half the requests time out and availability is about 50 percent, even though the server is busy all the time. The server is working, and the work is wasted.

Four mechanisms push goodput down, and the SRE book's cascading-failure chapter names the same family:

• Queues that run past the caller's patience. A server that takes 11 seconds to dequeue a request whose client deadline is 10 seconds does the work and earns nothing. The SRE chapter says to keep queues short: for steady traffic, a queue of 50% of the thread pool or less.
• Resource exhaustion. Threads, memory, file descriptors and connection pools run out. The SRE chapter works one example: 10 frontends with 100 threads each is 1,000 threads. At 1,000 QPS and 100 ms per request about 100 threads are busy. If 5% of requests hang until a 100-second deadline, they need 50 QPS x 100 s = 5,000 threads, far more than exist, and the chapter puts the resulting error rate near 80%. One slow request class starved everything else.
• Work amplification. Retries, failover and cache misses create more requests than users sent. The next section is this one.
• Death spirals. A task that crashes at 11,000 QPS comes back with a cold cache and no capacity. The SRE chapter's example: a service healthy at 10,000 QPS that starts crashing at 11,000 QPS does not recover when load drops to 9,000, because the few healthy tasks absorb their dead neighbours' share. If only 10% of tasks are healthy the offered rate has to fall to about 1,000 QPS before they can hold.

Metastable failure is the name for the loop that survives its trigger. Bronson, Aghayev, Charapko and Zhu (HotOS 2021) define it: a trigger pushes the system into a bad state that persists even after the trigger is gone, goodput stays unusably low, and a sustaining effect, often work amplification, keeps it there. Recovery needs a strong push such as a restart or a large cut in load. Their lifecycle has three states: stable, vulnerable (healthy, but one trigger from the bad state), and metastable. Many production systems run in the vulnerable state on purpose because it is more efficient.

Their worked example is small enough to do by hand. A database answers in under 100 ms below 300 QPS and an order of magnitude slower above it. The web tier sends one query per request, plus one retry if the first has not returned in 1 second. At 280 QPS a 10-second network outage hits the switch. When it ends, the queued requests and retries all arrive together, the database is overloaded, every query takes more than a second, and so every request now also sends a retry: the load is 2 x 280 = 560 QPS and the database never recovers. Goodput is zero. The paper's figure puts the stable region below 150 QPS (twice 150 is 300, so even with every request retried the database copes) and says recovery needs load under 150 QPS or retries held under 20 QPS. The trigger was a 10-second blip. What keeps the system down after it is the retry policy.

The same paper shows the cache version. A look-aside cache with a 90% hit rate lets the same database serve 3,000 QPS, since only one request in ten reaches it. Lose the cache while running above 300 QPS and every request now needs the database, a 10x amplification. The cache cannot refill because the slow database makes the fills time out. Hence the rule that a cache you depend on for capacity is part of your capacity, which connects to the stampede section of Caching.

:::key What a staff engineer says "It was blamed on the network blip, but the blip lasted ten seconds and the outage lasted an hour. The sustaining effect is the thing to fix." Ask of every outage: what kept it going after the trigger ended? :::

Retry amplification and retry budgets

Claim: a retry is a bet that the failure was transient. At scale, with many layers, the bet loses and the retries become the outage.

Rate Limiting already covered backoff, jitter and the 64-attempts example from the SRE book. The AWS Builders' Library gives the same effect in a deeper stack: with five layers and "three retries at each layer" it reports that the load on the database increases 243x, which makes recovery unlikely. (243 is 3^5, so the arithmetic counts three tries per layer; counting the first attempt plus three retries would be 4^5 = 1,024.) The principle both sources land on is that retries belong at one layer. AWS's guidance for low-cost operations is to retry at a single point in the stack, and the SRE book says to retry only at the layer directly above the one that is rejecting, so that the deeper layers fail fast.

Why layers cannot coordinate: each layer sees only "my call failed". The one above has no way to tell that the failure was the result of three retries below it. Fixing this takes a cap that does not depend on knowing the topology.

Retry budgets are that cap. The SRE book describes two:

• Per request: at most three attempts, then return the error to the caller.
• Per client: retry only while retries are under about 10% of the requests that client has sent. Without the cap the worst case approaches 3x load; with it, about 1.1x.

AWS describes the same idea as a token bucket kept in the client: each retry spends a token, successes refill it, and once the bucket is empty the client stops retrying (or retries at a fixed slow rate). AWS says it added this to its SDKs in 2016.

Worked arithmetic: three layers, each with a 10% retry budget, give at worst 1.1 x 1.1 x 1.1 = 1.33x load on the bottom service, against 4 x 4 x 4 = 64x for three retries per layer with no budget. That is the point of a budget: the multiplication still happens, but the base is 1.1, not 4.

A third tool is an explicit "do not retry" answer. The SRE book has backends return a different error when a large share of tasks is overloaded, so that callers stop retrying instead of treating it as a single unlucky task. Pair it with the status split in Rate Limiting: 429 means you are over your own share and slowing down helps; 503 means the service is full and the client should not add to it.

:::warn Retries and non-idempotent calls A retry that creates a second order is a correctness bug on top of a capacity bug. Reads retry freely. Writes need the client-generated idempotency key from Boundaries Are Where Guarantees Die. AWS's EC2 RunInstances call uses a client token for exactly this reason. :::

What the senior checks: the ratio of retried to first-attempt requests on the graph for every client library, and which layer owns retries for each call path. If two layers can both retry the same call, one of them is wrong.

Timeouts and deadlines propagated across hops

Claim: a timeout protects one caller. A deadline protects the whole call chain, and only the deadline stops work nobody is waiting for.

Choosing a timeout. AWS's advice: decide what rate of false timeouts you can live with (their example is 0.1%) and set the timeout at the matching percentile of the downstream's latency, here p99.9. Add room for clients with slow networks, and for services whose p99.9 is close to their median, since there a small slowdown crosses the line for everyone at once. They also report a service with a roughly 20 ms timeout that failed after every deployment, because new connections needed a secure handshake the timeout had not allowed for.

A timeout set too high is as bad as none: each stuck call holds a thread, a connection and memory. Little's law gives the cost. Worked arithmetic: a service takes 100 requests per second to a dependency and its thread pool is 200. If the dependency starts hanging and the timeout is 2 seconds, the calls in flight are 100 x 2 = 200, the whole pool. A 500 ms timeout holds 50. The timeout is a cap on how much of your capacity a sick dependency can borrow.

Deadlines. Independent timeouts at each hop are wrong in both directions: each hop can wait longer than the user will, and a deep hop can start work after its callers have given up. The fix in the SRE chapter is to pass the remaining deadline downstream. A request arrives with a 30-second deadline. Server A spends 7 seconds, so it calls B with 23. B spends 4, so it calls C with 19. If B instead used a hardcoded 20-second timeout, then whenever the earlier hops had used more than 10 of the 30 seconds, C would keep grinding on a request whose answer the user will never see. The chapter also says to subtract a few hundred milliseconds per hop for network transit.

Deadlines buy three behaviours that timeouts cannot:

• Early rejection. A server that receives a request with 40 ms left on a call that normally takes 200 ms can refuse at once and save the CPU.
• Cancellation. When the deadline passes, in-flight work upstream and downstream is cancelled rather than finishing and being ignored.
• A smarter retry gate. Retry only if the remaining time is longer than the call's typical latency (the last gate in the diagram above).

:::warn Timeouts that outlive the caller If the server queue is slower than the client timeout, every response is wasted. This is the same arithmetic as the leaky-bucket queue in Rate Limiting, applied to a whole chain. The SRE book's example is a server that takes 11 seconds to reach a request its client abandoned after 10. :::

What the senior checks: that the deadline travels in the request metadata (with HTTP that means a header you add and every service honours), and that the longest timeout in any call path is shorter than the user-facing one.

Load shedding: what to drop, by priority

Claim: when you cannot do everything, choose what to refuse on purpose, early and cheaply, instead of letting the queue choose by timing out.

Rate Limiting separated throttling (this caller is over quota) from shedding (the server is near its limit) and listed some priorities. Here is the mechanism. Two signals decide when to shed, and one rule decides what to shed.

When: in-flight work and queue age. CPU can lag, so the SRE book's chapter on overload describes using an executor load average: the number of threads that are active (running or waiting for a core), smoothed with exponential decay. Rejection starts when it exceeds the processors available. A simpler version is a cap on concurrent requests with an immediate 503 above it. The second signal is queue age. AWS recommends an upper bound on how long a request may wait, dropping the ones that are too old, and a LIFO queue where the protocol permits, so the newest requests, whose callers are most likely still waiting, go first.

Worked arithmetic: a service with 100 threads and 50 ms per request can do 100 / 0.05 = 2,000 requests per second. Offered 2,500 per second, a FIFO queue grows by 500 each second. After 2 seconds it holds 1,000 requests and a new arrival waits 1,000 / 2,000 = 0.5 s; the wait keeps growing until it passes the client timeout, and from then on nearly everything served is wasted. A rule of "drop anything older than 200 ms" caps the queue at 2,000 x 0.2 = 400 entries. The 20% of excess traffic is refused in microseconds and the other 80% is answered in time. Goodput stays at about 2,000 per second instead of falling.

What: criticality. Google tags every request with one of four levels, CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS and SHEDDABLE. Production traffic defaults to CRITICAL; batch defaults to SHEDDABLE_PLUS, because it can retry minutes or hours later. The tag travels with the request through every hop, so a shedder deep in the stack can drop a low-criticality sub-request without understanding its caller. Higher criticality has a higher utilization threshold before rejection starts.

AWS's priority order adds three more rules worth defending in an interview:

1. Health checks first. If the load balancer's ping goes unanswered, it pulls the server out of rotation, the fleet shrinks and the rest overload faster. Cap connections at less than the listener threads or file descriptors so pings still get through.
2. Finish what has started. A request for page 3 of a result, or the closing call of a multi-step operation, outranks a fresh page-1 request, because rejecting it wastes everything done so far (in AWS's example, a client that fails after page N-1 discards N-2 earlier calls).
3. In-quota before burst. Traffic above a client's throttle limit ranks below traffic within it.

Bronson and colleagues add a retry-specific rule: give retried queries a lower priority, so that fresh user requests succeed and the retry traffic dies out.

Batch traffic gets its own door. The SRE chapter routes large batch jobs through a separate proxy tier that absorbs the start-up surge of a big job and "acts like a fuse", so it can never starve interactive users.

:::warn Shedding must not lie in the metrics AWS's advice: keep fast-failed requests out of the latency graph. If you shed 60% of traffic, the median latency will look excellent while the 40% who got served are slow. Log each rejection with client and operation, alarm on the rejection volume, and aim for a false positive rate of zero: no shedding while there is spare capacity. :::

Shedding has a cost too. The Metastable Failures paper notes that shedding as a corrective action can itself disrupt users, which is why the policy should be rehearsed and not invented during the incident.

What the senior checks: a load test that goes well past the breaking point (AWS's target is goodput plateauing near full utilization and staying flat) and a dashboard showing rejects per client and per criticality.

Client-side throttling and adaptive backoff

Claim: the cheapest rejection is the one the client makes itself. A client that has seen the backend refuse most of its calls should stop asking for a while without spending a round trip.

A server that sheds still pays to receive, parse and reject every request, and at the top of the curve even that cost adds up. So Google's client library does adaptive throttling:

• Each client keeps two counts over the last two minutes: requests (everything the application tried) and accepts (what the backend accepted).
• While requests is under K x accepts, send everything. K = 2 is the book's preferred value.
• Above that, reject new requests locally with a probability that grows with the excess.

Worked arithmetic (a sketch of the rule, not Google's code): with K = 2, a client that tried 3,000 requests and had 1,000 accepted is over the cutoff of 2 x 1,000 = 2,000 by 1,000. Rejecting about 1,000 / 3,001, one request in three, locally brings the rate it sends to the backend back toward twice what the backend takes. With K = 1.1 the cutoff is 1,100 and the client is far more cautious; the book says the backend then rejects only about one request for every 10 it accepts. A lower K protects the backend harder and wastes more client capacity during a short blip.

Properties that make this work at fleet scale:

• No coordination. Every client decides from its own recent experience, so there is no central limiter to overload.
• It self-corrects. As the backend recovers, accepts rises and local rejection falls.
• It compounds with the retry budget. The budget limits retries, the throttle limits first attempts. Together they remove the 2x to 3x the backend would otherwise see.
• Unlike a fixed rate limit it needs no quota. It follows what the backend can actually do right now.

Adaptive backoff is the time-domain version. After a rejection the client waits, with the wait growing exponentially up to a cap and randomised (jitter) so that thousands of clients do not return in lockstep, which is the thundering-retry trap in Rate Limiting. AWS adds a point for periodic work: derive the jitter deterministically per host rather than randomly, so that any overload or race it causes repeats and can be debugged. On EBS and Lambda, AWS says jittering periodic traffic let the same work run with less server capacity.

The trade-off to say aloud: client-side throttling is only as good as the client. It protects you from well-behaved clients (your own services, your own SDK) and does nothing about hostile or buggy ones, which is why it complements server-side shedding rather than replacing it.

Bulkheads and circuit breakers with numbers

Claim: a bulkhead limits how much one failing thing can take. A circuit breaker limits how long you keep hitting it. Both fail in their own ways, and a mis-tuned breaker can be the outage.

Bulkheads. The Azure Architecture Center defines the pattern as isolating parts of an application into pools so that if one fails, the others keep working; the name comes from a ship's sectioned hull. Two forms:

• Consumer-side: a separate connection pool, thread pool or semaphore per dependency. If service A hangs, only the pool for A fills. Azure points to resilience4j and Polly as libraries.
• Service-side: separate instances (processes, containers, queues, whole cells) per consumer class or tenant, so one noisy client cannot exhaust the shared service.

Worked arithmetic: a service has a 200-thread pool shared by every dependency. Dependency X starts taking 2 seconds per call and receives 100 calls per second, so 100 x 2 = 200 threads are stuck on X and the service stops answering requests that never touch X. With a separate pool of 40 threads for X, X saturates its 40, further X calls fail instantly, and 160 threads keep serving everything else. The cost is idle capacity: 40 threads reserved for X are unavailable to others when X is quiet. Azure lists inefficient resource use as the reason not to use bulkheads where it is unacceptable. Use platform controls (Kubernetes CPU and memory limits, rate limits, per-tenant quotas) before writing your own.

Circuit breakers. The Azure pattern page describes a state machine:

Illustrative numbers (not a standard or any library's default): open when 50% of at least 20 calls in a 10-second window fail; stay open 30 seconds; allow 5 probes; require all 5 to succeed. Azure also suggests growing the open timeout (a few seconds first, then minutes) and returning a meaningful default instead of an error while open.

How breakers cause outages. These come from the Azure considerations and from AWS's own caution that breakers "introduce modal behavior into systems that can be difficult to test" and can add significant recovery time:

1. One breaker over independent resources. Azure warns about a data store with many shards: if one shard is unhealthy and the breaker merges all errors, the application is blocked from the healthy shards. Worked arithmetic: 16 shards, one down, 2,000 calls per second. Only 125 calls per second fail (6.25%) but if the trip rule is "50 failures in a window" the whole data store is refused. A 6% fault became a 100% outage. Keep a breaker per shard, per endpoint, per dependency.
2. Opening for the wrong errors. Azure notes that a timeout from an overloaded service and an outright "unavailable" differ in meaning. A breaker that trips on the caller's own quota rejection (429) can cut off a service that is healthy; Azure's own example does exactly that on purpose to protect a quota, so make it a decision, not a default. The error type should change the rule.
3. Open for too long. The dependency is back and the breaker still rejects. Azure lists this under recoverability, and the opposite problem too: switching to Half-Open too soon makes it flap.
4. Probes from every client at once. Worked arithmetic: 500 service instances each allow 5 probes in Half-Open, so a dependency that has just recovered gets 500 x 5 = 2,500 calls in an instant, can fail them, and sends everyone back to Open. Stagger the timers with jitter, allow a trickle (one probe, then a ramp) and use a health-check endpoint instead of real traffic where you can, which Azure lists as an option.
5. A long timeout inside the breaker. Azure: if the call timeout is long, threads block for that long before the breaker can count a failure, and many instances can tie up threads together before any breaker opens. The breaker is only as fast as the timeout beneath it.
6. Breakers on top of breakers and retries. With a retry above, the retry logic must stop when the breaker says the fault is not transient, or the retries pile onto the open breaker.

:::key Verified November 2026 The Azure bulkhead page (page dated March 2026) and circuit-breaker page (page dated February 2025) describe the states, the shard warning and the long-timeout warning above. AWS's timeouts-and-retries article states the "modal behavior" caution about breakers. The numbers in the example rule are illustrations, not defaults of any library. :::

What the senior checks: which breaker state changes show on a dashboard (Azure suggests raising an event per transition), that the tripping rule has a minimum call count so two failures at 3 a.m. do not trip it, and that the half-open path was tested in a game day, not just the open one.

Graceful degradation: serve less instead of failing

Claim: the aim of the whole stack is not to keep everything working, it is to keep the important part working while the rest is cut away on purpose.

Shedding says which requests to refuse. Degradation says what a request still gets when part of the system is missing. It is a product decision that engineering builds ahead of time, and the best answer to "what if the recommendations service is down" is a page, not a mechanism name.

A usable ladder, from least to most visible to the user:

The Azure circuit-breaker page shows the link: while the breaker is open on a throttled database, the application returns default or cached responses and tells users that some data is temporarily unavailable. That makes the open state a product feature and not an error.

Worked arithmetic on why degradation raises capacity and not only availability: if a full page costs 12 backend calls and 40 ms of CPU, and a degraded page costs 3 calls and 10 ms, each core serves four times as many degraded pages. Switching to the degraded page at the top of the curve raises the ceiling by 4x for the price of a less rich page. That is the same lever as shedding but it answers every user.

Rules that make it work:

• Decide the triggers in advance. Link the degraded modes to the signals that already exist: the breaker is open, the shedder is rejecting, the queue age passes a limit. A human flipping a flag during an incident is a fallback, not the plan.
• The failure path must be cheaper than the success path. Bronson and colleagues call out slow error handling as a sustaining effect: stack traces, a DNS lookup to name the client, and logging to a central service are all expensive and all run at their worst moment. A degraded path that calls the sick dependency again to produce its fallback is not degraded.
• Exercise it. AWS flags modal behaviour as hard to test, and the SRE chapter says to load test components until they break, with gradual and sudden load, and to run production tests such as quickly dropping task counts or blackholing a backend. A fallback that has never run will fail the first time it matters.
• Mind cold starts. The SRE chapter says to increase load slowly onto restarted tasks while caches warm, and to overprovision services that depend on caches for their capacity (the 90% hit rate example in the first section).
• Tell the client. The shared language in the SRE book is that the response carries the information that it is degraded or that the backend is overloaded, so layers above can adapt and not retry.

Interview move: when asked to design for failure, name the one thing that must never break (checkout, the payment, the login), then walk the ladder for everything else. That separates a design from a list of patterns.

Case: how a small outage becomes a large one

Claim: in a real cascading failure, the trigger is almost always small and boring. The size comes from the sustaining loops, and every defence in this lesson maps to a missing link.

The case is Amazon's own summary of the DynamoDB event in US-East on Sunday, 20 September 2015. Everything below is from that post.

Setup. DynamoDB storage servers hold partitions. Each one asks an internal metadata service for its membership, the list of partitions it should serve, and must get the answer within a time limit. In the previous months, customers had adopted global secondary indexes. Each index has its own partitions, so a table's contribution to membership "could quickly double or triple". Membership lists had grown large, and AWS says it had no monitoring on membership size and too little metadata capacity for the heavier requests. The system was in a vulnerable state in the paper's sense.

Trigger, 2:19 a.m. PDT. A brief network disruption affected some storage servers. That is normally absorbed without any change in performance.

Mechanism.

1. The affected servers asked the metadata service for membership at once. Large answers took longer, and some passed the storage servers' time limit.
2. Servers that missed the limit retried and removed themselves from the request path. Many large requests at once slowed processing further and pushed more servers over the limit.
3. Retries kept the load high. Healthy servers, on their normal check-in schedule, got slow responses, failed their renewals and went unavailable too. Servers that had recovered cycled back to unavailable.
4. By 2:37 a.m., about 18 minutes after the trigger, the error rate on customer requests reached about 55%, "far beyond any level" seen in the previous three years.

Why recovery was slow. AWS's early attempts to add metadata capacity failed because the overloaded service could not accept the administrative requests. The control path shared the overload with the data path. At 5:06 a.m., AWS paused requests to the metadata service, which cut the retry volume. With the load gone, administrative requests worked, and AWS added significant capacity, re-enabled requests and let the storage servers return. By 7:10 a.m. error rates were low enough for most customers and dependent services to resume. A residual 0.15% to 0.25% error rate on Monday traced to one metadata partition that was not yet getting its full share of traffic.

The blast radius. Services that depended on DynamoDB failed in their own ways. SQS send and receive kept working, but its metadata cache refreshes failed while DynamoDB traffic was disabled (no queue data was lost); EC2 Auto Scaling had API faults from 2:15 to 7:10 a.m. and delayed scaling until 10:52 a.m.; CloudWatch lost metrics; and Console logins slowed for tens of seconds because of a very long timeout on a call to a DynamoDB-dependent API.

Reading it against the defences.

Fixes AWS listed. More metadata capacity immediately; monitoring of dimensions like membership size; slowing the rate at which storage nodes request membership and giving queries more time; and in the long term segmenting DynamoDB into many metadata service instances, each serving a portion of the fleet, to contain failures. Three of the four are the lesson's defences, and the fourth is a bulkhead.

:::key Verified November 2026 Times, the 55% figure, the Sunday-to-Monday residual rate and the corrective actions are from Amazon's published event summary for 20 September 2015. :::

What the senior takes away: write down, for your own critical shared service, the answer to "if every client retried at once for ten minutes, what stays up?" and "how would we get the administrative tools in during overload?" The second question is the one the 2015 event answered the hard way.

Further reading: Site Reliability Engineering: How Google Runs Production Systems, edited by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy, chapters 21 (Handling Overload) and 22 (Addressing Cascading Failures), free at sre.google; the AWS Builders' Library articles on timeouts and retries and on load shedding; and the Metastable Failures paper (HotOS 2021).

How to say it in the interview

"My first question about any overloaded service is whether goodput is flat or falling, because if it falls, adding load makes things worse and the system is in a feedback loop. I bound every queue by age, I give each request a deadline that shrinks as it crosses hops, and I allow retries only in one layer, capped at about 10% of traffic, since retrying at every layer multiplies. When we are past capacity I shed low-criticality work early and cheaply, keep health checks answering, and isolate each dependency in its own pool so one slow neighbour cannot take all the threads. Then I ask what the degraded page looks like, because the best defence against a total outage is a partial one we planned."

Checklist

• Showed the goodput curve and said which mechanism turns overload into a loop (queues, exhaustion, amplification, death spiral)
• Asked what sustains an outage after its trigger ends, and named the retry or cache loop
• Put retries in exactly one layer, capped by a per-request limit and a budget near 10% of traffic
• Set timeouts from a latency percentile and propagated a remaining deadline across hops
• Bounded queues by age, shed lowest criticality first, and kept health checks answering
• Kept fast-failed requests out of latency metrics and alarmed on rejection volume
• Added client-side adaptive throttling (accepts versus K x requests) alongside server-side shedding
• Gave each dependency its own pool or breaker, per shard and per endpoint, with minimum call counts and jittered half-open probes
• Defined a degradation ladder with automatic triggers and exercised it in a game day
• Kept an operator path that overload cannot starve, and a way to cut load from outside

Related lessons

• SLAs, Availability Math and Error Budgets as Design Inputs
• Designing for Observability
• Metrics Pipeline
• Distributed Tracing System
• Disaster Recovery: Backup, Pilot Light, Warm Standby, Active-Active