Cruxly › System design › Reliability and Observability

Disaster Recovery: Backup, Pilot Light, Warm Standby, Active-Active

decide what counts as a disaster, pay only for the recovery that its RPO and RTO justify, and prove it works before the day you need it

Disaster recovery (DR) is the part of a design that answers one question: when the thing we depend on is gone, how much do we lose, how long are we down, and who presses the button? This lesson treats it as a system design decision, not a database setting. You choose the disaster you plan for, a recovery point and time per tier, one of four strategies that cost very different amounts, and a failover path that does not depend on what just broke.

It builds on Operating a database: backups, migrations, multi-region, cost (RPO, RTO, backups, point-in-time recovery), Replication and Consistency, Distributed Consensus, How top companies run their databases (the GitHub 2018 and GitLab 2017 stories, linked here rather than retold) and Running an Incident: Roles, Communication, and Not Making It Worse. The overload angle is in Rate Limiting and Scaling Up, Scaling Out and Splitting. Further reading: Site Reliability Engineering by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), chapters 22 (Addressing Cascading Failures), 23 (Managing Critical State: Distributed Consensus for Reliability) and 26 (Data Integrity: What You Read Is What You Wrote).

Define the disaster: host, zone, region, or the data itself

A disaster is defined by what was lost, not by the headline cause, and the scope you pick decides everything that follows. Google's disaster recovery planning guide makes the same point: requirements are framed around the scope of an outage (zonal, regional, multi-region), and the guide notes that infrastructure failures usually cause an outage in a single zone, while a regional outage affects more than one zone in a region. Before any strategy, write down which of these you are designing against.

The last two rows are the ones replication does not fix, and they are the ones candidates forget. The AWS whitepaper says it directly: continuous replication protects against some disasters but may not protect against data corruption or destruction unless the strategy also includes versioning or point-in-time recovery, and it recommends a different account per Region so that compromised credentials are not part of the disaster. Operating a database: backups, migrations, multi-region, cost shows the database-level mechanics (a replica is not a backup; base backup plus archived WAL); this lesson uses them as one layer of a larger plan.

Most outages are not disasters. A zone failing is an ordinary day for a well-built system, and it should cost nothing a user notices. Calling every incident a disaster is how teams end up failing over to a second region for a problem a rollback would have fixed, which is the most expensive way to be wrong (see the failover section).

What the senior engineer says in the room: "Which disaster? Zone loss is the baseline and costs nothing visible. Region loss is a business decision with a price. Data loss is a different problem that no amount of replicas solves." What a senior checks: that every tier has a recovery path for the data-corruption case, not only for the infrastructure-loss case, and that the backup account is not the production account.

RPO and RTO per tier

RPO and RTO are set per tier by what an hour of the loss costs, and the answer is almost never the same for the whole product. Operating a database: backups, migrations, multi-region, cost defines both: RPO is how much data you can lose, measured in time; RTO is how long you can be down. Google's guide puts the business question plainly: RTO is how long after a disaster before you are up and running; RPO is how much data you can afford to lose. It adds that lowering either target raises cost, and that near-zero regional RTO changes what it costs to run and maintain the service.

Two numbers per tier, chosen by a conversation, not a default. An example split (all values are assumptions to illustrate the shape, not benchmarks):

Google's guide gives a banding that agrees with this: for RTOs between an hour and a day a warm failover is a reasonable shape (databases running in standby, web servers scaled out after the disaster), and for RTOs over a day a service can often be recovered from a backup or recreated from scratch.

RTO is not one number, it is a sum. Detect that something is wrong, decide to act, change routing, bring up capacity, warm caches and let traffic ramp. Worked arithmetic with assumed values: 5 minutes to detect, 10 to decide, 2 to switch traffic, 8 to scale the standby and 5 to warm up gives 30 minutes, so an RTO of 15 is not reachable by that design however good the database promotion is. Adding it up is the first thing a senior does with an RTO claim.

Restore time scales with data size, and it is the number people never measure. Worked arithmetic, assumed values: restoring a 5 TB database at an effective 200 MB/s takes 5,000,000 MB / 200 MB/s = 25,000 s, about 6.9 hours, before replaying logs and before any application checks. A "2-hour RTO, nightly backup" design on that data is a plan that cannot be met. The remedy is a standby that is already restored (pilot light or warm standby), not a faster hope.

A service's tier cannot beat its dependencies. If checkout (tier 0) calls a tax service that is tier 2 and has no regional standby, checkout's real RTO is the tax service's. Draw the dependency graph and either raise the dependency or degrade gracefully when it is absent (accept the order, compute tax later).

:::warn The same tier for everything is the expensive mistake Giving the whole product tier-0 recovery multiplies the DR bill by the cost of running everything warm in two regions. Giving everything tier 2 loses the ledger for hours. Tiering is the work. :::

What the senior engineer says in the room: "RPO and RTO per tier, in minutes, from the product owner, then I sum the RTO and check the restore time of the biggest store." What a senior checks: the measured restore time of the largest data store against its tier's RTO, and the dependency graph for any tier-0 service that relies on a lower tier.

The four strategies and what each one costs

The AWS whitepaper groups DR into four strategies that run from the cheap and slow to the costly and fast, and the choice is a price for a recovery time. Three are active/passive: one site serves traffic and the passive one is used only for recovery. The fourth serves from several regions.

The AWS whitepaper's own distinction between pilot light and warm standby is the useful one: pilot light cannot process requests without additional action, while warm standby can handle traffic, at reduced capacity, immediately. Pilot light means you turn servers on, perhaps deploy more infrastructure and scale up; warm standby only means you scale up. It also names a hot standby: enough capacity for the full load in the second region, but serving no users until needed.

What each costs, in the dimension that matters.

• Backup and restore costs storage and transfer, and the lowest standing compute bill. It is the right answer for tiers 2 and 3, and the whitepaper says it also suits a well-built, highly available workload where the disaster you plan for is the loss of one data center. Its weakness is time: infrastructure redeploy plus data restore, with the restore growing with data size (see the arithmetic above). The whitepaper insists on infrastructure as code here, because without it redeploying in the recovery region is complex and may exceed the RTO.
• Pilot light keeps the data stores live and replicating, so RPO is the replication lag. It changes how you deploy: infrastructure changes and code changes must reach both regions together, and the whitepaper recommends automating that with infrastructure as code (full deployment to the primary, scaled-down or switched-off deployment to the recovery region).
• Warm standby is the usual tier-0 and tier-1 answer when users can tolerate minutes: an always-on small copy is also easy to test continuously, which the whitepaper lists as an advantage. The sting is the scale-up. See the static stability section: the standby that has to grow during the disaster depends on the very control plane that is likely to be impaired.
• Active-active is, in the whitepaper's words, the most complex and costly approach, and it can reduce recovery time to near zero for most disasters. It also makes you answer the write question (next section). The whitepaper adds that data corruption may still have to rely on backups, which means a non-zero recovery point: replication does not make an active-active system immune to a bad write.

How the price grows (worked arithmetic, assumed costs). If the production stack costs 100 units a month: backup and restore adds maybe 5 to 10 for storage and transfer; pilot light adds the always-on data tier, say 15 to 25; warm standby at a quarter of capacity adds about 40 to 60 once data and the small fleet are counted; active-active at full capacity in each of two regions roughly doubles the compute (and, as the static stability section shows, each region must be able to carry the whole load). The point is the order of magnitude of each step, which is what the owner needs to see, not these exact figures.

What the senior engineer says in the room: "Per tier, the cheapest strategy that meets the RTO. Most of the estate is backup and restore; only the tier-0 path pays for a warm copy; active-active only where users must not notice a region going." What a senior checks: that the second region has actually been deployed with the same infrastructure code as the first, with quotas raised there, and that the copy in the second region has served real traffic recently.

Cross-region data replication and the consistency it costs

Replication across regions is a latency-versus-loss trade you make once and then live with, and it decides your RPO. Replication and Consistency explains why: a leader that waits for a remote acknowledgement pays the round trip on every write, and one that does not can lose the tail. Across regions the round trip is tens to hundreds of milliseconds, so the tail is real.

Asynchronous replication: RPO equals the lag at the moment of failure. Google's guide notes that synchronous replication carries cost and latency trade-offs, which is why most cross-region designs are asynchronous. Worked arithmetic with assumed values: a primary taking 4,000 writes per second whose replica is 2 seconds behind has about 8,000 acknowledged writes at risk if the region disappears right then. Monitor the lag as a number against the tier's RPO. AWS says Aurora global database can replicate to a secondary region with typical latency under a second and lets you monitor the RPO lag of the secondaries; treat that as a vendor figure for that product, not a general one.

:::key Verified November 2026 AWS whitepaper "Disaster Recovery of Workloads on AWS": Aurora global database replicates to secondary Regions with typical latency of under a second, and a secondary Region can be promoted to read/write in less than one minute, even in a complete regional outage. For RDS instances other than Aurora, promoting a cross-region read replica takes a few minutes and includes a reboot. S3 replication into a DR region, with versioning enabled, keeps earlier versions; by default a delete in the source bucket adds a delete marker in the source only, so a malicious deletion there does not reach the DR copy. These are product behaviours on that date and change. :::

Active-active forces the write question. The whitepaper names three patterns and they are worth reciting:

• Read local, write global. Reads are served from the nearest region; every write goes to one region. If that region is lost, another is promoted. Simple and consistent, with the failover (and its lag) as the cost. This is the single-writer design in Operating a database: backups, migrations, multi-region, cost.
• Write local. Each region accepts writes. The whitepaper's example is DynamoDB global tables, which reconcile concurrent updates with last writer wins. Fast and available, but concurrent writes to the same item silently lose one: acceptable for a profile name, wrong for a balance.
• Write partitioned. Each key (a user, a tenant) is assigned a home region, so a given item is only ever written in one place and conflicts do not arise. The cost is routing every request to the key's home region, and a plan for what happens to that key when its home region dies.

What replication does not change. None of the three patterns makes a bad write recoverable. The whitepaper says that in an active-active system recovery from corruption or deletion still takes non-zero time, and that the recovery point is a time before the disaster was discovered. Keep point-in-time backups separate from the replication stream.

Two regions cannot form a majority. If you want a consensus-backed store to survive the loss of a region without human intervention, you need a quorum that survives it: with replicas in two regions, losing one leaves no majority, which is why Distributed Consensus style systems spread across three failure domains or use a tie-breaker (a witness) elsewhere. The alternative is a manual promotion with its split-brain risk, covered below.

Failing back is a second migration. After the failover the old primary has data the new one never saw (the lag tail), and the new one has newer data. Reconciling them is slow and hard; How top companies run their databases covers GitHub's October 2018 failover, where divergent writes in two data centers had to be reconciled from binary logs, and the choice to prioritise data integrity over a fast return. Plan failback as a real project, not as the reverse of failover.

What the senior engineer says in the room: "Async to the second region, lag alerted against RPO, writes to one home per key, and a separate point-in-time copy that replication cannot touch." What a senior checks: replication lag as a graph with an alert at the RPO, and whether anything in the system relies on cross-region writes being instantly visible.

Failing over: DNS, global routing, control plane versus data plane

Failover itself is the riskiest step in disaster recovery, because it is the one operation you run rarely, under pressure, against a system that is already degraded. It can cause a worse outage than the fault: a false alarm costs the same recovery time and potential data loss as a real one. The AWS whitepaper says so: automatically initiated failover from health checks or alarms should be used with caution; recovery time and recovery point are greater than zero, so an unneeded failover incurs those losses for nothing, which is why manually initiated failover is often used, with the steps automated so the manual initiation is like pushing a button.

Control plane versus data plane is the idea to bring to any failover design. The whitepaper's phrasing: the data plane delivers real-time service, while control planes are used to configure the environment; data planes typically have higher availability goals; so for maximum resilience use only data-plane operations in the failover. The AWS static stability article defines the same split for EC2: the control plane is the machinery for making changes (launching instances, allocating network interfaces, installing security group rules), the data plane is the daily business of those resources (routing packets, reading and writing volumes), and the data plane is simpler and aims higher. The practical consequence: do not make a failover depend on "create a VM", "change a record through the console" or "scale this group", because those are the calls most likely to be throttled or down when a region is unhealthy. Google's guide gives the same warning for business-critical operations with RTO and RPO targets: avoid depending on management-plane operations such as VM creation.

Google's guide adds the human side: manual changes to IP addresses or even DNS updates can take longer than expected, and for Cloud SQL, promoting a cross-region read replica during a regional outage is a manual operation because promotion can cause a split-brain scenario.

Split-brain is what automatic promotion without a quorum causes. If the old primary is only unreachable, not dead, and the standby is promoted, two writers exist. A fencing mechanism (the old primary is forced to stop accepting writes) or a quorum decision (see Distributed Consensus) prevents it. GitHub's 2018 story, linked from How top companies run their databases, is a case where a consensus-based failover tool moved the primary to a place the application could not use well.

The failover has side effects that kill the standby. When traffic arrives all at once at a region whose caches are cold and whose database connection pools are empty, the surviving side sees a load spike it never saw before. The standby needs headroom, warmed caches, and a way to ramp: shift traffic in steps (10%, 50%, 100%) rather than one switch, and make load shedding ready (Rate Limiting, and the priority shedding discussed in Scaling Up, Scaling Out and Splitting) so that the first minutes protect the ledger, not the recommendations.

What the senior engineer says in the room: "Manual trigger, automated steps, data plane only, staged ramp, fenced old primary. The thing I am afraid of is the failover, not the outage." What a senior checks: that the failover script has been run end to end in the last quarter, that it needs no console clicks, and the DNS TTL against the RTO.

Static stability: do not need to scale during the disaster

A statically stable system keeps working when a dependency is impaired without needing to change anything, so its recovery path has no steps that can fail. The AWS Builders' Library article defines it: the system keeps doing what it was doing before the dependency became impaired, though it may miss updates. Its approach is to prepare before the impairment, not to react to it. The EC2 example is the cleanest: if an instance is already running, an EC2 control-plane outage does not stop it serving, but if recovery requires launching a replacement, you depend on the control plane right when it may not respond.

The rule: pre-provision, do not scale. A standby that must scale up during the disaster has a hidden dependency on the launch API, on quotas in the recovery region, on image availability and on there being capacity in the region. The AWS DR whitepaper says it plainly: because Auto Scaling is a control-plane activity, depending on it lowers the resiliency of your recovery strategy; provisioning enough capacity for the full production load is the statically stable choice (hot standby), and the cheaper alternative takes the dependency, with some designs deploying enough for the first traffic and relying on scaling to ramp afterwards. It also says to set service quotas in the recovery region high enough that they do not block scaling.

The arithmetic of over-provisioning (zone level). The Builders' Library article uses three Availability Zones and over-provisions by 50 percent, so that each zone runs at about 66 percent of its load-tested capacity in normal times. Worked arithmetic with an assumed fleet: the peak needs 12 instances at the safe load. If a zone is lost, the two surviving zones must carry all 12, so 6 each, for 18 in total. In normal operation 12 / 18 = 66.7 percent of the capacity is in use, matching the article. With two zones you would need 2 x 12 = 24 (100 percent over-provisioned) to survive losing one, which is why three zones is the efficient number.

At region level the same arithmetic is worse. For two active-active regions that must each survive the other's loss, each region carries the whole load, so total capacity is 200 percent of the peak (worked arithmetic). That is the real cost line behind "active-active roughly doubles compute". With three regions, losing one leaves two to carry the whole load, half each, so total capacity is 150 percent of peak, at the price of three-way data placement.

Calls between services can erode it. The article's arithmetic: if a request crosses two regional services and each is hit independently by an impaired zone, the chance of avoiding the impaired zone is (2/3) x (2/3) = 4/9, worse than even; with zonal services that stay inside a zone the chance stays at 2/3, and in general it is (2/3)^N versus a constant. It recommends regional-calls-regional for most services and zonal designs for the critical data-plane components. The lesson for DR is that a hop count multiplies the chance of touching the broken part.

Static stability also means pre-resolved decisions. Configuration the standby needs (secrets, certificates, feature flags, routing tables, the list of endpoints) should already be in the standby region, not fetched from the primary region at failover time. A standby that has to read its config from the region that just died is not standby.

:::warn Autoscaling is not a DR plan Scaling reacts after the load arrives (see Scaling Up, Scaling Out and Splitting for the lag), through a control plane that is among the first things to degrade in a regional event. Use it to trim cost after you are safe, not as the way you get safe. :::

What the senior engineer says in the room: "Pre-provision for the failure we promise to survive, then use autoscaling for the cost. Anything that needs a launch call during the outage is a step that can fail." What a senior checks: the zone-loss capacity maths (survivors under the safe utilisation at peak), the quotas in the recovery region, and that standby config is local to the standby.

Testing: restore drills, game days, failure injection

A recovery plan that has not been exercised is a guess, and untested backups are the most common form of the guess. Google's guide says to test regional-outage mitigations regularly, warning that without testing teams may find out it does not work only when it happens for real. The AWS whitepaper says it is critical to regularly assess and test the DR strategy so you have confidence invoking it. GitLab's 2017 database outage, told in How top companies run their databases, is the standard illustration: several backup and replication mechanisms existed, and the one that was needed turned out not to be usable.

Three kinds of test, each answering a different question.

Restore drills deserve a schedule and automation. The AWS whitepaper notes that restoring from backup is a control-plane operation, so it recommends scheduling periodic restores into the recovery region: if the operation were unavailable in a disaster you would still hold operable data stores built from a recent backup. A drill that restores nightly into a scratch instance and runs a verification query turns "we have backups" into "we know the restore works and takes this long" (the measured number belongs next to the RTO).

Game days exercise the whole chain, not the database. The tests that find the real problems are the ones that include humans: does the on-call know who may decide, is the runbook current, can they reach the tooling without the region that is down, are the quotas in the recovery region enough, does the traffic ramp hold. GitHub's post-incident list in the 2018 story (see How top companies run their databases) includes fault-injection and chaos testing among its remedies.

Start where it is safe, and end where it is real. A sensible ladder, as an engineering suggestion rather than a sourced standard: restore a backup into scratch (no production impact); run the failover script against a staging copy; fail over a low-risk tier in production; fail over a whole tier-0 path at low traffic; and finally evacuate a region during business hours with staff ready. The AWS whitepaper notes that a warm standby makes continuous testing easier, because it already runs, and that for active-active the test focuses on whether traffic is routed away from the failed region and whether the remaining regions can take all of it; a test for a data disaster is still needed.

An untested backup is not a backup. Corruption in the backup itself, a missing encryption key, a backup that stopped silently months ago and a restore that exceeds the RTO are all found only by restoring. Record each drill as a dated result: time to restore, data checked, problems found, owner of each fix.

What the senior engineer says in the room: "We restore on a schedule, we fail over a real tier each quarter, and I can show you the last measured restore time." What a senior checks: the date and result of the last restore drill and the last failover exercise, and whether the drill used production-sized data.

Runbook and decision rights

Failover needs an owner who is allowed to decide before the incident starts, and a runbook that works when the primary region is gone. The technical design can be perfect and the recovery will still be slow if nobody is sure they may press the button. Running an Incident: Roles, Communication, and Not Making It Worse covers the roles (the incident commander coordinates and does not debug; communication is a separate job); this section adds what DR specifically needs on top.

Decision rights, decided in advance.

• Who may order a regional failover. Name a role (the incident commander on duty, or a named engineering director for tier-0 failover), not a person, and say who is the deputy. A failover that waits for the one executive who is asleep has an RTO set by their phone.
• What the criteria are. Write the trigger: which signals, for how long, with what scope. For example (an assumption to show the shape): region-wide elevated errors on tier 0 for 15 minutes with no rollback available and the provider reporting a regional event. Criteria prevent both panic failovers and a long wait.
• A time-box. If the primary cannot be fixed or rolled back in a set time, fail over. This turns a debate into a clock and matches the AWS note that a false alarm costs the same as a real failure.
• Who owns failback, and when (it is a separate, calmer decision, see the earlier section).

The runbook is code-adjacent and must survive the disaster.

• Keep it, and the tooling and credentials it needs, outside the region and account that may fail, and reachable when your single sign-on or your chat service is the thing that is down (a break-glass path that is tested).
• Make the steps executable: a script or a pipeline with a dry-run, not a document that says "update DNS". The AWS whitepaper's phrase for manually initiated failover is that the manual initiation should be "like the push of a button".
• Order the steps by risk. Check the replication lag first (that is the data you are about to give up), then fence the old primary, then promote, then shift traffic in steps, then verify. Put a go/no-go gate before the irreversible step.
• Write the recovery steps after the outage too. Google's guide asks, of post-outage recovery: what do I need to do to recover after an outage is resolved, and who does it?

Communication is part of the plan. The status page, customer support and internal teams need a script for "we are failing over, expect X minutes of errors and Y writes at risk". Pre-written templates save the minutes that the commander does not have.

Close the loop with the incident process. After any real or practised failover, run the blameless review described in Running an Incident: Roles, Communication, and Not Making It Worse and its postmortem lesson: what the measured RTO was against the target, what steps deviated, which assumption failed. The target is rewritten if the measured number says it was never achievable.

What the senior engineer says in the room: "The person who can order the failover is named, the criteria are written down, there is a clock, and the runbook is a script we ran last month." What a senior checks: that the runbook lives outside the failure domain, the date it last ran end to end, and that on-call can name who decides without looking it up.

How to say it in the interview

"First I ask what we are recovering from: a host, a zone, a region, or the data itself, because replication fixes the first three and copies the fourth. Then I get an RPO and RTO per tier from the business, not one number for the whole product. For the payments ledger I would take warm standby in a second region with asynchronous replication, accept that RPO is the replication lag at the moment of failure, and keep point-in-time backups in another account because a replica copies a bad delete. Failover is a push-button script that uses only data-plane operations, the standby is sized so it does not need to scale during the disaster, and we run a game day every quarter, because a backup nobody has restored is a hope."

Checklist

• State which disaster each tier is designed for (host, zone, region, account, data) and give the data-corruption case its own recovery path
• Get an RPO and RTO per tier from the business, in minutes, and sum the RTO from detect, decide, switch, scale and warm-up
• Measure the restore time of the largest data store at production size and write it next to its RTO
• Pick the cheapest of backup and restore, pilot light, warm standby or active-active that meets each tier's RTO
• Keep point-in-time backups in a separate account and region, versioned or immutable, apart from the replication stream
• Alert on cross-region replication lag against the tier's RPO, and write down the write pattern (write global, local or partitioned)
• Make failover a scripted, human-triggered, data-plane-only action with a fenced old primary and a staged traffic ramp
• Pre-provision the standby for the failure you promise to survive, with quotas, images and config already in that region
• Run scheduled restore drills and at least a quarterly failover exercise, and record the measured RTO and problems found
• Name the role that may order failover, write the trigger criteria and a time-box, and keep the runbook and tools outside the failure domain

Related lessons

• SLAs, Availability Math and Error Budgets as Design Inputs
• Designing for Observability
• Metrics Pipeline
• Distributed Tracing System
• Overload, Load Shedding and Cascading Failure