Cruxly › System design › Databases by Name: Internals and Terms

How top companies run their databases: postmortems, migrations and what to copy

ten sourced stories, read as an engineer: version, number, failure, what changed

The per-database lessons in this topic (for example PostgreSQL internals: MVCC cost, VACUUM, wraparound and WAL and Cassandra and ScyllaDB: the ring, repair, compaction and Discord's migration) teach mechanics. This closing lesson takes the stories that cross them: GitHub, GitLab, Sentry, AWS, Slack, Discord, Shopify, Stripe, LinkedIn and Cloudflare, each with a date, a published number and a change. It ends with the six causes that keep recurring, so you can test your own design against them.

How to read a company database post

A company's database post is a measurement of one workload, at one version, on one date, often written to explain a decision already made. Quote the number with its date and say who measured it.

Uber's July 2016 post by Evan Klitzke, "Why Uber Engineering Switched from Postgres to MySQL", is the best exercise in careful reading because it is the most quoted and the most misquoted. It analysed Postgres 9.2, and says that, to the authors' knowledge, the on-disk design had not changed significantly since at least 8.3; the post says the current generally available release was 9.5. The claims, as the post makes them:

• Write amplification. A small logical update needed at least four physical writes to the tablespace, because every index gets a new entry even when its key did not change, and each write is also logged in the WAL.
• Replication. Physical replication makes a verbose WAL stream. Cross-datacenter links strained and WAL archiving fell behind at peak traffic.
• Replica MVCC. The post says Postgres lacks true replica MVCC, so long replica queries can block WAL application and get cancelled after a timeout.
• A 9.2 replication bug. Some replicas misapplied WAL records during a timeline switch after a promotion, leaving superseded rows visible and returning duplicates.
• Upgrades. Physical replication does not work across releases; the 9.1 to 9.2 upgrade took many hours of downtime-prone steps.
• Connections. Process per connection cost memory; the team struggled past a few hundred active connections and used pgbouncer.

The other side, labelled as such. Robert Haas, a PostgreSQL committer, replied on 2 August 2016 on his own blog ("Uber's Move Away from PostgreSQL"). His points: he agreed write amplification from secondary indexes was the hardest problem Uber raised, said a non-HOT update must touch every index and that HOT is an optimisation and not a full answer, and argued PostgreSQL would need pluggable storage rather than a bigger HOT. He asked how much secondary-index traffic Uber really had and whether all those indexes were needed. He suspected Uber wanted logical replication, which core PostgreSQL then lacked (he named out-of-core tools and asked whether Uber tried them), and that replica query cancellation can be prevented with hot_standby_feedback=on, which he did not know whether Uber had tried. On upgrades, he said many hours for pg_upgrade should not happen and that later releases had fixes for databases with many objects. He agreed that Uber had identified areas where PostgreSQL can be improved, while still calling it a very good database.

:::key Verified October 2026: check on your version Haas's points are 2016 statements about PostgreSQL then. What current PostgreSQL does about each one is not covered here, so do not read his reply as a statement about today's version. Read the current release notes before you take either side of this debate into a room. :::

What this teaches. Three habits. First, put the version and year next to every number: "Postgres 9.2, 2016", not "Postgres". Second, separate three kinds of statement: what the company measured (four writes per update), what it concluded (switch engines), and what the community answered. Third, mark arithmetic you derive yourself. Quoting Uber as "Postgres does not scale" is wrong because the post is about one fleet, a set of 2016 limits, and a tradeoff in which secondary-index-heavy updates and cross-region replication were the pain.

Other date and version traps in this lesson. Discord's numbers come from 2017 and 2022 posts; Cloudflare's ClickHouse numbers are from March 2018; Sentry's "fewer than one million transactions left" is its 2015 reading, while the PostgreSQL 18 routine-vacuuming docs, read October 2026, say the system refuses new transaction IDs once fewer than three million remain. When a primary doc and an old post disagree, quote the doc and say the older number was different.

What experts say in the room. "A post is evidence about one workload at one date." The numbers: Postgres 9.2 against the 9.5 then current. The failure is quoting a verdict instead of a mechanism. The argument you will have: whether the company's problem was the engine or the way they used it (the index count, the replication topology, the missing pooler).

What a senior checks. What has changed in the product since the post, what the post's workload is compared with ours, and which numbers are measured, claimed or derived.

Failover and backup failures: GitHub 2018, GitLab 2017

Two famous outages share one lesson: the mechanism you built for availability or recovery behaved exactly as configured, and the configuration was the flaw. GitHub's failover moved the primary somewhere the application could not use; GitLab's four backup and replication mechanisms were not a tested restore.

GitHub, 21 to 22 October 2018. Source: GitHub's "October 21 post-incident analysis", published 30 October 2018.

At 22:52 UTC, maintenance replacing failing 100G optical equipment cut connectivity between the US East Coast network hub and the primary East Coast data center. Connectivity returned in 43 seconds. In that window, Orchestrator (the failover tool GitHub's post describes as working by Raft consensus among its nodes) in the primary data center began to deselect itself as leader, while Orchestrator nodes in the West Coast data center and East Coast public cloud formed a quorum and failed database clusters over to West Coast primaries. When connectivity returned, applications wrote to the new primaries. The East Coast primaries held writes not yet replicated, so failing back was not safe: both sides had data the other lacked.

Responders found the topologies in an unexpected state at 23:02, locked deployment tooling at 23:07, and at 23:19 paused webhook delivery and Pages builds to protect data already received. The post says the team found about 40 minutes of West Coast writes plus the unreplicated East Coast ones; on the busiest cluster it counted 954 writes not replicated. The recovery plan at 00:05 was to restore from backups, sync replicas in both sites, fall back to a stable topology and then resume queued jobs. By 00:41 a backup restore process was already running for every affected MySQL cluster. Clusters ranged from hundreds of GB to nearly 5 TB, backups ran every four hours, and the post names the slowest step: restoring multiple terabytes from remote cloud storage, then decompressing, checksumming and loading them. By 11:12 all primaries were back in the East Coast, but dozens of read replicas were hours behind, so users saw inconsistent data. The post notes that catch-up followed a power-decay curve, not a straight line, and peak traffic slowed it. Replicas were in sync at 16:24; the backlog included more than 5 million hook events and 80,000 Pages builds; about 200,000 webhook payloads exceeded an internal TTL and were dropped. Status returned to green at 23:03, 24 hours 11 minutes after it started (our arithmetic: 22:52 to 23:03 the next day).

The decision GitHub explains is the interesting one: it chose to fail forward and preserve data integrity over fast recovery, even though cross-country writes made the site slow. Its listed changes: configure Orchestrator to prevent promoting a primary across regional boundaries, build more active/active/active serving so a whole facility can fail, run fault-injection and chaos testing, reconcile the divergent writes from each cluster's binlogs, and publish better component-level status.

GitLab, 31 January 2017. Source: GitLab's "Postmortem of database outage of January 31".

A load spike (what GitLab suspected was spam, plus a background job removing an account that had been scheduled for removal by mistake, per the post) made replication lag, then fail, because the primary had removed WAL segments the secondary still needed and WAL archiving was not on. Rebuilding the secondary with pg_basebackup hung silently; max_wal_senders was raised from 3 to 32, and PostgreSQL then refused to restart until max_connections went from 8,000 to 2,000. Around 23:30 UTC an engineer, believing it was the secondary, wiped the data directory on the primary; the process was stopped after a second or two, but about 300 GB had been removed. Then the recovery paths failed one by one, as the post lists them:

Recovery used the 6-hour-old staging snapshot, copied from slow disks. The post gives the copy as roughly 18 hours at about 60 Mbps (our arithmetic: that is on the order of 490 GB moved). It lists about 5,000 projects, 5,000 comments and about 700 users affected, and says all database sequences were raised by 100,000 to prevent ID reuse. The post says no one owned testing the backup procedure.

What changed afterwards, in both: GitHub moved to constrain promotion and test failure; GitLab's own post is a list of the missing owner, alerts and monitoring. The common shape: a backup is a restore you have tested, and automatic promotion is a decision you constrain in advance. Operating a database: backups, migrations, multi-region, cost covers the mechanics; this is the field evidence.

What experts say in the room. "Integrity over availability, and say it in advance." The numbers are 24 hours 11 minutes and about 18 hours. The failure is the restore nobody tested. The argument is about automation: a human-gated failover is slower, an unconstrained automatic one can create a state worse than the outage.

What a senior checks. The date of the last full restore drill and its measured time against the stated recovery objective, the constraints on automatic promotion (region, replication lag, a human gate), and whether failed backup jobs page someone through a channel that works.

Maintenance debt: Sentry's wraparound

Background maintenance is a production dependency. Sentry's July 2015 outage was a database that stopped accepting writes because autovacuum, tuned too conservatively and not alerted on, fell behind a 32-bit counter.

Source: Sentry's post "Transaction ID Wraparound in Postgres" (Monday, 20 July 2015), and the PostgreSQL 18 routine-vacuuming docs (read October 2026) for the thresholds.

Mechanism. Every row-modifying transaction takes a 32-bit transaction ID, compared modulo 2^32. The docs say that for any normal ID there are two billion older and two billion newer, so every table must be vacuumed (frozen) at least once every two billion transactions or old rows would appear to be in the future. autovacuum_freeze_max_age defaults to 200 million transactions (the docs mention the default in their storage-cost note) and has a maximum of two billion. The docs say warnings start when about forty million transactions remain, and the server refuses to assign new transaction IDs once fewer than three million remain. In that state, in-flight transactions continue and read-only ones can start, but anything that modifies data fails. The deeper mechanics are in PostgreSQL internals: MVCC cost, VACUUM, wraparound and WAL; this section is the operational story.

What Sentry reported. The primary reached the protection limit and stopped accepting writes (the post quotes "fewer than one million transactions left", the older margin). They failed over to new hardware and ran vacuums there. Autovacuums ran for hours, one did not complete and the logs gave no clear reason, and about three hours in they found every relation done except one large table, which they truncated, accepting data loss (it mapped events to rollups and supported duplicate detection; full backups existed on replicas). The system recovered about five minutes later. The team did not complete a single-user-mode vacuum on production; two days later a test machine on older hardware was still running one after about 24 hours. Before the fix: autovacuum_freeze_max_age was too high, autovacuum used the default 3 workers, vacuum cost delay was too high so the system idled during maintenance, and verbose logging was off. After:

The post's update also corrects that vacuum_freeze_table_age was not updated and was removed from the config.

:::key Verified October 2026: two numbers differ from the post Sentry's "fewer than one million" is superseded by the three million in the PostgreSQL 18 docs. The docs also now say it is neither necessary nor desirable to stop the postmaster or enter single-user mode to recover; they say to resolve old prepared transactions, end long transactions, drop stale replication slots, and run VACUUM in the affected database. Sentry's 500 million is a 2015 choice for its workload; the docs' default is 200 million. :::

Arithmetic, not Sentry's or the docs' numbers. 2^31 is 2,147,483,648. At an illustrative 1,000 transaction IDs per second, 200 million lasts about 2.3 days, two billion about 23 days, and the final three million margin about 50 minutes. At 5,000 per second the margin is 10 minutes. A tripled workload turns the same configuration into a three-times-shorter runway, which is why the alert is on age, not on a calendar.

What experts say in the room. "Autovacuum is a production dependency, and it is silent when it fails." The numbers: 200 million default, 500 million at Sentry, 40 million warnings, 3 million refusal. The failure is no alert on transaction ID age. The argument: raise the freeze age to avoid vacuum I/O, or lower it and vacuum earlier in smaller bites; a higher value saves work now and gets you a bigger emergency later.

What a senior checks. An alert on the age of the oldest unfrozen transaction ID per database with a runway measured in hours at current ID consumption, the oldest long transaction, prepared transaction and replication slot (each can hold back freezing), and autovacuum logging turned on.

Retry storms and memory limits: AWS 2015, Slack's job queue

Two incidents from different systems failed the same way: a shared dependency got slow, clients retried, and the retries kept it slow. The fix in both was to put a buffer or a pause between the callers and the dependency.

AWS DynamoDB, Sunday 20 September 2015. Sources: the AWS post-event page for the DynamoDB disruption (the page gives the day but not the year; AWS's Post-Event Summaries index lists the event as 20 September 2015).

DynamoDB storage servers fetch their membership, which partitions they own, from an internal metadata service, and if the fetch does not return within a time limit they retry and temporarily stop accepting requests. The page says the rapid adoption of Global Secondary Indexes had increased partition counts and so enlarged membership lists; responses took longer and neared the time limit, and the metadata service had no monitoring of membership size and not enough capacity for these heavier requests. At 2:19 am PDT a brief network disruption hit some storage servers; the simultaneous large requests pushed processing past the limit, retries added load, and healthy servers due for routine renewal began failing too. By 2:37 the customer error rate reached about 55 percent. Adding metadata capacity failed at first because the service was overloaded. At 5:06 am requests to the metadata service were paused, which reduced retry load and let administrative requests through; capacity was then added, requests re-enabled and by 7:10 most customers could resume (4 hours 51 minutes after the start, our arithmetic). A residual 0.15 to 0.25 percent error rate lingered into Monday from one metadata partition receiving less traffic than intended. Dependants were hit too: SQS, EC2 Auto Scaling, CloudWatch and console logins, each of which AWS lists with its own timeline.

AWS's stated changes: much more metadata capacity, monitoring of dimensions such as membership size, a lower rate at which storage nodes request membership, longer processing time for those queries, and the longer-term plan to segment the metadata service into many instances, each serving a share of the storage fleet, to contain failures. The mechanics of partitions and adaptive capacity are in DynamoDB under the hood: partitions, adaptive capacity, quotas and the 2015 outage.

Slack's job queue, December 2017 post. Source: Slack Engineering, "Scaling Slack's Job Queue". The post says the outage was "about a year ago"; it does not give a date.

Contention in the database layer slowed job execution. Jobs drained more slowly than they were enqueued, and the Redis queue hit its configured maximum memory, so enqueueing failed and every Slack operation that depended on it failed. Dequeue also failed: it moves a job to an in-flight list and needs free memory, so the queue stayed locked even after the database was fixed, and recovery took heavy manual work. The post gives the scale: more than 1.4 billion jobs on the busiest days, 33,000 per second at peak (our arithmetic: 1.4 billion over 86,400 seconds averages about 16,200 per second, so the peak is about twice the average). It lists the structural problems (little memory headroom, every client connecting to every Redis instance, workers unable to scale independently of Redis, dequeue cost growing with queue length). The redesign put Kafka in front of Redis instead of replacing it: a stateless Go service (Kafkagate) accepts enqueues and writes to Kafka, waiting only for the partition leader's acknowledgement, which the post says deliberately favours availability and carries a small risk of loss if a broker dies before replication; a second Go service (JQRelay) relays each Kafka topic to its Redis cluster under a Consul lock, advancing offsets only after the Redis write, with rate limits read from Consul. The cluster was 16 Kafka brokers, 32 partitions per topic, replication factor 3, 2-day retention, and heartbeat canaries every minute on all 1,600 partitions (50 topics times 32). The rollout used double writes with the relay in shadow mode, count reconciliation at every hop, internal dogfooding and a job-type-by-job-type cutover.

What experts say in the room. "Shared metadata and in-memory queues fail by amplification." The numbers: 55 percent error rate, 33,000 jobs per second. The failure is a cascade of retries. The argument: where the buffer lives (a durable log in front of a fast store, as Slack did) against making the fast store bigger. For Kafka mechanics see Kafka as a log store: ISR, acks, compaction, transactions and LinkedIn scale.

What a senior checks. Retry budgets and jittered backoff on every client of a shared service, a way to shed or pause load to the dependency (AWS's 5:06 pause), memory headroom on in-memory queues as a correctness limit, not a cost one, and monitoring on the size of whatever a metadata service has to serve.

Migrations: Discord, Slack, Shopify, Stripe

Live migrations are protocols, and the good ones look alike: copy, tail the changes, verify, switch with a short gate, keep a way back. The expensive mistakes are skipping verification and assuming the engine swap is the whole job.

The shared sequence. Shopify's Ghostferry (post: "MySQL database shard balancing at terabyte scale") and Stripe's Data Movement Platform (2023 post on DocDB) are the same shape:

Shopify's stated reason to move shops: they grow unevenly; the post says the least and most used shards differed by almost 4x, and a simulated rebalance narrowed it to about 2x. Stripe's 2023 figures, from the same post (published June 2024): over 5 million queries per second, 2,000+ shards, 5,000+ collections, 99.999 percent uptime, built on MongoDB Community and extended in-house; a 2023 bin-packing migration moved 1.5 petabytes and reduced shards by about three quarters. Arithmetic: 5,000,000 over 2,000 shards is an average of 2,500 queries per second per shard, which is an average, not a limit, because the load is uneven by design. The Mongo mechanics are in MongoDB internals: oplog, WiredTiger, shard keys, the balancer and Stripe's DocDB.

Discord: fix the application layer, then swap the engine. Source: Discord's "How Discord Stores Trillions of Messages" (2022). Discord grew from 12 Cassandra nodes in 2017 to 177 holding trillions of messages by early 2022, with hot partitions (a heavy channel and time bucket raising latency across the cluster at quorum), compaction backlogs and JVM garbage-collection pauses. Before the engine swap it built a Rust data-services layer between the API and the databases with no business logic: it coalesces concurrent requests for the same row into one query and routes by channel id with consistent hashing so a channel's requests meet in one instance. The migration to ScyllaDB used a Rust migrator: a Spark estimate was about three months and the Rust one nine days, at up to 3.2 million messages per second (our arithmetic: nine days at that peak is under 2.5 trillion messages, consistent with "trillions"). It stalled at 99.9999 percent because the last token ranges contained large numbers of uncompacted tombstones in Cassandra; compacting that range let it finish. The primary switched in May 2022 and the post reports 72 ScyllaDB nodes with 9 TB disks each, against 177 Cassandra nodes averaging 4 TB. Our arithmetic: 177 to 72 is 59 percent fewer nodes, but 72 x 9 is 648 TB against 177 x 4 = 708 TB of disk, so the saving is per-node density and operations, not total bytes. The post reports p99 for historical message fetches falling from 40 to 125 ms to 15 ms, and inserts from 5 to 70 ms to a steady 5 ms. See Cassandra and ScyllaDB: the ring, repair, compaction and Discord's migration.

Slack on Vitess: years, not quarters. Source: "Scaling Datastores at Slack with Vitess" (1 December 2020). The migration began in 2017, and the post says Vitess served 99 percent of Slack's MySQL traffic at the time of writing, with the last 1 percent expected within about two months. Peak load was 2.3 million queries per second, about 2 million reads and 300,000 writes, with median 2 ms and p99 11 ms. Slack's own reasons: workspace-based sharding could not split a large team's load, building channel sharding into the application was judged too coupled and too slow, and Vitess kept MySQL while adding horizontal sharding, automated failover and topology management. The migration was table by table, with a generic backfill system and a double-read diffing system against the legacy databases. Problems the post lists: the application was tightly coupled to the storage layout, thousands of queries used MySQL-specific constructs that had compatibility gaps, and Vitess's topology metadata had scalability limits. In March 2020's usage surge it resharded the busiest keyspace using Vitess splitting workflows. See MySQL and InnoDB: locks, buffer pool, replication and Vitess.

The design question, not a company claim. Application-layer sharding against trusting the cluster layer: Slack's and Discord's posts both describe application code coupled to storage, and moved routing into a layer (Vitess, data services). Stripe and Shopify keep placement logic in their own control plane. No source here says one wins, so frame it as: who owns the routing table, who can move a shard live, and what is the verification story.

What experts say in the room. "Migrations are protocols." Numbers: 3.2 million per second, 2,000 shards. The failure: a long tail like Discord's tombstones at 99.9999 percent. The argument: dual-write plus diff (Slack) or snapshot plus change stream (Shopify, Stripe).

What a senior checks. A verifier that runs independently of the copier, a defined rollback path after cutover, a timeout on the cutover gate with a failed-closed outcome, and the tail case: the slowest token range, hot shop, or biggest chunk.

Scale snapshots worth knowing

Know the orders of magnitude so you can say whether your system is ordinary or exceptional. These are the published figures, each with its date, and the arithmetic you can derive from them.

Worked arithmetic. LinkedIn: 7 trillion a day is about 81 million messages per second on average; spread across 4,000 brokers it is about 20,000 per broker per second, ignoring replication and traffic skew. Seven million partitions over 4,000 brokers is about 1,750 per broker. Stripe: 2,500 queries per second per shard on average. Prime Day: 89.2 million per second is more than 38 times Slack's peak of 2.3 million (89.2 / 2.3 = 38.8). Cloudflare: 11 million rows per second over 36 nodes is about 305,000 per node.

What LinkedIn adds beyond the headline. The post says LinkedIn runs internal release branches on top of Apache Kafka, kept as close to upstream as possible; that large clusters suffered slow controllers and memory-pressure failures; that slow broker startup and shutdown delayed deployments because only one broker can be taken down at a time; and that a maintenance mode stops new partitions or replicas from being assigned to a broker so its replicas can move off before removal. It also added a minimum replication factor at topic creation to reduce data-loss risk. These are the kind of operating costs a headline number hides.

How to use the table. Compare your peak with these. A product at 5,000 queries per second on one Postgres primary is far below every number above, and the lesson is that most systems do not need Vitess, Kafka at LinkedIn scale or a custom shard mover. A system at 500,000 writes per second does need to say how it partitions. Designing for ten times less than you need or ten times more than you will ever see are both failures of estimation, and the interviewer is checking which one you tend to make.

What experts say in the room. "Know the orders of magnitude." The failure is quoting a headline number without its date or its denominator (a peak, an average, a cluster, a fleet). Argument: whether the largest company's architecture is a template or a warning; most of the posts above are about doing something expensive because the simple option ran out.

What a senior checks. Own peak and average per database, growth over the next 12 months, and which rung of Scaling Postgres and MySQL before you leave them the system is on.

Patterns to take into the interview: six repeating causes

The incidents above keep returning to six causes. Map any design to all six before you claim it is reliable.

Using it. Take the system you are designing and ask six questions: when did we last restore, and how long did it take; what exactly can promote a new primary and what is it forbidden to do; which maintenance process failing silently would hurt us, and what alerts on it; what are our retry limits against each shared dependency; which key can become hot and what happens to quorum if it does; and which artifact is generated from a query and what bounds it? An answer with a number and an owner for all six is stronger than an architecture diagram.

A second pattern across the stories: recovery needed the resource that had failed. GitHub's restore was bounded by pulling multiple terabytes from remote storage, then decompressing and loading them. GitLab's copy ran at about 60 Mbps for about 18 hours. AWS could not add metadata capacity at first because the metadata service was overloaded, and only a pause of requests let the fix in. Slack's dequeue needed free memory in the very Redis that was full. Sentry's maintenance_work_mem of 10 GB was limited by an internal 1 GB cap on the relevant paths. So for each failure in your design, ask a seventh question: what does the recovery path itself consume, and is that resource still available when everything else is failing? Measure the restore, keep an out-of-band path to the control plane, and keep spare headroom exactly where the repair will draw from.

What to copy, in one line each. From GitHub: write down in advance whether you prefer integrity or availability. From GitLab: assign an owner to backups. From Slack: put a durable log in front of a fast, bounded store. From Discord: coalesce duplicate reads and route by key before changing engines. From Shopify and Stripe: build moves as copy, tail, verify, gate, with a way back. From Cloudflare 2025: treat a generated file like untrusted input.

The quiz idea made explicit. Classify GitHub 2018 as cause 2; GitLab 2017 as cause 1; Sentry 2015 as cause 3; AWS 2015 as cause 4; Cloudflare 2025 as cause 6. Several incidents hit two: GitHub also included slow restore from remote storage (cause 1's cousin), and Slack's queue is cause 4 with a memory limit.

:::key The six, as one sentence each Restores you never ran, promotions nobody limited, maintenance nobody watched, retries nobody budgeted, keys that went hot, and files nobody validated. Each maps to a runbook item in the earlier lessons of this topic. :::

What experts say in the room. "Failures rhyme." The six are the number. The failure is designing only the happy path and presenting redundancy as if it were recovery.

What a senior checks. That each of the six has a named owner, a dashboard or drill with a date on it, and a written threshold, and that the list is revisited after every incident, including ones elsewhere.

How to say it in the interview

"A company database post is evidence about one workload on one date, so I quote the version and the year with the number. GitHub's October 2018 failover took 43 seconds of partition and 24 hours 11 minutes to heal, because an automatic promotion across coasts left both sides holding writes the other lacked. GitLab's 2017 outage showed four backup and replication mechanisms, none of them a tested restore. Sentry's 2015 wraparound was autovacuum tuned too lazily and not alerted on. The failures rhyme, so I map any design to six causes: an untested restore, an unconstrained failover, unmonitored maintenance, retry amplification, a hot partition under quorum, and a generated config past a hard limit."

Checklist

• Every company number I quote carries its source, year and version, and I say whether it is measured, claimed or our arithmetic
• The date and measured duration of our last full restore drill are written down and compared with the recovery objective
• Automatic promotion is limited by region and replication lag, and reconciliation of divergent writes is planned
• An alert fires on transaction ID age (or the equivalent maintenance debt) with a runway in hours
• Backup failure alerts are tested end to end, including email authentication or paging paths
• Every client of a shared metadata or coordination service has retry budgets, backoff with jitter and a pause switch
• In-memory queues have memory headroom treated as a correctness limit, and a durable buffer sits in front
• The migration protocol is written down: copy, tail, verify, gate, rollback, with an independent verifier
• Any config generated from query results has validation, size limits, a canary and a last-known-good fallback
• I can compare our own peak queries per second and write rate with the published scale figures and name our rung of the scaling ladder

Related lessons

• PostgreSQL internals: MVCC cost, VACUUM, wraparound and WAL
• MySQL and InnoDB: locks, buffer pool, replication and Vitess
• Redis in depth: cluster slots, eviction, persistence and the lock debate
• Cassandra and ScyllaDB: the ring, repair, compaction and Discord's migration
• DynamoDB under the hood: partitions, adaptive capacity, quotas and the 2015 outage
• MongoDB internals: oplog, WiredTiger, shard keys, the balancer and Stripe's DocDB