Cruxly › System design › Databases by Name: Internals and Terms

DynamoDB under the hood: partitions, adaptive capacity, quotas and the 2015 outage

a managed key-value store whose limits are per partition, and whose worst day came from a metadata service

DynamoDB looks like an infinitely scalable hash table, but it is a fleet of small replicated partitions behind routers, each with its own throughput ceiling. This lesson explains how a write is replicated, how admission control and burst decide who is throttled, which quotas you hit in real projects, and what the September 2015 outage shows about a metadata service. Key design and single-table modelling are in NoSQL modelling in practice; they are not repeated here.

Partitions, replication and the write path

A DynamoDB table is many small replication groups, not one big structure. The 2022 USENIX ATC paper by Elhemali and colleagues describes how: every partition is replicated across three availability zones, the replicas form a group, and a leader is chosen with Multi-Paxos. Only the leader serves writes and strongly consistent reads. A write is acknowledged once a quorum, two of the three replicas, has it.

Request path. A request arrives at a request router, which asks a metadata service which partition owns the key, then forwards it to the storage node holding the partition's leader. The paper describes the service as tens of microservices (routers, storage nodes, metadata, an auto-admin service). Remember the metadata service: it returns in the 2015 outage section.

Why this explains behaviour you can observe

• Eventually consistent reads cost half. They can be served by any replica, so the leader is not the only server that can answer. Strongly consistent reads must go to the leader. That is the mechanism behind the pricing difference, not a billing quirk.
• A lost leader is a short write stall. The group must elect a new leader before writes resume.
• Writes are limited by one leader per partition. Adding replicas does not add write throughput; adding partitions does.

Log replicas. The paper describes a second kind of replica that stores only the recent write-ahead log, with no key-value data, acting like a Paxos acceptor. Because it holds only the recent log, it is cheap to add. When a storage replica is unhealthy, adding a log replica restores a write quorum fast, long before a full copy of the data could be rebuilt. This is why a failed machine does not mean minutes of reduced durability while a big partition copies.

Scale statement. The paper reports that over the 66-hour Prime Day 2021 event, Amazon's systems made trillions of calls to DynamoDB and peaked at 89.2 million requests per second, with single-digit millisecond latency. Use it when someone asks "does it scale"; the answer comes from partitioning, not from a bigger machine.

Point-in-time restore. You can restore any time in the previous 35 days into a new table. The paper describes it as periodic snapshots plus write-ahead logs archived to S3, so a restore loads a snapshot and replays logs to the requested second. It never overwrites the live table, which makes a restore drill safe to run.

:::key Verified October 2026 Replication across 3 AZs, 2-of-3 write quorum, 35-day PITR window and the 89.2 million requests per second figure come from the 2022 ATC paper. Product details change; re-read the paper and the PITR docs before quoting them in a design review. :::

What experts say in the room. The argument: it is Paxos groups behind a router, so the useful questions are about per-partition limits and failover, not about "infinite scale". The numbers: 3 replicas, quorum 2, 89.2 million requests per second at peak on Prime Day 2021. The failure: leader loss causes a brief write stall, and strong reads wait with it.

What a senior checks. Whether each read path really needs strong consistency, because that choice doubles its read cost and pins it to the leader. Whether point-in-time recovery is switched on for every table that matters and a restore into a new table has been rehearsed. Throttle metrics broken down by table and index, covered next.

Admission control, burst and adaptive capacity

The throughput you provision is a table-level promise; the throughput you can use is bounded per partition. The DynamoDB best-practices page on partition-key design states the ceiling: a partition can serve up to 3,000 read units and 1,000 write units per second. Item size multiplies the cost. A strongly consistent read of a 20 KB item costs 5 read units, so one partition can serve only 3,000 / 5 = 600 such reads per second on that item. The division is arithmetic on the page's numbers.

Admission control is the service-side check that decides whether a request is allowed to run. The paper traces the history from provisioning at the partition level to on-demand in three steps.

1. Burst capacity. A partition that does not use its allocation keeps the unused part for up to 300 seconds and may spend it on a spike. It is best effort, not a guarantee. A test that hits the table hard for a minute succeeds on banked capacity and then a sustained run throttles. As arithmetic (not an AWS figure), a partition allocated 500 write units that sat idle could bank at most 500 x 300 = 150,000 units, enough to run at double its allocation for 300 seconds.
2. Adaptive capacity. Bursting could not absorb skew that lasts longer than 300 seconds, so the paper describes adaptive capacity, which shifts provisioned capacity toward hot partitions. The paper and the partition-key page both say it applies to provisioned and on-demand tables. It helps uneven load across keys, and the paper reports it removed over 99.99 percent of throttling caused by skewed access. It was reactive, though: it boosted partitions only after throttling had started, and like burst it was best effort. It does not raise the 3,000 / 1,000 ceiling for one partition, so one hot item is still throttled.
3. Global admission control. A later redesign decoupled table-level throughput from a fixed per-partition allocation, with admission checks against the table's total. The paper describes the path from partition-level provisioning through burst and adaptive capacity to on-demand. It is why a modern table throttles far less than a 2012 one for the same uneven load.

Table-level defaults. Read this from the DynamoDB Service Quotas page (read October 2026): on-demand tables start at 40,000 read request units and 40,000 write request units per table; provisioned tables at 40,000 read and 40,000 write capacity units per table, with an account total of 80,000 each. All of these are adjustable, and the page says they are initial defaults, not maximums.

Arithmetic, not AWS's number. A 40,000-write table needs at least 40,000 / 1,000 = 40 partitions' worth of write ceiling, and 40,000 reads need at least 13.3, so 14, partitions at 3,000. AWS does not publish partition counts, so treat this as a floor that explains why spreading keys is mandatory.

Decreases. For provisioned tables, increases are allowed as often as needed, but decreases are rationed: 4 available at the start of each UTC day, plus 1 more each hour, with a maximum of 4 banked. Across a full day that is 4 + 23 = 27. A cost-saving script that scales down every few minutes will be rejected after the fourth call. A table and each of its GSIs have separate counters, but a single request that decreases both is rejected if either is out of budget. Switching a table from provisioned to on-demand mode is also limited, to four times in a rolling 24 hours.

:::warn A hot item is throttled in every mode On-demand removes capacity planning, not the partition ceiling. A single item read at 800 strong reads per second when it is 20 KB costs 4,000 read units per second, above the 3,000 limit. One design question is a cache in front, a copy of the item under several keys, or smaller items, not by switching capacity mode. :::

What experts say in the room. The argument: provisioned versus on-demand is a cost and burst-shape decision, not a scaling one. Provisioned with auto scaling is cheaper for steady load; on-demand is simpler for unpredictable load. The numbers: 3,000 and 1,000 per partition, 300 seconds of burst, 40,000 default table quota. The failure: a hot item throttled in on-demand mode, with a team blaming the mode.

What a senior checks. ThrottledRequests per table and per index, consumed versus provisioned capacity, and Contributor Insights to name the hot keys. Whether the sustained peak, not the test peak, exceeds what 300 seconds of burst plus adaptive capacity can cover. Whether the table quota is above the launch forecast, requested days ahead.

Quotas and limits seniors cite

Quotas are design inputs, not details for later. In a design review, a staff engineer who cites the numbers saves a quarter of rework. These come from the DynamoDB Service Quotas page, read October 2026.

:::key Verified October 2026 Quotas are per Region unless noted, default values, and change. The page says size units are binary (1 KB = 1024 bytes). Recheck before a launch. :::

GSI fan-out. Worked arithmetic: a GSI is a maintained copy of the data. An item that carries the key attributes of all 20 allowed GSIs can trigger one write to the base table and 20 more to indexes, 21 in total. Even a few GSIs triple write cost, and the quota page tracks each GSI's throughput separately. That is why indexes are added for named queries only.

Table-per-tenant designs. At 2,500 tables per Region as the initial quota, a SaaS design with a table per customer fails at customer 2,501 unless you request an increase, with a cap of 10,000, and then needs more AWS accounts. The design alternative is a shared table with a tenant id in the partition key, which trades the table quota for a noisy-neighbour risk you must manage.

Per-item and per-call limits (DynamoDB constraints page, October 2026).

• Item size: 400 KB at most, counting attribute names as well as values. Large payloads belong in S3 with a pointer in the item.
• Batches and queries: BatchWriteItem takes up to 25 put or delete requests, BatchGetItem up to 100 items, each capped at 16 MB in total; a Query or Scan returns at most 1 MB per call, so you page with LastEvaluatedKey.
• Transactions: at most 100 unique items and 4 MB, and a transactional read or write costs twice the capacity of the plain operation.
• Local secondary indexes: a table with one or more LSIs cannot let any one partition key's item collection (base items plus LSI entries) exceed 10 GB, because an LSI shares the partition. A quiet table can hit this wall in production as a write error on a popular key.

What experts say in the room. The argument: know the limits before you propose the design. The numbers: 20 GSIs, 5 LSIs, 2,500 tables, 40,000 per table. The failure: surprise throttling on an index, or a table creation that fails on a Friday launch.

What a senior checks. A quota headroom dashboard (tables, GSIs, account throughput) with alerts at 70 percent, and quota-increase requests filed well before launch.

Streams, global tables and the limits that bite integrations

The failures that surprise teams are in the integrations, not the table. Three limits from the Service Quotas page (read October 2026) show up in real designs.

DynamoDB Streams readers. For single-Region tables you can design for up to 2 simultaneous processes reading the same stream shard; exceeding that can throttle requests. For global tables the page recommends 1. A third consumer (an audit service, an analytics loader and a Lambda trigger already make three) will therefore be throttled in a way that looks like random lag. The usual fix is a fan-out: one reader that publishes to a queue or a log, and the others subscribe there.

Global tables. The quota page lists two consistency modes, multi-Region eventual consistency (MREC) and multi-Region strong consistency (MRSC), with 40,000 read and 40,000 write request or capacity units per table in each, and 400 MRSC global tables in total. New replicas are backfilled from existing data, with a quota of 10 TB per account, per Region, per day. Two traps are written in the page's own notes:

• Global table throughput quotas must be equal to or greater than the per-table quotas for replica creation to succeed.
• The table-level write throughput limit must be the same in every Region that has a replica. If you raised it in one Region, raise it in every other replica Region before adding the replica, otherwise creation can fail with an error saying the replica exceeds your account limit in the other Region.

The 10 TB backfill. Worked arithmetic: adding replicas that total more than 10 TB to one destination Region within 24 hours needs a quota increase first. A 25 TB table cannot be replicated to a new Region in a day under the default; it takes at least 3 days (25 / 10 = 2.5, rounded up) or a raised quota.

Streams and write capacity. The page lists 40,000 write capacity units per table for tables with Streams enabled in provisioned mode, the same as the default table write quota.

:::warn Mismatched Region quotas fail late Replica creation is the step that reveals a quota mismatch, usually in the middle of a migration. Compare quotas across all target Regions in the design phase, not at cutover. :::

What experts say in the room. The argument: know the limits before you propose the integration pattern. The numbers: 2 stream readers (1 for global tables), 10 TB per day of replica backfill. The failure: a third stream consumer throttled, or a replica that cannot be created because quotas differ by Region.

What a senior checks. How many consumers read each stream and whether a fan-out layer is in place, MREC versus MRSC chosen on a stated business reason (the quota page treats them as separate modes with separate throughput quotas, so the choice is a design decision, not a default), and the quota comparison across Regions recorded in the design doc.

The September 2015 outage: metadata, GSIs and retry storms

On Sunday 20 September 2015 a service that looked stateless failed because its metadata service was overloaded by its own retries. AWS's post, titled "Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region", gives its times in Pacific time (PDT).

Mechanism, in the post's terms.

1. Storage servers ask a membership (metadata) service which partitions they should serve. If the answer takes longer than their time limit, they remove themselves from service and retry.
2. Rapid adoption of global secondary indexes had made those membership lists bigger, so the answer took longer to compute and send. The post says list size was nearing the processing time limit, and monitoring did not track it.
3. At 2:19am a brief network disruption hit some storage servers. Many requested membership at the same moment; the metadata service, already close to its limit, answered slowly. Servers timed out and retried, adding load, and the overload spread to servers that had been healthy.

Published numbers. Customer error rate reached about 55 percent by 2:37am (18 minutes after the trigger) and stabilised there. At 5:06am metadata requests were paused to reduce retry load so capacity could be added. Traffic was disabled while capacity was added (finished at 5:45am) and re-enabled at 7:10am, when error rates fell far enough for most dependent services to resume. Counting from 2:19am to 7:10am gives 4 hours 51 minutes (worked arithmetic; rounded, about 4 h 50 m). After that Sunday recovery a residual error rate of 0.15 to 0.25 percent remained for some customers; AWS closed it out on the Monday by restoring a metadata partition that still carried too little traffic.

Knock-on effects named in the post. EC2 Auto Scaling API faults began at 2:15am and ran until 7:10am, CloudWatch delivered delayed or missing metrics for part of the window, and the console had trouble with logins. A data store used as a dependency by other services spreads its failure to them.

What AWS changed. The post lists: much more capacity for the metadata service (already done), stricter monitoring of dimensions such as membership size, fewer membership requests from storage nodes and longer allowed processing time, and, longer term, segmenting DynamoDB into many metadata service instances, each serving part of the fleet. SQS, a dependent service whose cache refreshes failed after 5:45am, also changed its metadata caching so it would not need continuous access to the metadata table.

:::key Say the generic lesson Retries without backoff and without headroom turn a brief blip into an outage. Any shared metadata service needs a capacity signal that grows with the thing it describes (here, the number of indexes) and a way to shed load. The fix named for the blast radius was segmentation, not just more capacity. :::

What experts say in the room. The argument: control-plane metadata is a single point of failure for a data plane that looks stateless. The numbers: about 55 percent errors, roughly 4 h 51 m. The failure: a retry storm against shared metadata.

What a senior checks. Per-dependency retry budgets with exponential backoff and jitter in SDK clients, a capacity signal for every shared metadata service that scales with entities (indexes, tenants, partitions), and a documented way to pause or shed load at the source. For any managed database, ask what its own dependencies are and what happens when they are slow rather than down.

What a staff engineer checks before running DynamoDB at scale

Close the lesson as a runbook. Everything above becomes a list you can ask a team in a review.

Interview framing. When asked "why not just use DynamoDB", answer with a trade-off: it removes operations, replication and failover, and it makes you design every query up front, pay per request, and live with per-partition ceilings and quotas. Name which quota will bind first for the workload under discussion.

Replication, retries and the metadata question in an interview

A Say-it-in-the-room walk through. A common prompt: "Design a session store on DynamoDB for 200,000 requests per second." Answer in this order.

1. Key. Session id as the partition key spreads load; avoid a low-cardinality key such as the day.
2. Per-partition arithmetic. If each session read is a small item, count read units. Check no single session sees more than the 3,000 / 1,000 per second ceiling; one session will not.
3. Quotas. 200,000 is above the 40,000 default table quota, so request an increase early, or shard sessions over several tables or accounts if a quota cannot be raised in time.
4. Failure. Ask what the clients do when the service answers slowly: bounded retries, backoff with jitter, and a circuit breaker so the 2015 loop cannot form from your side.
5. Regional failure. Global tables, with MREC for latency or MRSC for strong consistency, and equal quotas in every Region.

:::key What the staff answer adds It names the number that binds first (table quota, hot partition, GSI writes or stream readers), how you would see it coming, and what you would do about it before launch. :::

How to say it in the interview

"DynamoDB is not a magic hash table. Each partition is a replication group of three replicas across availability zones, a leader chosen by Multi-Paxos takes the writes, and two of three must acknowledge. Each partition tops out at 3,000 read units and 1,000 write units per second, so a single hot item throttles in on-demand mode as well. Burst banks about 300 seconds of unused capacity and adaptive capacity moves capacity toward hot partitions, but neither lifts that ceiling. Before launch I raise the table quota above the 40,000 default and watch throttle metrics by key."

Checklist

• Said that each partition is a three-AZ Multi-Paxos group with a 2-of-3 write quorum
• Checked every hot key against 3,000 read and 1,000 write units per second, including item size
• Separated burst (about 300 seconds), adaptive capacity and the table quota
• Compared the forecast against the 40,000 per-table default and filed the increase early
• Counted GSIs against the default of 20 and provisioned their write capacity
• Planned stream consumers around about 2 readers per shard, with a fan-out layer
• Matched write throughput quotas across all global-table Regions before creating replicas
• Enabled point-in-time recovery (35-day window) and rehearsed a restore into a new table
• Set client timeouts, bounded retries, backoff with jitter and a circuit breaker

Related lessons

• PostgreSQL internals: MVCC cost, VACUUM, wraparound and WAL
• MySQL and InnoDB: locks, buffer pool, replication and Vitess
• Redis in depth: cluster slots, eviction, persistence and the lock debate
• Cassandra and ScyllaDB: the ring, repair, compaction and Discord's migration
• MongoDB internals: oplog, WiredTiger, shard keys, the balancer and Stripe's DocDB
• Kafka as a log store: ISR, acks, compaction, transactions and LinkedIn scale