Cruxly › System design › Databases by Name: Internals and Terms

MongoDB internals: oplog, WiredTiger, shard keys, the balancer and Stripe's DocDB

documents are easy; the oplog window, the cache and the shard key are what you operate

MongoDB's data model is the easy part. What you operate is the replication log and its window, the WiredTiger cache and checkpoints, the shard key and the balancer. This lesson gives the defaults and the arithmetic for each, then shows how Stripe built a database platform around MongoDB. Embedding versus referencing, write and read concern semantics and the list of shard-key properties are in NoSQL modelling in practice and the consistency lesson; they are not repeated.

The oplog and elections: what a replica set actually does

A replica set is a primary plus secondaries that replay the primary's operation log. The MongoDB manual's oplog page describes local.oplog.rs as a capped collection holding a record of every operation that changes data. Secondaries copy and apply those entries. Each oplog operation is idempotent: applying it once or several times leaves the same result. That is what makes replay after a crash or a restart safe, because a secondary does not need to know exactly where it stopped.

The oplog window. Because the oplog is capped, old entries are overwritten. The oplog window is the time between the newest and the oldest entry. A secondary that is disconnected for longer than the window can no longer catch up from the log and needs a full initial sync. This is the number that decides how long maintenance, a network partition or a bad deploy can last before you pay for copying the whole dataset.

Default size (WiredTiger, manual page read October 2026). 5 percent of free disk space, with a minimum of 990 MB and a maximum of 50 GB. The page also says the oplog can grow beyond its configured size to avoid deleting the majority commit point, so it is not a hard cap during lag.

Worked arithmetic, with an assumed write rate. At 10 MB/s of oplog-producing writes, a 50 GB oplog holds 50,000 / 10 = 5,000 seconds, about 83 minutes. A big disk gives no more than 50 GB by default, so a busy cluster can have a window under 90 minutes. At 5 MB/s, a 990 MB oplog holds 198 seconds. The write rates are illustrative; measure yours.

Elections. Per the replica-set elections page (October 2026): members heartbeat every 2 seconds; a secondary that hears nothing from the primary for the election timeout, electionTimeoutMillis, which defaults to 10 seconds, calls an election. The page says the median time to elect a new primary should not typically exceed 12 seconds with default settings. During that window there is no primary, so writes fail or wait. Higher-priority members call elections sooner, priority-0 members never become primary, and a primary that can see only a minority steps down, which prevents two primaries accepting writes.

Resizing and retention. The oplog can be resized on a running member with replSetResizeOplog, so a window that proves too short is a fixable mistake rather than a rebuild. The oplogMinRetentionHours setting keeps entries for at least that many hours, but only once the oplog has already reached its configured size; it stretches the window at the cost of disk when writes are heavy.

Write concern and rollback. Since MongoDB 5.0 the implicit default write concern is w: "majority", except in a replica set with an arbiter where the non-arbiter data-bearing members only just reach a voting majority (for example two data members plus an arbiter), where the default drops to w: 1. A w: 1 write is acknowledged by the primary alone. If the primary steps down before any secondary has it, the old primary rolls that write back when it rejoins, and the manual says the rolled-back documents are written to a rollback directory as BSON files (inserts and updates only; rolled-back deletes and drops are not saved there). A w: "majority" write on journaled voters should not roll back. Reads at the default local read concern can still return data that is later rolled back. Rollbacks are bounded by rollbackTimeLimitSecs, 24 hours by default. The election page also lists catchUpTimeoutMillis as the knob that trades a faster failover against preserving w: 1 writes the new primary has not yet caught up on.

:::key Verified October 2026 Oplog 5 percent of free disk (minimum 990 MB, maximum 50 GB, WiredTiger), election timeout 10 s, heartbeat 2 s, median election at most about 12 s by default. :::

What experts say in the room. The argument: oplog size is a recovery-time budget, not a disk-space setting, and w: 1 is a decision to accept rollbacks at failover. The numbers: 990 MB to 50 GB, a 10 second timeout, a 12 second median election, a 24-hour rollback limit. The failure: a replica falls off the window during a long maintenance task and must resync while the remaining members carry the load.

What a senior checks. The oplog window in hours (not just size) against the longest maintenance you perform, with an explicit oplog size where the default is smaller. Replication lag per secondary, the election count per week as a stability signal, and that the write concern chosen matches the data-loss you can accept at failover.

WiredTiger: cache, checkpoints, compression, tickets

WiredTiger is the default storage engine, and its memory model is not Postgres's. The manual's WiredTiger page (October 2026) gives the cache default as the larger of 50 percent of (RAM minus 1 GB) or 256 MB, and advises against raising it above the default. The rest of memory is used by the operating system's file cache, which holds the compressed pages, so a read can be served from the file cache without touching disk and then pay a decompression cost.

Checkpoints and the journal. WiredTiger writes a consistent snapshot to disk every 60 seconds, a checkpoint. Between checkpoints the journal records writes. After a crash, per the journaling page, WiredTiger finds the last checkpoint identifier in the data files, locates the matching journal record, and replays the journal from there. Journal records are synced to disk every 100 milliseconds, when a write includes or implies j: true (w: "majority" implies it when writeConcernMajorityJournalDefault is true), after each batch of oplog entries on a secondary, and when a new journal file of about 100 MB starts. The page warns that between syncs, a hard shutdown can lose updates that are only in the buffers. Since MongoDB 6.1 journaling is always on. If journal disk space runs out, the page says the server crashes.

Compression. Collections use Snappy block compression by default and indexes use prefix compression; time-series collections default to zstd. You trade CPU for disk, and capacity plans must use physical, not logical, sizes.

Concurrency. Writes lock at the document level, and conflicts are retried transparently. A hot single document, such as a global counter, still serialises all its writers. The classic fix is to spread such a counter across several documents and sum on read.

Tickets. The page says read and write tickets never exceed 128 each, and that from MongoDB 7.0 they adjust dynamically. A ticket is permission to run inside the storage engine; when all are in use, new operations queue. Ticket exhaustion is a classic "the database is slow" signal: CPU and disk look fine, but a few slow queries hold tickets and everything else waits.

:::warn Do not size the cache like Postgres shared_buffers Raising the WiredTiger cache to most of RAM starves the file cache that holds the compressed pages and leaves nothing for connections and aggregation. The manual advises not exceeding the default. :::

What experts say in the room. The argument: MongoDB relies on the file cache differently from Postgres, so copying a shared_buffers rule is wrong. The numbers: 50 percent of RAM minus 1 GB, a 60 second checkpoint, 128 tickets. The failure: cache pressure plus ticket exhaustion, where p99 climbs while hosts look idle.

What a senior checks. Dirty cache percentage and eviction activity, queued and available read and write tickets, and checkpoint duration. Whether w: majority with journaling is the write concern for data you cannot lose. Physical storage per collection after compression.

Sharding: shard key, balancer and the 384 MB rule

The shard key decides distribution, and modern versions let you change it. The manual's shard-key page defines a shard key as a single indexed field or a compound index that determines how documents spread across shards, either hashed or ranged. A monotonic key such as a timestamp or an ObjectId with a ranged strategy sends every new insert to the last range, so one shard takes all the writes. A hashed key spreads inserts but gives up efficient range queries on that field. A low-cardinality key cannot split below one value, which creates oversized chunks.

The old advice is out of date. "The shard key is forever" no longer holds. The shard-key and balancer pages say a key can be refined by adding fields, or changed completely by resharding, and since MongoDB 8.0 you can reshard to the same key to move data. Moving data between shards costs I/O and time, so treat resharding as a recovery tool, not a design strategy.

Routing cost. A query that includes the shard key is sent only to the shards that hold the data; one that omits it is broadcast to every shard and does not get faster as shards are added. From MongoDB 7.0 the analyzeShardKey command reports a candidate key's cardinality, frequency and monotonicity from sampled queries, so a key can be tested before it is chosen.

Balancer. Per the balancer administration page, the balancer runs on the config server primary and moves ranges between shards. A shard takes part in one migration at a time, so n shards can run at most n / 2 migrations at once. That gives a floor on rebalancing time: 2 shards, 1 migration; 5 shards, 2; 10 shards, 5 (arithmetic from the rule). Adding shards speeds a rebalance but never makes it fully parallel.

The 384 MB rule. The default range (chunk) size is 128 MB. A collection is considered balanced when the data difference between the most and least loaded shards is less than 3 times the range size, 384 MB by default. A few hundred megabytes of skew is therefore normal, and two shards that differ by 300 MB will not trigger migrations. This is a threshold on data per collection per shard, so on a very large collection, balanced still allows a visible difference in load.

Jumbo chunks. A range that is too large to move is called jumbo. By default a range cannot move if its document count exceeds 2 x (range size / average document size); with 128 MB ranges and 1 KB documents that is about 2 x 131,072 = 262,144 documents (arithmetic from the rule). The page says the balancer can be told to attempt oversized ranges with attemptToBalanceJumboChunks, and moveRange and moveChunk accept forceJumbo. The shard-key page adds the cause: a key where most documents share a few values makes those ranges indivisible, and a key with only seven distinct values can never use more than seven shards usefully. Low-cardinality and monotonic keys create jumbo chunks, and the fix is a better key (refine or reshard), not a flag.

:::key Verified October 2026 Range size default 128 MB, balanced threshold 3 x range size = 384 MB, at most n/2 concurrent migrations, resharding to the same key since 8.0. These come from the shard-key and balancer-administration pages; check them for your server version. :::

What experts say in the room. The argument: the shard key decides your fate, but you can now refine or reshard, at a cost. The numbers: hashed versus ranged, 128 MB ranges, 384 MB threshold, n/2 migrations. The failure: a monotonic key sends all inserts to one shard while the balancer says everything is fine.

What a senior checks. Chunk distribution per shard, the jumbo count, migration duration and the balancer window, and the share of queries that scatter to all shards. The shard key checked against the top five queries before sharding, since resharding is possible but heavy.

The balancer: n/2 migrations and the cost of moving data

Moving data is expensive, so the balancer is deliberately slow. A migration moves a range from one shard to another. Because a shard is in one migration at a time, the n / 2 rule bounds throughput. Rebalancing a collection after adding shards takes time proportional to the data moved divided by the migrations in flight; with 10 shards at most 5 run together.

What this means operationally

• Adding a shard is not instant relief. A hot shard keeps taking load while ranges move, and the move itself adds I/O to both ends.
• A balancer window (the balancer page lets you restrict when it runs) keeps migration I/O away from your peak, at the price of a longer backlog.
• Jumbo ranges do not move without being forced, so one shard can hold far more data than the others while the balancer reports no work it can do.
• A migration is a copy, then a cut-over. Per the balancer page, the source keeps serving the range, the destination builds missing indexes, copies the documents and then applies the writes that arrived during the copy; the source then updates the config database with the new location and deletes its copy once no open cursors remain on the range. The _secondaryThrottle setting makes each moved document wait for a write acknowledgement, trading speed for replication headroom, and the balancer starts the next migration without waiting for the previous delete phase.
• Skew under 384 MB is allowed. If your queries all concern the newest data on one shard, "balanced" still means "hot".

Design question, not a company claim. Should an application shard over many independent databases, or trust the cluster to rebalance? Trusting the balancer is cheaper to run and works while the key is sound. Application sharding gives full control of placement and migration timing but makes you own routing and rebalancing. The next section shows a company that built a platform to own placement.

What experts say in the room. The argument: the balancer rebalances data, not load. The numbers: 128 MB range, 384 MB threshold, n / 2 migrations. The failure: a monotonic key keeps sending all inserts to one shard, and the balancer chases it forever, moving ranges the writes already left.

What a senior checks. Whether the balancer has caught up (migrations pending), the duration of recent migrations, jumbo chunk count, and whether the chosen window overlaps with peak traffic.

Stripe's DocDB: MongoDB as a platform

At large scale MongoDB becomes a platform problem, and Stripe's DocDB is the documented example. Stripe's 2024 engineering post, "How Stripe's document databases supported 99.999% uptime with zero-downtime data migrations", describes DocDB as an in-house database-as-a-service built on MongoDB Community, with custom services added: a chunk metadata service for routing and a Data Movement Platform for migrating data. These are Stripe's own claims.

Published numbers (Stripe's post, 2024). Over five million queries per second; more than 10,000 distinct query shapes; petabytes of financial data in more than 5,000 collections across more than 2,000 shards; 99.999 percent uptime in 2023 while processing 1 trillion dollars in payments volume.

Arithmetic, not Stripe's. 5,000,000 / 2,000 = 2,500 queries per second per shard on average; 5,000 / 2,000 = 2.5 collections per shard on average. Five nines is 365 x 24 x 60 x 0.00001 = 5.26 minutes of unavailability per year.

The Data Movement Platform, as the post describes it.

1. Register the migration intent.
2. Bulk-import a snapshot of the data to the target.
3. Asynchronously replicate ongoing writes.
4. Run correctness checks on point-in-time snapshots.
5. Switch traffic, with versioned gating.
6. Deregister, then drop the source.

The post adds that the traffic switch relies on a custom MongoDB patch that rejects requests carrying a stale version token, and reports that a switch takes under two seconds. Stripe says it uses the platform to split shards by size and throughput, merge underused shards and upgrade major versions; in one 2023 consolidation it moved 1.5 petabytes and cut the shard count by about three quarters. That is a copy, tail, verify, cut-over pattern. Its important steps are the check before the switch and the versioned gating on the switch, because routing metadata that goes stale is how a move loses writes.

Design question, not a company claim. Why wrap MongoDB instead of trusting the stock balancer? The post's framing is routing, migration and tenancy under an uptime target. The cost is a team to build and run it, which only pays when the fleet is large.

What experts say in the room. The argument: MongoDB at scale is a platform problem, not a driver problem. The numbers: 5 million queries per second, 2,000-plus shards, 99.999 percent. The failure: an unsafe chunk move that loses writes or serves stale routing.

What a senior checks. That any migration tool has a correctness check before cut-over and versioned routing metadata, that move progress is observable, and whether a team smaller than the one in the post should build this or buy a managed service.

What a staff engineer checks before running MongoDB at scale

The operating checklist, with the reason for each.

Interview framing. Asked "why MongoDB", name the operations you accept: sizing an oplog, a shard key you may have to refine, and a cache that is not Postgres's. Then name the tooling you would add above a threshold, which Stripe's DocDB illustrates.

Journal, recovery and what a crash costs

Know the loss window of each durability setting. The journaling page says journal records sync every 100 ms by default, or earlier when a write includes or implies j: true. A write acknowledged without journaling can be lost in a hard shutdown if it has only reached the buffers. With w: majority and writeConcernMajorityJournalDefault true, the acknowledgement implies the journal sync on the members that count.

Recovery, worked arithmetic. Checkpoints run every 60 seconds and journal files are about 100 MB, so a crash recovery replays at most the journal since the last checkpoint, which is bounded by the write rate: at 10 MB/s of journal, up to 60 x 10 = 600 MB across several files. Recovery time is that replay, not the dataset size.

:::warn In-memory members and majority The journaling page warns that if any voting member uses the in-memory engine, set writeConcernMajorityJournalDefault to false, otherwise a majority write could roll back after a transient loss of a majority of nodes. :::

What experts say in the room. The argument: acknowledged is not durable unless the write concern says so. The numbers: 100 ms sync, 60 s checkpoint. The failure: a majority-acknowledged write rolling back with a non-durable voter.

What a senior checks. The write concern in every service's driver configuration, journal disk headroom, and recovery time in a crash drill.

How to say it in the interview

"A MongoDB replica set replicates through the oplog, a capped collection of idempotent operations, so oplog size is a recovery-time budget: a secondary down longer than the window needs a full resync. The default oplog is 5 percent of free disk, between 990 MB and 50 GB, and I size it on purpose. After a primary failure, the default 10 second election timeout gives a write outage of about 12 seconds at the median. In WiredTiger the cache defaults to half of RAM minus 1 GB and checkpoints run every 60 seconds. For sharding, I pick a key with cardinality and no monotonic prefix, and I know the balancer treats shards as balanced until they differ by 3 x 128 MB."

Checklist

• Measured the oplog window in hours and set the oplog size explicitly
• Knew the failover write outage: 10 s election timeout, 2 s heartbeat, about 12 s median
• Matched write concern and journal settings to the data-loss the product can accept
• Left the WiredTiger cache at its default, 50 percent of (RAM - 1 GB)
• Monitored dirty cache, evictions and ticket availability (128 each)
• Checked the shard key for cardinality and monotonic prefixes against the top five queries
• Tracked chunk distribution, jumbo count and migration duration per shard
• Planned rebalancing around n/2 concurrent migrations and a balancer window
• Required a correctness check and versioned routing before any data-movement cut-over

Related lessons

• PostgreSQL internals: MVCC cost, VACUUM, wraparound and WAL
• MySQL and InnoDB: locks, buffer pool, replication and Vitess
• Redis in depth: cluster slots, eviction, persistence and the lock debate
• Cassandra and ScyllaDB: the ring, repair, compaction and Discord's migration
• DynamoDB under the hood: partitions, adaptive capacity, quotas and the 2015 outage
• Kafka as a log store: ISR, acks, compaction, transactions and LinkedIn scale