Cruxly › System design › Databases by Name: Internals and Terms

Elasticsearch and OpenSearch: shards, refresh, translog and shard sizing

a search index that is never quite real time, and a shard count you cannot easily change

Elasticsearch stores each shard as a Lucene index, shows new documents only after a refresh, and fixes the primary shard count when an index is created. This lesson teaches the three clocks (refresh, fsync, flush), the shard-sizing arithmetic from Elastic's own guidance, why deep pages fail, and what Discord's 2017 search post says about sharding in the application. It reads Elastic's documentation; OpenSearch forked from the same code; check any setting there in its own docs.

A shard is a Lucene index

An Elasticsearch index is a set of shards, and each shard is a complete Lucene index with a hard document cap and a count you cannot change after creation. This is the inverted-index structure named in Data models: what each family is built for, and where it fails and the derived-search pattern in Choosing the database: seven worked case studies; this lesson is about what sits under it.

Per Elastic's shard-sizing guide (read October 2026), each shard can hold at most 2,147,483,519 documents, which is 2^31 minus 129, and the count includes deleted documents that have not yet been merged away. Arithmetic: 2^31 - 129 = 2,147,483,519. A single giant shard hits that wall as a hard failure, which is why the guide advises far lower practical ceilings (section 4).

Primary shard count is a creation-time setting. The index-modules reference says index.number_of_shards defaults to 1 and can only be set when the index is created; index.number_of_replicas defaults to 1. A wrong count means copying the index. Elastic's shard-sizing guide lists three routes: the split API to increase the primary count (to a multiple of the old one), the shrink API to reduce it on an index you no longer write to, and a reindex into a new index with the right settings. Each copies the data, so the safe design assumes you will not get to change the count cheaply.

Cluster state is a shared cost. The guide says every node holds a copy of the cluster state, including field mappings, and mapping metadata uses heap. Many indices with large mappings burden the master and every node's memory. Mapping size is reported by the cluster stats API, and index.mapping.total_fields.limit caps the number of fields per index; check its value for your version before designing wide dynamic mappings.

:::key Verified October 2026 Values are from the Elastic shard-sizing and index-modules pages on that date. Defaults can differ by version and by Serverless, where the platform manages shards for you. :::

What experts say in the room. The primary shard count is a one-way door. The number is 2^31 - 129 and the failure is a single giant shard or an index created with the default of one. What a senior checks: documents per shard including deleted ones, mapping size in cluster stats, and the written plan for how an index would be rebuilt if its shard count were wrong.

Refresh, flush and translog: three clocks

There are three separate clocks in Elasticsearch and mixing them up causes most of the durability mistakes. Refresh makes data visible to search, fsync makes it durable, and flush commits Lucene segments and trims the log.

Refresh (visible). Per the Elastic index-modules page, index.refresh_interval defaults to 1 second in the Elastic Stack (5 seconds on Serverless), and -1 disables refresh. When refresh_interval is not set explicitly, a shard that has had no search for index.search.idle.after (30 seconds by default) is skipped by the background refresh until a search arrives. A refresh makes recent writes searchable by creating a new segment, so a short interval means more small segments to merge and more CPU. Discord's 2017 post (section 6) reports that frequent refreshes drove up CPU and disk use in its early deployment.

Translog (durable). Between Lucene commits, writes are protected by the translog. The Elastic translog page says index.translog.durability defaults to request: each index, delete, update or bulk request is fsynced and committed before success is reported. The alternative is async, where the translog is fsynced on index.translog.sync_interval, default 5 seconds, with a minimum of 100 ms. With async, acknowledged writes since the last commit can be lost in a failure. So request mode ties ingest throughput to disk fsync latency; async buys speed by accepting a loss window of about the sync interval (arithmetic: 5 seconds of writes at 30,000 messages per second, Discord's published peak, would be about 150,000 documents).

Flush (committed). A flush performs a Lucene commit and starts a new translog generation. It is neither visibility nor the per-request fsync. index.translog.flush_threshold_size caps the size of operations not yet committed to Lucene; when it is reached, a flush creates a new commit.

:::warn Two defaults, depending on the version index.translog.flush_threshold_size defaulted to 512 MB in the Elasticsearch 7.17 documentation and in every 8.x page up to 8.14. The 8.15 documentation and every later page, including the current one, say 10 GB. A cluster on 8.14 or older therefore flushes (commits to Lucene and rolls the translog) far more often than a current one, and a crash on a current version can leave up to 10 GB of operations to replay. Check the version before quoting either number. :::

What experts say in the room. Three clocks: visible, durable, committed. The numbers are 1 s and 5 s. The failure is an ingest storm with refresh_interval at 1 second and a pile of tiny segments. What a senior checks: indexing latency, segment count, merge time and disk fsync latency, and that no one changed durability to async without recording the loss window.

Ingest tuning: refresh and bulk load

Refresh is the one setting a team should choose per index on purpose. Searchable-in-one-second is a product promise with a cost, and a bulk load does not need it.

The standard move is to set index.refresh_interval to -1 for a bulk load into an index nobody queries yet, then set it back and let one refresh make the data visible. The index-modules page documents -1 as disabling refresh; the habit of restoring the interval afterward is what a reviewer asks about. Worked arithmetic: at most 60 refreshes per minute at the default interval versus at most 2 at 30 seconds, and each refresh that finds new data produces a segment that later has to be merged. For a heavy ingest, 30 times fewer segments is the difference between merges keeping up and not.

The visibility trade. Choosing 30 seconds means a user can wait up to 30 seconds to find a document they just saved. Search lags the primary store; here the lag is by the refresh interval, plus whatever delay your indexing pipeline adds. Say the number to the product owner.

Durability and speed. request durability fsyncs per request, so bulk requests (many documents per fsync) amortise the cost, while single-document writes pay one fsync each. Moving to async is a different bargain: faster, with the 5-second exposure above. Moving it silently is the failure.

What experts say in the room. Refresh is visibility, not durability, so it is the cheap knob and translog durability is the dangerous one. The number is 1 s versus the 30 s a bulk job tolerates. The failure is forgetting to restore -1, after which nothing new is ever searchable. What a senior checks: the refresh interval on every index and a runbook step that resets it after a load, indexing rate versus merge backlog, and any index with async durability.

Shard sizing arithmetic

Shard count is capacity planning with a floor and a ceiling. Elastic's shard-sizing guide (read October 2026) says to aim for shards of 10 GB to 50 GB, and under 200 million documents per shard. Too many small shards waste resources; too few large shards recover slowly after a failure.

Worked example (worked arithmetic, using 1 TB = 1,000 GB). 1 TB of data is 20 shards at 50 GB or 100 shards at 10 GB, so the guidance gives a range of 20 to 100. For a log workload at 1 TB per day with 30 days kept, 20 primary shards per day times 30 days gives 600 primaries, and with the default one replica that is 1,200 shard copies. If you had aimed at 10 GB per shard it would be 3,000 primaries. That is the thousands-of-tiny-shards failure the guide warns about.

Oversharding. The guide explains that each shard runs its part of a search on a single thread, so a query across many shards can exhaust the search thread pool. Fewer, larger shards beat many small ones until recovery time hurts.

Per-node ceiling. The guide also warns against more than 1,000 non-frozen shards per node, cluster.max_shards_per_node caps open shards, and raising that cap is described as a temporary fix. Another quiet cost: a shard's deleted documents count toward the 2,147,483,519 limit until merged away.

Master heap. The current guide says to keep fewer than 3,000 indices per GB of heap on master-eligible nodes and to add about 0.5 GB of extra heap for overhead. Worked arithmetic: 10,000 indices needs 10,000 / 3,000 = 3.33 GB, about 3.8 GB with the overhead. An older rule of thumb, 20 shards per GB of heap, was deprecated (per an Elastic blog for version 8.3), so do not quote it in a design review.

ILM rollover sizes time-series indices by shard size instead of by day. The guide names max_primary_shard_size (for example 50gb), max_age, max_docs and max_size, and suggests min_primary_shard_size of 10gb so a rollover does not produce tiny shards.

How to defend a number in the room. Start from the data, not the cluster: expected size, growth per day, retention, and the slowest recovery you can tolerate. Divide size by a target shard size inside the 10 to 50 GB range, then check the result against documents per shard (under 200 million) because small documents can hit the document ceiling before the size ceiling. Worked arithmetic: a 50 GB shard holding 200 million documents implies 250 bytes per document on average; if your documents average 2 KB, a 50 GB shard holds about 25 million, so the size limit binds first, but for 100-byte documents the 200 million limit binds first, at about 20 GB. Then multiply by replicas, divide by nodes, and ask whether one node loss leaves enough room to re-create its shards elsewhere. Elastic's guide gives the range and the limits; the headroom judgement is yours.

What experts say in the room. Shard sizing is a floor and a ceiling, and the numbers are 10 to 50 GB, 200 million documents and 3,000 indices per GB. The failure is thousands of tiny shards from one-index-per-day-per-tenant. What a senior checks: shards per node, the distribution of shard sizes against 10 to 50 GB, documents per shard, and recovery time after a node loss.

Deep paging and search_after

A "next page" over a large result set is a stateful cursor, and from plus size is the wrong tool once it gets deep. Per the Elastic pagination and index-modules docs, index.max_result_window defaults to 10,000: from plus size above that fails.

Why it fails, mechanically. The pagination page says each shard loads its requested hits plus all the earlier hits into memory. Worked arithmetic: page 1,000 at size 10 means from of 9,990, so each shard sorts 10,000 hits; with 20 shards that is 200,000 candidate hits merged to return 10. Page 1,001 breaches the 10,000 window, and page 5,000 (from 49,990) fails outright. Raising the window just moves the memory blowup.

What replaces it. The docs recommend search_after with a point in time (PIT): the PIT preserves the index state so the pages are consistent while documents change, and search_after takes the sort values of the last hit as the cursor. Add a unique tiebreaker to the sort or documents with equal sort values can be skipped or repeated. Scroll is no longer recommended for deep pagination; it remains fine for bulk work such as reindexing.

The product consequence. search_after goes forward from a cursor; it cannot jump to page 5,000. Few real users need that; they need better filters. Say so in the interview: it turns a technical limit into a product decision.

What experts say in the room. The argument is whether to raise max_result_window; the answer is no, change the access pattern. The number is 10,000, and the failure is a node running out of memory or a circuit breaker tripping on a crawler. What a senior checks: the slow log, circuit breaker trips, and any client that loops over pages.

Discord's message search: sharding in the application

Discord's 2017 post is the published example of choosing to shard Elasticsearch yourself instead of trusting one big cluster. Everything below is from Discord's own post, "How Discord indexes billions of messages", and describes 2017, not today. Nothing here describes their current design.

The numbers they report. About 26 billion documents, roughly 16,000 indices, peak indexing about 30,000 messages per second, on 14 nodes across 2 clusters, each node an n1-standard-8 with a 1 TB SSD. Arithmetic, not Discord's: 26 billion / 16,000 is about 1.6 million documents per index, 26 billion / 14 is about 1.9 billion documents per node, and 30,000 per second is up to 2.6 billion messages per day at peak rate.

The design. Messages go to a pool of smaller clusters; Cassandra holds the shard mapping as the source of truth and Redis caches it. The stated reason was to avoid large clusters: spreading data over smaller ones limits the blast radius of any one outage. Each index has one shard and one replica, and a "shard" in Discord's vocabulary is a cluster and index pair. Workers batch messages and bulk insert them, accepting a short delay before a message is searchable. That delay is a product decision made explicit.

The incident-shaped lesson. Discord reports that Elasticsearch's default 1-second refresh drove CPU and disk use up, and that simply disabling refresh did not work for them. Instead the application records which shards have received writes in an expiring Redis hashmap and refreshes a shard only when a search touches it, with an automatic refresh every 60 minutes as a backstop. That is the section 2 trade-off in a company's own words: the default near-real-time promise cost more than the product needed.

The debate (both sides). Application sharding gives control over where data sits, how clusters scale and how a failure is contained, and the cost is that your code owns routing, rebalancing and the mapping store. Trusting the cluster means fewer moving parts and every limit from section 1 and 4 applies. The post is Discord's own account, one side of a design choice; it does not prove that application sharding is better for other workloads.

What experts say in the room. Shard in the application or trust the cluster. The numbers are 26 billion documents, 16,000 indices and 30,000 messages per second. The failure is refresh cost. What a senior checks: indexing lag, bulk rejection rate, and where the mapping store sits in the failure domain.

Search is a derived index: source of truth and the reindex runbook

An Elasticsearch index is a derived, rebuildable view, so the record lives elsewhere and the rebuild path is part of the design. This is the shape in Choosing the database: seven worked case studies: one record store plus one derived search index.

Everything in this lesson points the same way. The shard count is fixed at creation (section 1), so a wrong count means a reindex. A refresh interval means search lags the write. Async durability can drop a window of acknowledged writes. Deep paging forces a product change. A cluster you lose may need to be rebuilt from the source.

The runbook to have written. Where is the source of truth? How long does a full reindex take (arithmetic: 26 billion documents at 30,000 per second is about 10 days of Discord's peak indexing rate, 26e9 / 30,000 / 86,400 = 10.0, which is worked arithmetic and a lower bound on an old design, not Discord's claim)? What serves search while it runs, and how is the new index swapped in? Scroll is still the documented tool for bulk reads, per section 5.

Search and the database disagree, and that is normal. If a document was saved but is not found, check the pipeline and the refresh interval before suspecting data loss. If it is found but stale, the delete or update has not been applied to the index yet.

Check on your version. Mapping explosion limits, doc values and fielddata behaviour, and OpenSearch differences vary by version. Where an interviewer asks about those, say you would check the docs for the running version.

What experts say in the room. Search is never the system of record. The number is the reindex duration, and the failure is discovering it during an outage. What a senior checks: a timed, tested reindex, the source-of-truth store identified, and monitoring that compares record counts between store and index.

How to say it in the interview

"A shard is a Lucene index, and the primary shard count is fixed at index creation, so shard sizing is a one-way door. I aim for shards of 10 to 50 GB and under 200 million documents, per Elastic's sizing guidance. New documents become searchable on refresh, which defaults to 1 second, and durable through the translog, which by default is fsynced before a write is acknowledged. I never page past 10,000 hits with from and size; I use search_after with a point in time. And search is a derived index, so the source of truth lives elsewhere and I have a tested reindex runbook."

Checklist

• Docs per shard (deleted ones included) and shard size distribution are tracked against 10 to 50 GB and 200 million documents
• Primary shard count is chosen at creation from a written capacity calculation, with rollover by size for time series
• refresh_interval is set deliberately per index, with -1 for bulk loads and a runbook step that restores it
• Translog durability is request unless a recorded exception exists, and fsync latency is monitored
• Segment count and merge time are on the indexing dashboard
• Master heap is sized from the index count (under 3,000 indices per GB plus about 0.5 GB) and mapping size is watched in cluster stats
• Deep pagination uses search_after with a point in time and a unique tiebreaker; max_result_window is not raised casually
• Shards per node are capped, and a thread-pool and circuit-breaker alert exists for search
• The source of truth is outside search, and a full reindex has been timed and rehearsed with a swap procedure
• Settings for OpenSearch or a newer Elasticsearch version are re-read from that version's docs before reuse

Related lessons

• PostgreSQL internals: MVCC cost, VACUUM, wraparound and WAL
• MySQL and InnoDB: locks, buffer pool, replication and Vitess
• Redis in depth: cluster slots, eviction, persistence and the lock debate
• Cassandra and ScyllaDB: the ring, repair, compaction and Discord's migration
• DynamoDB under the hood: partitions, adaptive capacity, quotas and the 2015 outage
• MongoDB internals: oplog, WiredTiger, shard keys, the balancer and Stripe's DocDB