Cruxly › System design › Reliability and Observability
Metrics Pipeline
write volume so high the storage shape is the design
Open the interactive lessonPrompt: collect metrics from a large fleet of servers. Support dashboards over the last hour at fine granularity, and over the last year at coarse granularity. Alerting on thresholds.
40 minutes first. Here the write rate is so far beyond a normal database that the shape of storage stops being an implementation detail.
Applying the four moves: Scope — ingest from a fleet, fine-grained last hour, coarse last year, threshold alerting; arbitrary ad-hoc analytics out. Estimate — the write rate is the whole story: hosts × metrics per host × samples per second lands far beyond what a general-purpose database can absorb, which is what makes storage shape a design decision rather than a detail. Sketch — an ingestion buffer, a time-series store organised by (metric, time) with compression, and a separate downsampling path for the year-long view. Deep dive — cardinality: one careless high-cardinality label multiplies series count and is the thing that actually takes these systems down, not raw volume.
Scope and numbers
In scope: ingest, query at two granularities, threshold alerting. Out of scope: distributed tracing, log aggregation, anomaly detection. Traces and logs are genuinely different problems — high cardinality and full-text, respectively — rather than "more of the same".
Estimate. 100,000 hosts, 1,000 metric series each, sampled every 10 seconds.
- Ingest: 10⁵ × 10³ / 10 = 10 million points/s.
- Raw at 16 bytes/point: 160 MB/s, 14 TB/day, 5 PB/year.
:::key Lead with this number 5 PB/year of data almost nobody queries is not a storage plan. That number alone forces the design. Total steady state after the ladder is a few tens of terabytes — two orders of magnitude, from one decision. :::
Separating series from points
The single most important modelling decision, and the one candidates miss.
A metric point is nominally (metric_name, {tags}, timestamp, value). Stored literally, the name and tags dwarf the value — a hundred bytes of text for eight bytes of number, repeated ten million times a second.
Split it:
- Series index.
series_id -> (metric_name, tags). Written once when a series first appears; thereafter a cache lookup. Millions of rows, not trillions. - Points.
(series_id, timestamp, value)— 16 bytes, and with delta-of-delta timestamp encoding and XOR float compression, closer to 1–2 bytes per point in practice. Time series compress extraordinarily well because consecutive values are similar and timestamps are regular.
That compression is not a footnote; it is a 10x on your entire storage bill and it is the reason purpose-built time-series stores exist. Mention the mechanism, not just the fact.
Cardinality is the failure mode. The number of distinct series is what kills these systems, and it explodes when someone puts a unbounded value in a tag — a user ID, a request ID, a URL with parameters. One bad deploy turns 1,000 series per host into 10 million. Defences: reject tags at ingest above a per-metric cardinality budget, alert on cardinality growth rate rather than only on absolute level, and document forbidden tag values. Raising cardinality unprompted is the strongest single signal in this problem, because it is the thing that actually takes these systems down.
The ingest path
Agent on each host. Pre-aggregates locally over a 10-second window — counters summed, gauges last-valued, histograms bucketed — and ships a batch. Batching is not an optimisation here; a request per point is 10 million requests/s and is simply not possible.
Push, not pull. At 100,000 hosts with autoscaling, a central scraper must maintain an accurate and constantly changing target list, and it becomes a bottleneck and a coordination problem. Push with the agent owning retry is simpler at this scale. Say the trade: pull gives you free liveness detection (a target that does not answer is down), so with push you need an explicit heartbeat. Knowing what you gave up is the point.
Ingest tier. Stateless receivers that validate, look up or create series_id, and append to a partitioned durable log. Partition by hash(series_id) so all points for one series land in order on one partition — which makes the consumer's job trivial and gives you per-series ordering for free.
The log is the buffer. With a durable log between ingest and storage you can lose the storage layer for an hour and lose nothing; consumers catch up. This is the difference between a metrics outage being an inconvenience and being a hole in your incident data — and the irony that your monitoring must survive the outages it monitors is worth one sentence.
Late and out-of-order points. A host was partitioned and ships an hour-old batch. Accept within a bounded lateness window (say 15 minutes) and drop beyond it, because indefinite acceptance means aggregates are never final and every rollup must be rewritable. State the window and the consequence.
Storage and rollups
Hot tier. Recent data, in memory plus local SSD, organised as per-series chunks covering a fixed time range (two hours, say). Writes append to the open chunk; a completed chunk is compressed and made immutable. Immutability is what makes everything downstream simple — compaction, replication and caching all get easier.
Rollup on completion. When a chunk closes, compute the 1-minute aggregates from it and write them to the warm tier; when a day of 1-minute data closes, compute hourly. Rollups are derived, so they are recomputable — if a rollup job has a bug you rerun it from the tier below rather than reconciling.
Store five aggregates per bucket, not one: count, sum, min, max, and a sketch for quantiles. An average alone is a trap — you cannot recover p99 from it, and p99 is what people actually alert on. And you cannot average averages across buckets correctly without the count; keeping count and sum lets you re-aggregate any coarser window exactly.
Quantiles do not average at all, which is why you need a mergeable sketch (t-digest or similar) rather than a stored percentile. Saying "percentiles are not averageable, so I store a sketch" is a sharp, specific signal.
Cold tier. Hourly rollups to object storage, partitioned by time, queried by scan. Slow and nearly free, which matches how often anyone looks at last year.
Query routing. The query planner picks a tier from the requested range and resolution: a 1-hour dashboard hits the hot tier, a 90-day chart hits warm, a 2-year chart hits cold. A query spanning tiers reads from both and stitches. Say the rule explicitly — "resolution requested picks the tier, and I refuse a request for 1-second resolution over a year rather than serving it slowly" — because refusing impossible queries is a real design decision.
Alerting, and the deep dives
Alerting is a streaming problem, not a query problem. Polling the store every 10 seconds for every rule does not scale past a few thousand rules. Instead, evaluate against the ingest stream: rules are indexed by the series they match, and an incoming point evaluates only the rules watching it.
State per rule. An alert needs hysteresis — fire when the condition holds for N consecutive windows, resolve when it clears for M. Without it, a metric oscillating at the threshold produces a flapping storm at 3am. That state must be durable, or a deploy re-fires every active alert.
Missing data is the hardest case. A host stops reporting. Is it down (alert loudly) or decommissioned (silence)? A threshold rule never fires on absent data, which is exactly backwards — the most serious failure produces the quietest signal. You need explicit staleness rules and an inventory to compare against.
Other dives:
Hot series. One series with a very high sample rate. Bounded by per-series rate limiting at ingest, and by the agent's local aggregation making it hard to exceed.
Rebalancing. Adding storage nodes moves series between partitions. Consistent hashing on series_id keeps the movement to 1/N, and because chunks are immutable, moving one is a copy rather than a migration.
Backfill. Importing historical data at high rate competes with live ingest for the same resources. Give it a separate, lower-priority path — live data losing to a backfill is the wrong priority and it happens by default.
The recursive problem. The metrics system needs monitoring. It cannot be its own monitor, so you need a small independent watchdog with its own storage. One sentence, and it lands, because it is the kind of thing only people who have run one of these think about.
How to say it in the interview
"The write volume here is tens of millions of points a second, which rules out anything that stores a point per row. So my first decision is the storage shape - append-only, pre-aggregated, and tiered by age - and everything else fits around it."
Checklist
- Computed raw volume and used it to reject storing raw data long-term
- Defined a retention and downsampling ladder with explicit resolutions
- Separated the series index from the points
- Raised cardinality explosion as the real failure mode, with a defence
- Pre-aggregated at the agent and batched, rather than a request per point
- Chose push vs pull and named what the other gives you
- Put a durable log between ingest and storage
- Set a bounded lateness window for out-of-order points
- Stored count, sum, min, max and a quantile sketch per bucket
- Routed queries to a tier by requested range and resolution
- Evaluated alerts against the stream, with durable hysteresis state
- Handled missing data as an alert condition in its own right