Cruxly › System design › Reliability and Observability

Designing for Observability

you will be debugging this at 3am with only what you built in

The senior-interview question: when this design fails in production, how will anyone find out, and how will they know where? Most candidates bolt it on at the end ("and we'd add monitoring"). Treating it as a design input is a staff-level signal.

Start from the SLO, not from the dashboard

Design top-down: set the target first, then decide what to measure. Bottom-up ("instrument everything, then pick alerts") produces dashboards nobody alerts on.

Term	Example	Note
SLI, what you measure	Share of checkout requests that succeed in under 500 ms	Measure at the edge, where the user is. Your service's internal latency skips the network and every hop in front of you
SLO, the target	99.9% over a rolling 28 days	A negotiated number, not an aspiration
Error budget, what that allows	0.1% of 28 days = ~40 minutes	Spend it on risky deploys, planned maintenance, a flaky dependency. At zero, feature work stops and reliability work starts: a policy, not an argument someone has to win

The target shapes the architecture. 99.99% leaves about 4 minutes and likely forces multi-region failover and synchronous replication, ruling out any single-region dependency. 99.9% doesn't. Ask "what's the availability target?" early; almost nobody does.

The three signals, and what each is actually for

Signal	Answers	Cost shape	Cardinality	The rule
Metrics	What is happening, in aggregate	Cheap, fixed per series	Must stay bounded	No unbounded values in labels, ever. User ID, request ID or full URL multiplies the series count until the metrics system falls over, the most common way teams take down their own monitoring
Traces	Where the time went across services	Expensive per request, so sample	High, but sampled	Head-based sampling (decide at the entry point) misses the rare slow requests you most want. Prefer tail-based: buffer, then keep traces that were slow or errored
Logs	Why this request did that	Expensive at volume	Unbounded, so structure them	Key-value, not prose: `event=payment_failed order_id=123 reason=insufficient_funds` is greppable and aggregatable; a sentence is neither at 3am across ten million lines

Alerting runs on metrics because they are cheap to store and query. Only a trace answers "checkout is slow, which of the eleven services is responsible?"

The correlation ID is what makes all three usable together

Mint a trace ID at the edge and propagate it through every call. Attach it to every log line and every span. Include queue messages and async jobs: a trace that stops at the queue boundary loses the asynchronous half of the system, where the hard bugs live.

Step	Signal	Says
1	Metric	Error rate is up
2	Trace for one failing request	`payments` took 840 of the 900 ms
3	Logs filtered by that `trace_id`	Why that hop did it

Without the shared ID, each step is a manual guess, correlating timestamps across systems at 3am.

Logs at scale: structured, sampled, shipped

Logs are the signal that grows without anyone deciding it should. Metrics are bounded by the series you declare; a log line is emitted every time a developer types a log call, and the bill follows traffic. Design the log path with the same care as the data path.

Structure first. Emit one JSON object per event, with a fixed core and free-form detail:

Field	Why it is there
`ts`, `level`, `service`, `version`	Sort, filter, and tie a spike to a deploy
`trace_id`, `span_id`	The join to the trace. The OpenTelemetry log data model carries both as first-class fields next to the timestamp, severity and body, so a log line and a span can reference each other
`event` (a stable name, not a sentence)	Aggregate by `payment_failed` without regexes
`order_id`, `reason`, `duration_ms`	The detail. High cardinality is fine here, because a log is an event, not a time series

Levels are an operational contract, not a mood. OpenTelemetry normalises severity into ranges: TRACE 1-4, DEBUG 5-8, INFO 9-12, WARN 13-16, ERROR 17-20, FATAL 21-24. Agree what each means for you: ERROR means someone may need to act; a user typing a wrong password is INFO or WARN, never ERROR. Production runs at INFO; DEBUG is switchable per service at runtime, so an incident can turn it on without a deploy. If ERROR is noisy, people stop reading ERROR.

The arithmetic that decides everything. Worked arithmetic: a service at 2,000 log events per second, 400 bytes each, is 800 KB/s, which is about 69 GB a day and about 2.1 TB for 30 days of retention. Scale that by 50 services and the log system is a bigger data platform than the product. The levers are the three factors: fewer events (levels, sampling), smaller events (drop fields nobody queries, never log whole request bodies), shorter retention (tiers below).

:::key Verified November 2026 The AWS CloudWatch pricing page's Logs worked examples (US East, N. Virginia) use $0.50 per GB ingested and $0.03 per GB-month archived; the Infrequent Access class is shown at $0.25 per GB ingested. At those rates the 69 GB/day service costs about $1,040 a month to ingest (69.1 GB x 30 x $0.50) and at most about $62 a month to hold 30 days (2,074 GB x $0.03; treat it as an upper bound, since archived logs are stored compressed). Ingestion dominates, which is why the cheapest log is the one you never ship. Prices are region-specific and change; check the pricing page before quoting them. :::

Sample the routine, never the exceptional. Keep every ERROR, every WARN, and every audit event (who did what to which record): an audit trail with holes is worthless, and it often carries a legal duty. Sample the high-volume success lines, say 1 in 20, and rate-limit any single line that can loop (a retry that logs "connection refused" 5,000 times a second should emit one line a second with a suppressed_count). Sample by trace_id, not per line, so a kept request keeps all its lines. On the example above, if 95% of events are routine successes, 1-in-20 sampling leaves about 9.75% of the volume: roughly 6.7 GB a day and about $100 a month, a tenfold saving that costs you nothing during an incident, since the failing requests are all kept. The SRE book makes the same point about metrics: very fine-grained collection is expensive to collect, store and analyze, so sample at the source and aggregate downstream.

Ship it so logging can never take the service down.

The service writes to stdout and nothing else; an agent on the node tails, adds host and pod metadata, scrubs, and forwards. Between agent and indexer sits a durable buffer such as Kafka (see Queues and Async Work and Kafka as a log store: ISR, acks, compaction, transactions and LinkedIn scale), so an incident that multiplies log volume by ten backs up into the buffer, not into the indexer. Backpressure policy is a design decision: a request thread must never wait on a log write. Use a bounded in-memory queue; when it fills, drop by priority (DEBUG first, ERROR and audit last), and count what you dropped as a metric, so a gap in the logs is visible instead of silent. A synchronous call to the logging backend on the request path turns a logging outage into a product outage.

Retention tiers follow how logs are actually used. Most queries are about the last hours. Keep hot (indexed, fast, expensive) for days; warm (cheaper, slower search) for weeks; cold (object storage, restored or scanned on demand) for the months a compliance rule asks for. Indexing is the cost lever: Grafana Loki indexes only a small set of labels per stream and stores compressed log chunks in object storage, which its documentation credits for a small index and much lower cost than indexing every line. The trade is that a query for an arbitrary string inside the lines has to scan chunks, so keep labels few and low-cardinality (service, environment, level) and put order_id in the line, not the label.

PII and secrets: scrub at the source. Logs are copied to more places than the database and read by more people. Never log passwords, tokens, card numbers or full request bodies; redact in the logging library (an allow-list of fields beats a deny-list of patterns) and again in the agent as a backstop. Tokenise identifiers you still need to correlate, such as a hash of the user ID. Retention is a legal question as well as a cost one: a deletion request or a data-residency rule applies to logs too, so a year of unscrubbed logs is a liability, and keeping everything forever is not the safe default.

Which signal, when. Start from the symptom: is it happening, and how much? is a metric. Where in the call graph? is a trace. Why did this one request do that? is a log, found by trace_id. Do not search logs to compute a rate you could have counted; and do not put a user ID on a metric to make it behave like a log.

What to alert on, and the anti-patterns

Alert on symptoms, not causes. "Checkout success rate below 99% for 5 minutes" is a page: a user is affected. "CPU above 80%" is not. If it isn't hurting users, waking someone trains them to ignore pages. Page on SLO burn rate (how fast you consume the error budget): it fires on real user impact and stays quiet otherwise.

Multi-window burn rates: a fast burn (budget gone in hours) pages immediately. A slow burn (budget gone in days) opens a ticket. One threshold can't express both, which is why alerting ends up too noisy or too slow.

Anti-pattern	Why it hurts	Fix
Alerting on every cause	Fifty pages for one incident; responders mute them	Page on symptoms and burn rate
No alert on absence	A service that stops reporting produces no metric, so a threshold never fires. The worst failure is the quietest	Staleness detection against an expected inventory
Dashboards instead of alerts	A dashboard nobody watches at 3am detected nothing	Alert; keep dashboards for diagnosis
Monitoring shares fate with what it monitors	A cluster-wide failure takes out the thing that would have told you	Run the watchdog on independent infrastructure

End a design by naming: the SLI and SLO, the one or two golden-signal metrics you'd page on, that traces propagate end to end including through queues, and one failure you'd build a staleness alert for. Fifteen seconds, and a different signal from "we'd add monitoring".

How to say it in the interview

"I want to name the SLO first, because it decides what I need to measure. Then the three signals — metrics for what is happening, traces for where the time went, logs for why a specific request did what it did — and the rule that every one of them has to be attributable to a request I can actually find."

Checklist

Defined SLI and SLO before deciding what to instrument
Converted the SLO into an error budget as a spendable quantity
Noted that the availability target changes the architecture, and asked for it during scoping
Distinguished the three signals by what question each answers
Named unbounded metric label cardinality as the way teams break their own monitoring
Preferred tail-based trace sampling and said why head-based misses the important requests
Specified structured key-value logs over prose strings
Propagated a trace ID end-to-end, explicitly including through queues and async jobs
Costed the log pipeline (events/s x bytes x retention) and sampled routine lines while keeping errors and audit
Alerted on user-facing symptoms and SLO burn rate rather than on causes like CPU
Used multi-window burn rates so fast burns page and slow burns file a ticket
Raised absence-of-signal as its own alert condition
Required the monitoring path not to share fate with the system it monitors

Practice this interactively