Cruxly › Cloud and DevOps › Cloud & DevOps: Observability
Metrics, Logs, and Traces: Picking the Pillar That Actually Answers Your Question
different questions, different costs, and the cardinality trap
Open the interactive lessonPrompt: a service's p99 latency has spiked. Design the investigation workflow across metrics, logs, and traces such that the on-call engineer converges on a root cause in minutes, not by grep-ing every log line across every service the request might have touched.
30 minutes first. The naive approach reaches for logs first because they feel the most detailed — and drowns in volume across a distributed system before narrowing down anything.
Each pillar answers a question the others structurally cannot
A metric has already discarded the individual events that produced it — an aggregated p99 latency number can tell you the tail got worse, never which specific requests were in that tail or why. This isn't a tooling limitation, it's what makes metrics cheap: throwing away per-event detail is exactly the trade that makes a metric fast to query and affordable to alert on continuously, at the cost of losing the ability to explain any single instance after the fact.
A trace's unique value is showing time spent across service boundaries, in one request's actual path — no single service's own logs or metrics can show that request A spent 400ms waiting on service B specifically, because service A's own telemetry only knows "I made a call and it took 400ms," not what happened inside B during that window. Only a trace, propagated with a shared identifier across every hop, reconstructs the full picture.
A log is the only pillar with full per-event context — but that context is only useful once you already know approximately where and when to look. Searching logs across every service, for a time window wide enough to be sure you caught the right request, is the expensive, slow-to-narrow-down version of an investigation that a trace could have pointed at one specific service and time range first.
Cardinality: the cost that compounds silently until it doesn't
Every additional label on a metric multiplies its stored time-series count, not adds to it. A metric with an endpoint label (100 distinct values), a region label (20 values), and a status_code label (60 values) already produces up to 100 × 20 × 60 = 120,000 distinct series from those three, individually reasonable-looking labels — and a fourth label with even modest cardinality multiplies that number again, not by 120,000 more, by that many times the new label's own cardinality.
And that's per target. Each pod exports its own copy, so 30 replicas multiply it by 30 through the pod label. Worse, pod names change on every deploy, so each rollout creates a whole new set of series while the old ones linger until they expire — series churn, which grows storage even when the live count looks stable.
Labels drawn from an unbounded or near-unbounded set — a raw user ID, a full request path with query parameters, a generated request/trace ID — are the specific, named mistake that turns a manageable metric into a cardinality explosion. This is explicit guidance from the OpenTelemetry specification itself: high-cardinality values like these belong in trace spans or structured log fields, which are built to handle per-event uniqueness, not in metric labels, which are built to be aggregated over a bounded, enumerable set of values. Use the route template (http.route = /users/{id}), never the raw URL. OpenTelemetry SDKs also cap cardinality per instrument (2,000 attribute sets by default) and fold anything beyond it into a single overflow series marked otel.metric.overflow=true — a guardrail worth knowing exists before you see that label in production.
The failure mode is not a gradual slowdown that gives you time to react — it's the time-series storage and query layer degrading or failing outright once cardinality crosses whatever limit that specific backend was built to handle, which is why this is worth designing around deliberately (choosing which dimensions actually need to be metric labels versus trace/log attributes) rather than discovering only once a well-intentioned "let's add more detail" label change takes down the metrics pipeline it was added to.
Making the metric-to-trace jump actually work
"Pull a trace for an affected request" assumes that request was traced — usually it wasn't. Most systems sample traces at 1-10% at the start of the request (head-based sampling), decided before anyone knows whether the request will be slow or fail. The slow request behind your p99 spike was probably thrown away. Two things fix this:
- Tail-based sampling in the OpenTelemetry Collector buffers each trace until it finishes, then keeps every error and every slow trace plus a small random sample of the rest. The interesting requests survive.
- Exemplars attach a trace ID to individual samples in a metric's histogram bucket. Clicking the slow bucket on the latency graph takes you straight to a trace that landed in it — the real metric-to-trace bridge.
Use histograms for latency, not summaries. A summary computes p99 inside each instance, and you can't average p99s across 20 instances to get the service's p99 — the math doesn't work. A histogram stores bucket counts that add up across instances, so the true aggregate percentile can be computed. Native (exponential) histograms keep good accuracy without hand-picking bucket boundaries.
Run telemetry through a Collector pipeline. The usual shape is an OpenTelemetry Collector agent on each node (or as a sidecar) forwarding to a central gateway, which does tail sampling, drops or renames high-cardinality attributes, and sends data to whichever backends you use — so changing vendors doesn't mean re-instrumenting every service. Continuous profiling is emerging as a fourth signal, answering "which function burned the CPU" where traces stop at the span.
How to say it in the interview
"Logs feel the most thorough because they have the most detail, which is exactly why starting there in a distributed system is slow - you're searching the most expensive, highest-volume pillar before you've used the cheap ones to narrow down where to even look. Metrics tell you something is wrong; a trace tells you where in the call chain; only then do logs from that specific span explain why."
Checklist
- Started an investigation with metrics (alert), then a trace (localize), then logs (explain) — not logs first
- Understood a metric has already discarded per-event detail, which is what makes it cheap enough to alert on continuously
- Recognized cardinality multiplies across labels rather than adding
- Kept unbounded-cardinality values (user IDs, full URLs, request IDs) out of metric labels, in trace spans or logs instead
- Used tail-based sampling and exemplars so the slow request's trace actually exists and is one click from the metric
- Used histograms rather than summaries for latency across instances