Cruxly › System design › Reliability and Observability
Distributed Tracing System
reconstructing one request's path across fifty services from logs that don't know they're related
Open the interactive lessonPrompt: design a distributed tracing system (like Jaeger or Zipkin) — a single user request fans out across dozens of microservices, and when it's slow or fails, an engineer needs to see the entire request's path, timing, and where it broke, as one coherent picture.
40 minutes first. Every individual service already has logs. The entire exercise is connecting them.
Applying the four moves: Scope — follow one request across dozens of services and see its path, timing, and failure point as one picture; log aggregation and metrics out. Estimate — spans per request × request rate, which is large enough that storing every trace is usually unaffordable — so sampling is forced by the arithmetic. Sketch — context propagation (a trace ID and parent span ID carried through every call), span collection, and a trace store indexed by trace ID. Deep dive — sampling: head-based sampling decides before you know whether the request was interesting, tail-based keeps the slow and failed ones but has to buffer complete traces first — name which one and what it costs.
Why per-service logs alone can't answer 'why was this request slow'
Scope. In scope: propagating trace context across service boundaries, span collection, storage, and a queryable UI concept, sampling strategy. Out of scope: the specific storage engine's internals, alerting on trace data (a downstream consumer of this system, not part of it).
Each service's own logs are complete for that service, and useless for answering a cross-service question. The order service logs "processing order 4471, took 40ms," the payment service logs "charged card, took 200ms," and the inventory service logs "reserved item, took 15ms" — each log is accurate, and none of them, on their own or grepped together by timestamp, reliably reconstructs "this one user's checkout request took 280ms, and here's the exact sequence and parent/child relationship of every service call inside it," especially once there are dozens of concurrent requests from different users interleaving through the same services at the same time. The core insight is that this has to be solved by injecting shared identity into every request as it happens, not by cleverly correlating independent logs after the fact — post-hoc correlation from timestamps and heuristics is unreliable at any real concurrency.
Trace context propagation: the one thing every service must do
A trace ID is generated once, at the system's entry point (the API gateway, typically), and must be propagated through every single downstream call — passed as a request header (traceparent, in the W3C Trace Context standard) that every service is required to read from its incoming request and forward on every outgoing call it makes. Each individual unit of work (one service handling one call) gets its own span ID, plus a reference to the span ID of whichever call caused it — this parent-child reference is what lets the collector reconstruct the full call tree afterward, purely from independently-reported spans that each only know their own trace ID and immediate parent.
The critical discipline this requires of every service in the fleet: propagate the trace context on every outgoing call, even calls to services the trace itself doesn't care much about. A single service in the chain that fails to forward the trace header breaks the chain at that point — every span below it in the call tree either gets a fresh, disconnected trace ID or, worse, silently vanishes from the reconstructed trace, producing a confusing gap that looks like "this branch of the request did nothing" when it actually did plenty, just invisibly. This is why tracing libraries are typically built into a shared HTTP client/framework layer rather than left to each service to implement correctly by hand — correctness here depends on universal compliance, and manual per-service implementation is where that compliance breaks.
Sampling: why you cannot and should not trace every request
Recording a full trace for every single request, at real production volume, is both too expensive to store and mostly unnecessary — a million identical, fast, successful requests produce a million nearly identical traces with no diagnostic value. Sampling decides which requests get fully traced, and the decision needs its own design:
- Head-based sampling — decide whether to trace a request at its entry point (e.g., 1% of all requests, chosen randomly), and propagate that yes/no decision alongside the trace ID so every downstream service knows whether to record its span or skip it. Simple and cheap, but a request that turns out to be slow or erroring might just as easily have been in the 99% that wasn't sampled — you don't get to trace the interesting cases specifically, only a random cross-section.
- Tail-based sampling — every service does record its span data temporarily, but the decision to actually persist the full trace is made only after the request completes, based on whether it was actually slow, erroring, or otherwise interesting. This reliably captures the requests worth looking at, at the cost of needing to buffer every span temporarily somewhere until the completion-time sampling decision is made — a real storage and complexity cost head-based sampling avoids.
Naming this tradeoff explicitly — cheap-but-blind versus expensive-but-targeted — and picking one with a stated reason, is the actual test in this section, not simply knowing that "sampling" is a word associated with tracing systems.
Failure modes and the honest edges
Clock skew across services makes span timing comparisons unreliable at face value — if service A's clock is 50ms ahead of service B's, a child span's reported start time can appear to be before its parent's, which is causally impossible but will show up in raw data if clocks aren't treated carefully. Real systems either rely on relative durations rather than absolute timestamps for display, or run NTP-disciplined clocks tightly enough that the error is smaller than the granularity that matters — worth naming as a real, easy-to-miss data-quality issue.
Asynchronous, queue-based hops break naive trace propagation. A request header naturally propagates across a synchronous HTTP call, but if a service publishes a message to a queue and a different, decoupled consumer picks it up later, there's no HTTP request to attach a header to — the trace context has to be explicitly embedded in the message payload itself and read back out by the consumer, or that segment of the request's journey silently disappears from the trace.
Storage volume for a large fleet, even with sampling, is still substantial — with dozens of services each producing spans for every sampled request, trace data itself becomes a significant storage and retention-policy problem, typically solved with the same downsampling- over-time ladder (full detail for recent traces, aggressively pruned or deleted after a retention window) used for this track's metrics-pipeline exercise, applied to trace data instead of numeric time-series points.
How to say it in the interview
"Every service involved in one request already logs its own activity perfectly well in isolation — the actual problem is that none of those logs know they belong to the same request, and the fix is threading one identifier through every hop before any single service is called, not analyzing logs after the fact."
Checklist
- Explained why post-hoc log correlation across services is unreliable, requiring identity injected at request time instead
- Propagated a single trace ID from the entry point through every downstream call via request headers
- Used span ID plus parent-span-ID references to let independently-reported spans reassemble into a call tree
- Identified that a single non-compliant service silently breaks the trace below it, not just for its own span
- Recommended building trace propagation into a shared client/framework layer rather than per-service manual implementation
- Compared head-based and tail-based sampling on a real tradeoff — cheap-but-blind versus expensive-but-targeted
- Named clock skew across services as a real cause of causally-impossible-looking span timestamps
- Identified that asynchronous, queue-based hops need trace context embedded in the message payload, not a header
- Applied a downsampling-over-time retention ladder to trace storage, the same idea used for the metrics-pipeline exercise