Cruxly › Forward deployed engineer › Decomposition Cases
Re-Engineering a Broken On-Site Pipeline
the system is down right now, and the fastest fix and the right fix are not the same fix
Open the interactive lessonPrompt: you're on-site at a manufacturing customer. Their overnight data pipeline — which feeds a dashboard the morning shift supervisors rely on — has been silently producing stale numbers for the last three days. Nobody noticed until a supervisor made a staffing decision based on wrong data yesterday. You have today to fix it, and the customer is understandably anxious.
45 minutes first. There is real pressure to patch whatever broke and move on. The actual job here has two separate problems wearing one incident: making today's number correct, and making sure "silently wrong for three days" cannot happen again undetected.
Clarifying questions that matter here
"What actually failed — did the pipeline error out, or did it run and produce wrong numbers silently?" These are completely different problems. An erroring pipeline is usually a fast, visible fix. A pipeline that ran successfully and produced wrong output for three days without any alert firing means either there's no data-quality monitoring at all, or the monitoring exists and didn't catch this specific failure mode — and that gap, not today's specific bug, is the actual incident.
"Who else, besides this dashboard, consumes this pipeline's output?" If the same stale data fed anything else — a report, another team's system, an automated decision — the blast radius of the three-day staleness is larger than the one dashboard everyone's currently focused on, and needs to be checked before declaring the incident closed.
"What decision did the supervisor make yesterday based on the wrong number, and can it be corrected?" This isn't just diplomacy — it tells you the real-world cost of this specific failure mode, which calibrates how much monitoring investment is actually justified afterward. A staffing decision that's easily reversed is a very different severity than one that isn't.
"Was there a recent change — a deploy, a config edit, an upstream data-source change — around when this started?" Three days of silent staleness usually has a trigger, and finding it is usually faster than reasoning from first principles through the entire pipeline. This question alone often cuts the debugging time in half.
The decomposition: fix today's number, separately from fixing the blindness
Slice 1 (today, first hour) — determine the actual current correctness of the dashboard and communicate it, before touching any code. The most urgent thing is not a fix, it's an accurate answer to "can the morning shift trust today's number or not" — because a wrong fix rushed out under pressure that produces a different wrong number is worse than an honest "don't trust this yet, we're on it."
Slice 2 (today, next few hours) — find and fix the root cause of the specific staleness, using the trigger-hunting question from clarifying (recent deploys, config, upstream changes) rather than a broad rewrite. The goal here is narrowly "make today's pipeline run correctly again," not "improve the pipeline" — scope discipline under pressure matters as much as it does anywhere else, because a broad refactor attempted in a single anxious day is how a one-bug incident becomes a two-bug incident.
Slice 3 (this week, separate from the fire-fighting) — add the monitoring that would have caught this on day one. A simple check — does this pipeline's output differ meaningfully from yesterday's in ways that make sense, does it complete within its expected time window, does row count fall inside a sane range — turns "silently wrong for three days" into "loudly wrong within the hour," which is the actual fix to the actual incident. This is explicitly scoped as separate, slower work, done after the fire is out, not bolted onto the same rushed day.
Rebuilding trust, and the follow-up that actually matters
The supervisor's trust in the dashboard, not just the dashboard's correctness, is what was actually damaged, and a silent code fix does not repair that on its own. Communicating clearly what went wrong, in plain terms, and what's now in place so it can't happen silently again is part of the deliverable — a system that's technically fixed but that the morning shift now quietly double-checks by hand every day has not actually solved the customer's problem.
"Can this happen again in a different way we haven't thought of?" is the honest question to ask yourself once the immediate fire is out, and the honest answer is usually yes — the specific bug is fixed, but the class of failure (a pipeline can run "successfully" while being wrong) is general. The monitoring slice from the decomposition should be framed to the customer as addressing that general class, not just this specific incident, because that's what actually restores confidence: not "we fixed the bug" but "we can no longer be wrong about this without knowing."
The postmortem is customer-facing, not just internal, and it's a forward-deployed engineer's job to write it in terms the anxious stakeholder can act on: what happened, what it cost, what's different now, and — critically — what to do the next time something looks slightly off, so the next anomaly gets reported in an hour instead of discovered three days later by a bad staffing decision.
Follow-ups they will ask
"Can you guarantee this specific failure never happens again?" The honest answer is no, and saying so directly is stronger than a reassuring-sounding promise you cannot keep: what can be guaranteed is that this specific failure mode is now monitored, and that the general class it belongs to (a pipeline running successfully while silently wrong) has a detection layer where before there was none. Overpromising here trades a moment of customer comfort for a much worse conversation the next time something different breaks.
"We want the same monitoring on our other twelve pipelines, today." The temptation is to say yes to rebuild trust immediately, but the monitoring built for this pipeline was shaped by its specific failure — the checks that would have caught a three-day staleness here are not automatically the right checks for a pipeline with a different shape and different failure modes. The honest scope is triaging which of the twelve are highest-risk first, rather than mechanically copying one pipeline's monitoring config onto eleven others and calling the job done.
"The root cause was a config change nobody remembers making. How do we stop that?" This pushes past the specific incident into a process gap: if nobody can say who changed a config that broke a production pipeline, there's no change-tracking discipline at all. The technically narrow fix (revert the config) does not touch this, and naming it as a separate, real finding — worth raising even though it wasn't what today's fire was about — is part of being honest about what the incident actually revealed.
"How much did the three days of bad data actually cost us?" This is worth answering even when it's uncomfortable, because a customer who never gets a real cost number cannot judge whether the incident response was proportional, and a forward-deployed engineer who avoids the question reads as someone managing perception rather than solving the problem — the credibility cost of dodging it usually exceeds the cost of an honest, even unflattering, number.
How to say it in the interview
"I want to separate two things that are getting collapsed into one panic: getting today's dashboard number correct again, which is urgent, and understanding why it went silently wrong for three days without anyone knowing, which is the actual failure. Fixing only the first one means we're back here in a month."
Checklist
- Distinguished 'the pipeline errored' from 'the pipeline ran and produced silently wrong output'
- Checked who else consumes this pipeline's output before declaring the incident's blast radius understood
- Asked about recent changes near the start of the staleness window before debugging from first principles
- Determined and communicated today's actual data correctness before attempting any code fix
- Scoped today's fix narrowly to the specific root cause, not a broader rewrite under pressure
- Scoped the monitoring fix as separate, slower follow-up work, not bolted onto the same rushed day
- Framed the fix and follow-up to the customer as addressing the general failure class, not just this specific bug