Cruxly › Forward deployed engineer › Decomposition Cases

Detecting a Rare Pattern in High Volume

no labeled data, no ML spec, and a customer who wants a number by Friday

Prompt: a shipping-logistics customer wants to flag packages that might contain prohibited items, based on package metadata (weight, dimensions, declared contents, sender/recipient history) — they process millions of packages a day, the pattern they're worried about is rare, and they have no labeled historical data of confirmed violations. They'd like "something working" to evaluate by the end of the week.

45 minutes first. There's no dataset, no ground truth, and a five-day clock. The instinct to reach for a model is exactly the instinct this prompt is designed to test, because a model needs labels that don't exist yet.

Clarifying questions that matter here

"When staff have caught a prohibited item before — manually, by luck, through a tip — what tipped them off?" Even with zero formal labeled data, there is almost always informal institutional knowledge: certain declared-content categories, certain sender patterns, certain weight/dimension mismatches that experienced staff have learned to eye. This is the closest thing to ground truth available, and it's usually never been written down.

"What happens today when something suspicious is found — is there a manual inspection process, and what does it cost?" This defines the actual constraint on any detector's false-positive rate. If manual inspection is expensive and slow, a detector that flags 5% of volume is operationally useless even if it's statistically excellent, because nobody can act on that many flags.

"Is there any external labeled reference — a customs seizure list, a known-violation database from an industry body?" Companies rarely have their own labels but sometimes have access to shared industry data that can bootstrap a first pass, which changes "build a detector from nothing" into "adapt a starting signal to this customer's specific volume and format."

"What does 'something working to evaluate by Friday' actually mean to the stakeholder asking for it?" If it means "show me you understand the problem and have a credible plan," a demo of a simple, explainable rule set against real data is a completely achievable and honest answer. If it means "show me a system already catching real violations," that's a different, likely un-meetable ask that needs to be corrected now, gently, rather than discovered as a disappointment on Friday.

The decomposition: build the labeling loop before the detector

Slice 1 (days 1-2) — turn informal staff intuition into an explicit, explainable rule set. Sit with whoever has manually caught something before and translate their intuition ("this weight-to-declared-contents ratio looks off," "this sender has had three returns flagged this way before") into a small number of concrete, checkable rules. This is not the final detector — it's a first-pass filter that's honest about being a first pass, and it's achievable by day two because it requires no model training, only conversation and simple logic.

Slice 2 (days 3-4) — run the rule set against real volume and route flags into a lightweight review queue, not an automated action. The queue's purpose is double: it gives the operations team something immediately useful (a shortlist instead of manually scanning everything), and every reviewed flag — confirmed or dismissed — becomes the first labeled data point this customer has ever had. The detector isn't the deliverable this week; the labeling pipeline is, and that reframing is the single most important thing to communicate clearly, because it changes what "success by Friday" should even mean.

Slice 3 (day 5) — present the rule set's real precision on this week's actual flagged volume, honestly, alongside the plan for what comes next. If it caught anything real, that's a genuine result. If it didn't, or the operations team dismissed most flags, that's also a genuine, valuable result — it tells you which of the informal signals from day one actually hold up against real volume, which is exactly the information needed to decide the next iteration.

Managing the expectation gap, and what 'success' looks like honestly

The gap between "a working detector" and "the start of a labeling pipeline" has to be closed explicitly with the stakeholder, ideally on day one, not discovered awkwardly on Friday. The honest framing: "there's no historical data confirming what a real violation looks like in your specific volume, so anything claiming to be a trained, evaluated detector by Friday would be presenting an untested guess as a result. What I can show you Friday is a system that's already surfacing candidates for review and building the evidence base that makes a real detector possible after that — which is genuinely faster than it sounds, because most of this problem's difficulty was never having any labels at all."

A rule set presented honestly as a first pass, with a visible plan for what it becomes, is a stronger Friday outcome than an impressive-looking model whose actual accuracy nobody can state, because the second one creates false confidence that fails expensively later — exactly the failure mode this whole category of interview and this whole category of job is built to select against.

The follow-up question to ask yourself before Friday: "if this rule set flags something that turns out to be completely benign, what does that cost the customer, and have I made that cost visible?" A shipping delay for a false positive is a real, if small, business cost, and naming it — rather than only celebrating the catches — is what separates a credible plan from a demo optimized to look good in the room.

Follow-ups they will ask

"Three months in, we now have thousands of reviewed flags. Build the real model." This is the moment the week-one framing was pointed toward, and the temptation is to treat it as a pure modeling exercise — but the reviewed flags are not a clean labeled dataset; they're biased by whatever the rule set chose to surface in the first place, so anything the rule set never flagged is invisible to the new model too. The honest next step names this selection bias directly and proposes a way to sample some never-flagged packages for review, rather than training confidently on a dataset that was never a random sample of reality.

"The rule set's precision has been dropping for the last month. Why?" Before assuming the rules broke, ask what changed on the other side — a customer's product mix shifting, a sender population evolving, or a seasonal pattern the rules were never tuned for. Treating a precision drop as purely a modeling problem, when it might be a data-drift problem, is the same category of mistake as jumping to a fix before understanding a bug: the honest first move is diagnosis, not immediate retuning.

"Regulators are asking why a specific package was flagged. Can you explain the model's decision?" This is exactly why the explicit rule set from week one was the right choice over an opaque model, and it's worth saying so directly when this question comes up: every flag traces to a specific, stateable rule, which a black-box model could not offer without a separate, harder explainability project. The follow-up question worth asking yourself is whether a future model-based version preserves this property or quietly gives it up for a precision gain.

"A competitor's detector claims 99% accuracy. Why isn't ours there yet?" A headline accuracy number on a problem with a rare positive class is close to meaningless without knowing the false-positive rate and what "accuracy" was measured against — a detector that simply never flags anything is also extremely "accurate" on a rare-event problem, because it's right most of the time by doing nothing. The honest response reframes the comparison around the actual operational metric (cost per false positive, catches per week) rather than accepting a number that doesn't measure what matters here.

How to say it in the interview

"There's no labeled data, which means 'train a detection model' isn't a Friday-achievable plan, or possibly even a real plan yet at all — it's a plan for a project that doesn't exist. What I can build by Friday is the thing that starts generating the labels and the operational muscle this needs, which is actually the harder and more valuable problem underneath the one they asked for."

Checklist

Recognized that 'train a detection model' isn't achievable with zero labeled data and said so directly
Asked what tipped off staff the times they caught something manually before
Asked what manual inspection costs today, before proposing any detector's target false-positive rate
Asked what 'something working by Friday' actually means to the stakeholder
Built an explicit, explainable rule set from staff intuition rather than attempting a trained model
Routed flags to a human review queue rather than an automated action
Framed the week's real deliverable as the start of a labeling pipeline, not a finished detector
Named the cost of a false positive to the customer, not just the value of a true catch

Practice this interactively