Cruxly › AI engineering › NLP and Text Classification

Building the Labeled Dataset

label quality caps model quality — no architecture recovers from bad labels

Prompt: you need a labeled dataset for the 12-way ticket classifier. The first attempt already exists: one annotator labeled 5,000 tickets over three weeks, and the model trained on it plateaus at 71% macro F1 no matter what you change about the architecture, the learning rate, or the amount of data. Diagnose it, then design the labeling process you should have run.

40 minutes first. The plateau is the whole clue, and the answer involves no change to the model at all.

The label ceiling: why the plateau is diagnostic

A model cannot be more consistent than its labels. If the same ticket appears twice in training with two different labels — because the annotator changed their mind in week two, or because the guideline was ambiguous — the optimizer's best available behaviour is to split probability mass between them. Aggregate that over a third of your corpus and the model converges to a number set by the label noise, not by the architecture. That is what a plateau immune to every modeling change is.

So the first measurement is agreement, not accuracy. Take 300 tickets from the existing corpus, have two annotators independently re-label them without seeing the original label or each other's, and compute the agreement. The single-annotator corpus has no agreement number at all, which is the actual defect in the process described — you cannot know your ceiling if only one person ever labeled anything.

Raw percent agreement is misleading and you should say why. On a 12-way task where one class is 60% of traffic, two annotators who each picked labels at random according to the class frequencies would agree at least 36% of the time (0.6² = 0.36 from the big class alone, plus a little from the rest) without reading a single ticket. Reporting "we agree 78% of the time" without chance-correcting it overstates the quality of the process.

A concrete diagnosis worth stating. In this scenario the likely finding is agreement in the 0.45-0.60 kappa range concentrated in a few class pairs. Be precise about what that does and doesn't cap:

• It caps the measured score. If the test labels disagree with each other that often, even a perfect model can't score well against them — part of every "error" is label noise in the test set.
• It doesn't automatically cap the model. Random noise in training labels is often averaged out — models learn toward the majority label. What really hurts the model is systematic noise: one annotator consistently labelling "billing" what the other calls "refund."
• Don't compare kappa to F1 directly — they're different units. The useful comparison is the model's score against the score a second human annotator gets on the same test set.

So a model plateauing at 71% macro F1 when a second annotator scores about the same against the first is not underfitting. It is at the measurement ceiling. Saying that out loud, and fixing the guidelines and test labels instead of the learning rate, is the senior move in this conversation.

Inter-annotator agreement: the numbers and what to do at each

Cohen's kappa for two annotators, Fleiss' kappa for three or more. Both answer the same question — how much do annotators agree beyond what chance alone would produce — with the same shape:

kappa = (observed_agreement − expected_agreement) / (1 − expected_agreement)

Worked: two annotators agree on 78% of 300 tickets. The class distribution makes chance agreement 0.36. Then kappa = (0.78 − 0.36) / (1 − 0.36) = 0.42 / 0.64 = 0.66. That "78% agreement" is a 0.66 kappa, which is a materially less comfortable number and the honest one.

The response to a bad kappa is never "hire better annotators." It is almost always the guideline document, and the diagnostic step is to compute kappa per class pair rather than globally. A global 0.55 usually decomposes into nine classes at 0.85 and three specific pairs near chance — "billing" versus "subscription," "bug report" versus "feature request," "account access" versus "technical issue." Those pairs are not annotator failures; they are places where your taxonomy has no boundary. The fixes, in order of how often they are the right one: merge the two classes (if the business cannot articulate a different downstream action for each, they are not two classes); write a tie-break rule into the guidelines ("if the ticket mentions a charge, it is billing regardless of the underlying cause"); or add an explicit third category for the genuinely-both case, which is honest about the ambiguity instead of forcing a coin flip.

Then re-measure. A guideline revision that does not move kappa did not work, and the only way to know is to run the double-labeled sample again. Two rounds of guideline-revise-and-remeasure before scaling up labeling is cheap; discovering a 0.4 kappa after 5,000 items is the three weeks this scenario already wasted.

Two process rules that cost nothing. Annotate an anchor set of 50 items that every annotator labels, so you can compute agreement continuously rather than once, and re-label a small slice from week one at the end, because a single annotator's own definitions drift across three weeks — intra-annotator agreement below 0.8 is a real and commonly-missed finding.

Weak supervision: labeling functions instead of labels

When you have domain knowledge but not annotator hours, write rules that vote instead of labeling items one at a time. A labeling function is a small piece of code that looks at a document and either emits a label or abstains — a keyword match, a regex, a lookup against a product-name table, a metadata check on which form the ticket arrived through, or the output of an off-the-shelf model. Each one is individually mediocre and that is fine, because they are not used individually.

def lf_refund(text):
    return "billing" if re.search(r"\brefund|chargeback|invoic", text, re.I) else ABSTAIN

def lf_cannot_login(text):
    return "account-access" if re.search(r"can.?t log ?in|password reset|2fa", text, re.I) else ABSTAIN

def lf_via_crash_form(ticket):
    return "bug-report" if ticket.source == "crash-reporter" else ABSTAIN

The mechanism that makes this more than a rule engine is the label model. Twenty labeling functions over 200,000 unlabeled tickets produce a matrix of votes and abstentions. A label model then estimates each function's accuracy and their correlations from the agreement structure alone, with no ground truth, and emits a probabilistic label per item. Where functions agree, confidence is high; where they conflict, the estimated-more-accurate function dominates; where all abstain, the item gets no label. The encoder then trains on those probabilistic labels and — critically — generalizes past the rules, learning lexical and contextual patterns that correlate with the rule outputs but that no rule encodes. That generalization is the whole point; a candidate who describes weak supervision as "just use regexes" has missed it.

Two failure modes, both worth pre-empting. Correlated functions double-count: five labeling functions that all pattern-match slight variants of the word "refund" are not five independent votes, and a label model that assumes independence becomes overconfident on exactly the cases those five all fire on. Model the correlations or write functions that look at genuinely different signals — text, metadata, source system, customer tier. And coverage is not uniform across classes: rules are easy to write for classes with distinctive vocabulary and nearly impossible for the residual "other/general inquiry" class, so weak supervision silently produces a training set skewed toward the easy-to-rule classes. Measure coverage per class, and fall back to human labels for the classes where coverage is under some floor.

Where it genuinely wins: hundreds of thousands of unlabeled documents available, real domain knowledge in a subject-matter expert's head, and a need to re-label the whole corpus in an hour when the taxonomy changes — because you edit a function and re-run rather than commissioning another annotation round. That last property is the underrated one. Where it does not win: tasks where the signal is semantic rather than lexical, because no regex captures tone, sarcasm, or implied intent.

Active learning, and the outlier trap

The premise: not all labels are worth the same. A model that already predicts a ticket with 0.99 confidence learns nearly nothing from being told it was right. The item that would teach it most is the one it is most unsure about. So: train on a small seed set, score the unlabeled pool, send the most informative items to annotators, retrain, repeat.

Uncertainty sampling is the standard acquisition function, in three flavours worth distinguishing. Least confidence takes items with the lowest top-class probability. Margin sampling takes items with the smallest gap between the top two classes — usually better on multi-class problems, because it targets specific confusable boundaries rather than generic confusion. Entropy takes items whose full predicted distribution is flattest, which is the right choice when many classes are plausible at once.

The payoff is real and worth quantifying. Reaching a target macro F1 with roughly half to a third of the labels a random-sampling schedule would need is a typical result on a 12-way task, because random sampling spends most of its budget on items from the majority class that the model already handles.

Now the failure mode, which is the part this section exists for. Uncertainty sampling chases outliers. The items a model is least confident about are not only the informative boundary cases; they include the garbage. Empty tickets, pure-emoji messages, a single "?", automated bounce emails, base64-encoded blobs pasted into the description field, text in a language the model has never seen. A model is maximally uncertain about all of it, so an uncertainty-driven queue fills with items that are unrepresentative, unlabelable, and actively harmful to train on — and your annotators burn their hours on junk while the real boundary cases are still unlabeled. Unchecked, this turns a labeling-efficiency win into a labeling-efficiency loss.

Four mitigations, and a strong candidate names at least two. Combine uncertainty with representativeness: score uncertainty × density so an item must be both confusing and similar to many other unlabeled items, which structurally excludes one-off oddities. Batch with a diversity constraint: selecting 200 items per round by pure uncertainty often returns 200 near-copies of the same confusion, so cluster the candidates and sample across clusters. Give annotators an explicit "unlabelable / junk" button and route those items to a preprocessing filter rather than forcing a label onto noise. And keep a random-sample stream alongside the active stream — say 20% random — because a purely active-learned training set has a distribution that no longer matches production, which quietly invalidates every evaluation number computed on it.

The stopping rule, since active learning has no natural end. Track macro F1 on a fixed held-out set after each round and stop when two consecutive rounds move it by less than the metric's own noise — roughly 0.5 points on a test set of a few thousand. In practice the curve bends hard around a few hundred examples per class and is nearly flat past about a thousand, so the honest answer to "how many labels do I need" is 300-500 per class to get a usable model and ~1,000 per class before more labels stop helping — after which the remaining error is label ambiguity and genuinely hard cases, and more of the same labels buys nothing. Spend the next rupee on guideline quality or on the specific confused class pair instead.

LLM-assisted labeling, and where the human has to stay

The cheapest labeling strategy available in 2026 is to let a frontier model label everything and put humans only where the model is unsure — which is the same pipeline as the hybrid from the previous lesson, seen from the annotation side. 20,000 items at roughly $0.0026 each is about $52 of API cost against $1,500-$4,000 of annotator time.

The discipline that makes it trustworthy is how you find "unsure," and there are three signals, in increasing order of usefulness. Raw token log-probs are the cheapest and the least reliable, because LLM confidence scores are miscalibrated out of the box — a topic the evaluation lesson takes apart. Prompt-variant disagreement is better: run the same item through two differently-worded prompts, or the same prompt with the label list reordered, and route the items where the two runs disagree. Two-model disagreement is best and most expensive: two different models, route the disagreements. In all three cases the disagreement set is typically 10-15% of the corpus, and it is enriched for genuinely hard cases in a way that random sampling is not — which means human hours land exactly where labels are contested.

Three rules that separate a usable LLM-labeled corpus from a plausible-looking one.

The test set is human-labeled, always, and no LLM touches it. If the LLM labels both train and test, the test score measures how faithfully the encoder reproduces the LLM — including every mistake — and it will look excellent while the system is wrong in production. A few hundred human-gold items per class is enough and is non-negotiable.

Audit a random slice, not only the disagreements. The disagreement set finds cases where the LLM was uncertain; it cannot find cases where the LLM was confidently and systematically wrong, which is the dangerous failure. Pull 200 random high-confidence items and have a human check them. A 4% error rate there is acceptable; a 15% error rate concentrated in one class means the label description for that class is wrong, and you have just saved yourself a retraining cycle.

Watch for the LLM collapsing the ambiguous middle onto one class. Where a human annotator would hesitate and eventually pick, an LLM often has a consistent prior and picks the same way every time. This makes the labels look more consistent than human labels — kappa between two LLM runs can exceed kappa between two humans — while being systematically biased on exactly the class pairs the guidelines handle badly. High agreement is not the same as high accuracy, and citing an LLM-versus-LLM kappa as evidence of label quality is a specific mistake worth avoiding out loud.

The closing process for the scenario in the prompt. Throw nothing away, but stop trusting the existing 5,000 as ground truth. Double-label 300 of them to get the kappa and find the confused pairs. Fix the taxonomy — probably merging two classes and writing one tie-break rule. Re-measure kappa to confirm the fix moved it. Then LLM-label 20,000 fresh tickets stratified to production, have a human adjudicate the 15% disagreement set, hand-label 300 per class as a gold test set nothing else touched, and retrain. The plateau moves because the ceiling moved, which was never something the architecture could do.

How to say it in the interview

"A plateau that survives every architecture and hyperparameter change is almost never a modeling problem — it is a label ceiling. If two competent annotators would disagree on a third of these tickets, then a third of the training signal is contradictory, and no model can fit contradictory labels; it will average them and sit exactly where this one is sitting. So the first thing I want is not a bigger model, it is an inter-annotator agreement number, because I currently have no idea whether 71% is the model failing or the labels being unlearnable."

Checklist

• Diagnosed the plateau as a label ceiling rather than a modeling problem
• Asked for an inter-annotator agreement number before touching the model
• Chance-corrected agreement with Cohen's or Fleiss' kappa instead of quoting raw percent
• Computed kappa per class pair and fixed the taxonomy, not the annotators
• Re-measured kappa on a fresh sample after revising the guidelines
• Described weak supervision's label model as estimating LF accuracy without ground truth
• Named correlated labeling functions and uneven per-class coverage as weak supervision's failure modes
• Named the outlier trap in uncertainty sampling and gave a concrete mitigation
• Kept a random-sampled stream alongside the active-learning stream
• Gave a labels-per-class number with a stopping rule attached
• Required a human-gold test set even when training labels came from an LLM
• Audited a random high-confidence slice, not only the LLM's disagreements

Related lessons

• Text Classification: When a Fine-Tuned Encoder Beats an LLM
• Evaluation Beyond Accuracy
• Multilingual and Domain-Specific Text