Cruxly › AI engineering › NLP and Text Classification

Text Classification: When a Fine-Tuned Encoder Beats an LLM

the break-even arithmetic, and the hybrid everyone lands on

Prompt: a B2B SaaS company routes 2 million support tickets a month into 12 categories. Today a frontier-model API call classifies each one. Finance wants the line item cut. Decide whether to move to a fine-tuned encoder, show the arithmetic for both paths — per-item cost, p50/p99 latency, accuracy, and the labeled-data cost one path needs and the other does not — and state where the crossover sits.

Do the numbers on paper before reading on. The trap is answering with a preference. "Fine-tuned models are cheaper" and "LLMs are more accurate" are both defensible and neither is an answer; the answer is a break-even volume with a number attached.

Path A: the frontier API, priced honestly

Start by pricing what is running today, because nobody in the room has usually done it per-item. A support ticket plus a classification instruction and 12 label descriptions is roughly 600 input tokens; the output is a single label, call it 10 tokens. At upper-tier list pricing of $4 per million input tokens and $20 per million output tokens (Claude Opus 5.5's list price on 31 October 2026). Treat it as an illustrative upper-tier assumption and re-price before quoting — the box in the crossover section re-runs the same arithmetic on dated mid and small tiers:

input :  600 tok  × $4/1M   = $0.0024
output:   10 tok  × $20/1M  = $0.0002
per item                    = $0.0026  ≈ ₹0.23
2,000,000 items/month       = $5,200/month  ≈ ₹4.5 lakh/month

Now the latency, which is the part that decides whether it can even be synchronous. A frontier model classifying a ticket returns in roughly 400-900 ms at p50 and comfortably 2-4 seconds at p99 once you include provider queueing, a retry on a 429, and the occasional slow response. The published extreme is worse: a 2025 benchmark measured GPT-5, then OpenAI's flagship, at about 5.5 seconds per instance (a single-paper, single-setup figure; newer models have since shipped, so measure the model you would actually use), but it shows why bulk annotation is impractical — 2 million items at 5.5 seconds of serialized latency is 127 days of wall-clock time, so any LLM-based bulk pass is a concurrency-engineering project, not a script.

Accuracy: give it full credit, because zero-shot is genuinely good. On a 12-way ticket taxonomy with well-written label descriptions, expect something in the low-to-mid 80s macro F1 with no training data whatsoever. That is the real selling point and it should be stated generously — the LLM got there without a single annotated example, which no encoder can do.

Then name the costs that are not on the invoice. The provider can deprecate or silently update the model under you, so last quarter's classification is not reproducible; tail latency is outside your control; the ticket text leaves your perimeter, which is a data-residency conversation rather than an engineering one; and per-item pricing means your cost scales linearly with success forever.

Before you abandon the API, pull its two cheap levers — an interviewer will ask whether you did.

• Batch APIs. Major providers offer asynchronous batch endpoints at roughly half price, with results back within hours. Any classification that doesn't need to be synchronous — the nightly backfill, the bulk labeling pass in the hybrid below — should go through them: ~$2,600/month instead of $5,200 for the same 2M items.
• Prompt caching. The classification instructions and 12 label descriptions are identical on every call. Cached input is typically billed at a small fraction of the normal rate. Check the provider's minimum cacheable prefix, though — some require around 1,024 tokens or more, so a 600-token prompt may not qualify unless the static part is padded with useful content such as few-shot examples.

These can halve the API path's cost and move the crossover below up by roughly the same factor, so do the comparison against the cheapest API configuration, not list price.

Path B: the fine-tuned encoder, including the bill nobody quotes

Name the model class, not a vibe. DeBERTa-v3-base (~184M parameters) is the accuracy-per-parameter workhorse, and ModernBERT-base (~150M) is the current sensible default — a BERT-family encoder retrained with a modern recipe, an 8,192-token window instead of 512, and materially faster inference. Either fine-tunes on a single mid-range GPU in under an hour for a 12-way task.

Inference cost, computed from a GPU-hour rather than asserted. ModernBERT-base on one A10G-class GPU with batching handles on the order of 1,000-2,000 short documents per second. At roughly $1.00/hour:

throughput      ≈ 1,200 docs/sec  → 4.3M docs/hour
cost per item   ≈ $1.00 / 4.3M    ≈ $0.00000023
2,000,000/month ≈ under $0.50 of GPU time

Even provisioning a single instance 24/7 for availability rather than utilization — about $730/month — and even after that deliberate waste the per-item cost is about $0.00037 ($730 / 2,000,000), roughly a seventh of the $0.0026 LLM path at this volume (about 7× cheaper). It falls toward the $0.00000023 compute-only figure as utilization rises. On CPU it is slower but still single-digit milliseconds per document, which is why an encoder can live inside the request path of a web service with no queue behind it.

Latency: p50 around 8-15 ms, p99 around 30-50 ms, and — the part that matters more than the median — that p99 is yours. No provider queue, no 429, no mystery 4-second outlier. For a synchronous "route this ticket as the agent presses submit" flow, that difference is the whole product decision.

Accuracy, with the honest direction of the gap. Fine-tuned on a few thousand labeled examples per the literature, a DeBERTa/ModernBERT-class encoder lands competitive with and often above zero-shot frontier prompting on a fixed label set — call it 88-92% macro F1 against the LLM's low-to-mid 80s. The encoder is learning your taxonomy's actual boundaries, including the ones your guidelines describe badly, which is exactly what a prompt cannot transmit.

Now the bill that never appears in the comparison table. 12 classes × 400 labeled examples = 4,800 items. At 30 seconds per item, two annotators for double-labeling on a subset, plus guideline authoring, adjudication and QA, a realistic figure is 60-100 annotator hours, so roughly $1,500-$4,000 depending on where the annotators sit and whether they are domain experts. Add ~$200 of engineering and GPU time for training, evaluation and the serving wrapper. Call the encoder path $2,000-$5,000 upfront. That number is the entire reason this is a decision and not an obvious win.

The crossover, and reading it correctly

Set the two cost functions equal and solve. Using the fully-loaded figures — $3,500 upfront for the encoder, $730/month to run one always-on instance, $0.0026/item for the LLM:

LLM      cost(V) = 0.0026 × V
Encoder  cost(V) = 3,500 + 730 × months

Over a single month: break-even at 0.0026V = 3,500 + 730, so V ≈ 1.63 million items. Over a twelve-month horizon, the upfront cost amortizes and the comparison becomes 0.0026V = 3,500/12 + 730, so V ≈ 393,000 items/month.

Two things to say out loud about this table. First, the crossover is around four hundred thousand items a month on a one-year horizon, which is high enough that most features genuinely should stay on the LLM — building a labeling pipeline for 20,000 documents a month is engineering theater. Second, the encoder's curve is nearly flat while the LLM's is linear, so the ratio keeps widening: at 20 million items the encoder is not 5× cheaper, it is about 30× cheaper, because you added one GPU instead of ten million API calls.

Dated box: the same arithmetic on three price tiers (list prices read from Anthropic's public pricing page on 2026-10-31; check the current page before quoting any of this). Assumptions: 600 input and 10 output tokens per ticket, 2M tickets a month, no batch or caching discount, encoder at $730/month plus $3,500 upfront amortized over 12 months ($1,022/month).

:::key Verified October 2026 Anthropic's pricing page, read 31 October 2026, lists Claude Opus 5.5 at $4 / $20, Sonnet 5.5 at $2 / $10 and Haiku 5.5 at $0.10 / $0.50 per million input / output tokens (Haiku 5.5 costs more on prompts over 100,000 tokens, which these tickets are far below). Every dollar figure in this lesson follows from those three rows and the token counts above; the GPT-5 latency is a 2025 measurement, not a current one. Prices change, so redo the arithmetic with the current page. :::

Read the last row carefully. On a small tier the API is cheaper than the always-on encoder at 2 million tickets a month, so cost alone no longer picks the encoder; the answer becomes volume × tier × latency/residency. The small tier still has the 2-4 second p99, the reproducibility risk and the data-leaves-the-perimeter problem, and its accuracy must be measured on your labels before you rely on it. The hybrid below remains sound on any tier: it turns the LLM into a one-time labeler and keeps the p99 yours.

Then state the three things that move the line, since an interviewer will push. A cheaper small-model API tier shifts the crossover up by roughly the price ratio (the box above shows by how much) and is the correct first thing to try. Latency requirements can override the arithmetic entirely — if the SLO is 50 ms p99, the LLM path is disqualified at any volume and the cost table is irrelevant. And data residency does the same in the other direction: if the text cannot leave the perimeter, you are self-hosting something, and the only question is whether it is an encoder or a much more expensive open-weights generative model.

The hybrid: LLM to bootstrap labels, encoder to serve

The two paths are not competitors; one of them is the other's data pipeline. The LLM's unique advantage is producing labels with no human annotation budget. The encoder's unique advantage is serving a decision for effectively nothing. Compose them and you pay the LLM's per-item cost once per training example instead of once per production request.

The concrete pipeline, with the volumes attached:

1. Sample 20,000 real tickets stratified to the production class distribution, so the corpus reflects what actually arrives rather than what is easy to find.
2. Label all 20,000 with the frontier model, at a one-time cost of 20,000 × $0.0026 = $52. That is the entire annotation budget, versus $1,500-$4,000 of human hours.
3. Have humans review only where the LLM was unsure — low log-prob, disagreement between two prompt variants, or disagreement between two models. (Not every provider exposes token log-probs, and reasoning models often don't; prompt-variant or model disagreement works everywhere.) That is typically 10-15% of the corpus, so 2,000-3,000 items instead of 20,000, which is roughly 20 annotator hours rather than 80, and it concentrates the human effort exactly on the boundary cases where the labels are actually contested.
4. Fine-tune ModernBERT on the result and evaluate it against a small human-gold test set that no LLM ever touched. That last clause is load-bearing: if your test set is LLM-labeled too, you are measuring how well the encoder imitates the LLM, including its mistakes, and the number will look great while the system is wrong.
5. Serve the encoder. Route the low-confidence tail to the LLM — if the encoder's calibrated confidence is under threshold, roughly 3-8% of traffic, escalate that request to the frontier model. You buy back most of the accuracy gap on hard cases while paying LLM prices on 5% of volume rather than 100%.

Price the composed system: 2 million items where 95% are served by the encoder and 5% escalate. 0.05 × 2M × $0.0026 = $260/month of API cost, plus ~$730 of GPU, plus a $52 one-time labeling pass and ~20 hours of human review. Roughly $990/month against $5,200, with p99 latency of 30-50 ms for 95% of traffic and accuracy at or above the pure-LLM path.

Name the failure mode of the hybrid too, because it is real. Training on LLM labels inherits the LLM's systematic biases — if the frontier model consistently confuses "billing" with "subscription," your encoder will learn that confusion as ground truth and be confidently wrong, which is worse than being uncertain. The human-gold test set and the confusion matrix from the evaluation lesson are what catch it. There is no version of this pipeline that is safe without a test set humans labeled.

What the encoder gives up, and when to say so

Be specific about the encoder's real limitations rather than selling it.

A fixed label set. Adding a 13th category means relabeling, retraining and redeploying — days, not minutes. The LLM path adds a category by editing a string. For a taxonomy that is genuinely still churning, the LLM's flexibility is worth real money and you should say so instead of pretending the cost table settles it.

No reasoning, no explanation, no generalization past the labels. An encoder emits a distribution over 12 classes. It cannot tell you why, cannot say "this is actually a new kind of problem you don't have a category for," and cannot handle a compound ticket that is two categories at once unless you trained it multi-label on purpose. If the product needs a rationale shown to an agent, that is an LLM feature.

A 512-token window on classic BERT-family models, which silently truncates long documents — and truncation from the front or the back changes which cases you get wrong. ModernBERT's 8,192-token window is the main reason to prefer it for anything longer than a support ticket, and naming that as the reason shows you know what the limitation was.

A model you now operate. Versioning, retraining cadence, drift monitoring, a rollback path, GPU or CPU capacity, and an on-call story. The API path outsources all of that to the provider, and "we own an ML model now" is a real organizational cost that a per-item cost comparison does not capture.

So the closing statement in the room. "At 2 million tickets a month we are five times over the break-even, the latency requirement makes the API path marginal anyway, and the taxonomy has been stable for a year — so: LLM to label 20,000 tickets for $52, humans to adjudicate the 15% it was unsure about, a ModernBERT classifier serving 95% of traffic at 30-millisecond p99, and an LLM escalation path for the low-confidence tail. Roughly a fifth of the current spend, better latency, and no accuracy regression. If the taxonomy were still changing monthly, I would stay on the API and revisit at volume."

How to say it in the interview

"Both options work, so this is arithmetic, not architecture. The LLM path has near-zero setup cost and a per-item cost one to two orders of magnitude higher; the encoder path has a real upfront labeling cost and a per-item cost close to zero. That means there is a monthly volume where they cross, and at 2 million tickets a month, on an upper-tier API price, we are far above it — so the encoder serves, and I would use the LLM to build its labels rather than to answer the requests. On a small-model API tier the cost gap closes and latency, residency and reproducibility decide it."

Checklist

• Computed per-item cost for both paths from token counts and a GPU-hour rate
• Stated the crossover as a monthly volume, with the amortization horizon attached
• Counted the labeled-data cost as a real line item on the encoder path
• Gave the zero-shot LLM full credit for needing no annotated data
• Named the encoder by model class — DeBERTa-v3 or ModernBERT — with a reason
• Separated p50 from p99 and noted that the encoder's p99 is under your control
• Proposed the hybrid: LLM labels the corpus once, encoder serves every request
• Insisted the test set be human-labeled even when training labels came from an LLM
• Routed only the low-confidence tail to the LLM rather than all traffic
• Named a condition that reverses the decision, not just the one that supports it

Related lessons

• Building the Labeled Dataset
• Evaluation Beyond Accuracy
• Multilingual and Domain-Specific Text