Cruxly › AI engineering › NLP and Text Classification

Multilingual and Domain-Specific Text

token inflation, code-switching, uneven degradation, and per-language evaluation

Prompt: the ticket classifier works well and is being rolled out across India. Tickets arrive in English, Hindi, Hinglish, Tamil and Marathi. The aggregate macro F1 on a new multilingual test set is 0.84, which looks acceptable, and the LLM fallback path's cost per ticket has roughly tripled with no change to the prompt. Explain both observations and design what you would actually do.

40 minutes first. The cost increase and the reassuring aggregate score have different causes, and neither is fixed by "use a multilingual model.

Token inflation: the cost problem hiding in the tokenizer

Subword tokenizers are fit on a training corpus that is overwhelmingly English, so scripts that are rare in that corpus get chopped into far more pieces per unit of meaning. Latin-script English text runs roughly 4 characters per token. Devanagari, Tamil, Telugu and Bengali text commonly runs 1 to 2 characters per token, because the vocabulary lacks the frequent subwords of those languages and falls back toward near-character or even byte-level segmentation, with a multi-byte UTF-8 character sometimes costing more than one token by itself.

Work the same sentence across languages. "My payment failed and I was charged twice" and its translations. Treat the table as illustrative round numbers, not a measurement of one tokenizer. Measured with tiktoken on 2026-10-24 on my own translations of this sentence (54 and 76 characters for Hindi and Tamil), the older cl100k (GPT-4-class) tokenizer gave 8 tokens for English, 56 for Hindi and 106 for Tamil, i.e. even worse than the table, while o200k (GPT-4o onward) gave 8, 17 and 24. Large-vocabulary tokenizers (o200k, the ~256k-vocabulary Gemini/Gemma tokenizers) cut Indic inflation substantially, so the multiple for your model has to be measured, not quoted:

That is the tripled cost, fully explained, with no model change involved. A 600-token English prompt becomes ~2,000 tokens in Hindi, so per-item API cost goes up by the same 3.4×, and it is invisible in any dashboard that tracks cost per request without splitting by language. Instrument tokens-per-request by detected language — that single chart turns "the bill went up" into "Tamil tickets cost 4.5× what English ones do," which is a fact you can act on.

Then the second, less obvious consequence: it eats the context budget. At the table's 1.2 characters per token, a 512-token window holds only about 600 characters of Tamil — a long ticket's worth of English fits easily, but a long Tamil ticket gets cut off. (Encoder tokenizers such as BERT's differ from LLM tokenizers, so measure chars/token on the exact model you serve.) And the failure is silent — the model returns a confident prediction on the first third of the ticket, and the fact that the actual complaint was in the truncated remainder appears nowhere in the output. Every per-language quality number computed without checking truncation rate is confounded by it, so log the truncation rate per language before you interpret any per-language score.

Four things that actually reduce it, in order of leverage. Prefer a model whose tokenizer was trained with meaningful representation of your scripts — the character-per-token figure is measurable in an afternoon on 1,000 real tickets per language, and it is a legitimate model-selection criterion nobody checks. Prefer a multilingual long-context encoder so inflation stops causing truncation — XLM-R, MuRIL or IndicBERT v2 for Indic coverage, or mmBERT (a 2025 multilingual ModernBERT-style model with an 8K window). Don't reach for English ModernBERT here: its tokenizer and pretraining don't cover Tamil or Marathi, so a longer window of badly-tokenized text doesn't help. For the encoder path specifically, extending the tokenizer vocabulary with frequent subwords from your languages and continuing pretraining is a real intervention that cuts token counts materially. And do not translate to English as a preprocessing step unless you have measured it, because translation adds latency, cost and its own error rate, and it destroys exactly the code-switched and colloquial signal that the next section is about.

Code-switching: the case that breaks the pipeline's assumptions

"Mera payment fail ho gaya, please refund kar do urgently" is one sentence, two languages, one script, and every stage of a conventional NLP pipeline has an assumption it violates. Hinglish is not an edge case in the Indian market; in consumer support it is frequently the plurality of traffic, above both pure English and pure Hindi.

What it breaks, stage by stage. Language detection returns a single label with high confidence, and whichever it picks is wrong for half the tokens — so any "detect language, then route to the per-language model" architecture routes this ticket incorrectly by construction, and does it confidently. Tokenizers handle the Latin-script Hindi words as unrecognized character sequences, since "kar do" and "ho gaya" appear in no English subword vocabulary. Transliteration is not standardized: kripya / kripaya / krupya, nahi / nahin / nai, paisa / paise / pesa are all the same words with different spellings, and each variant is a different token sequence, so the lexical signal fragments across spellings in a way an English-only model never encounters. And translation preprocessing mangles it — a translation system fed code-switched input typically produces something either garbled or blandly paraphrased, losing the urgency and register that carry real classification signal.

Three design responses, in the order you should try them.

Drop the language-routing architecture. A single multilingual model handling all traffic beats per-language models plus a detector, primarily because the detector is unreliable on exactly the largest traffic segment. This also removes an entire failure mode instead of tuning it, and saying so is the strongest version of this answer.

Get code-switched examples into training and evaluation as their own category. Not "Hindi" and "English" buckets with Hinglish distributed between them, but an explicit Hinglish slice with its own metrics — because a model that looks fine on both monolingual slices routinely fails on the mixed one, and a taxonomy of test slices that has no name for your largest segment cannot report on it.

Normalize transliteration variants where it is cheap. A modest lookup table of the few hundred most frequent Hindi words and their common romanizations, applied as preprocessing, consolidates the fragmented lexical signal. It is unglamorous, takes a day, and typically moves Hinglish F1 by several points — more than most model changes, and worth saying out loud precisely because it sounds too simple to mention.

And name the annotation consequence. Labeling Hinglish tickets requires annotators fluent in both languages and in the colloquial register, which is a smaller hiring pool and a slower annotation rate. If your labeling process quietly routed Hinglish to whoever was available, the kappa on that slice is probably your worst — and per-slice kappa from the labeling lesson is how you find out.

Why a 'multilingual' model degrades unevenly, and per-language evaluation

Multilingual capability is not a property a model either has or lacks; it is a per-language quantity that tracks that language's share of pretraining data. English is a large fraction of most web-scraped corpora. Hindi is a small one. Marathi and low-resource Indian languages are a fraction of that. The resulting quality profile mirrors the data profile, so degradation is stepped and uneven, not a smooth decline — and it moves in a way that is not predictable from the language's number of speakers, which is why "Marathi has 80 million speakers so it should be fine" is a wrong inference people make out loud.

Decompose the 0.84 from the prompt and the whole picture changes.

0.40(0.91) + 0.30(0.83) + 0.18(0.79) + 0.08(0.68) + 0.04(0.61) ≈ 0.83-0.84. The aggregate is arithmetically correct and operationally useless. Marathi users are getting a classifier that is wrong nearly 40% of the time, and the aggregate is structurally incapable of showing it because Marathi is 4% of the weight. This is the same averaging failure as weighted-versus-macro F1 in the previous lesson, one level up: report per language, and report macro across languages, or you have not evaluated a multilingual system.

Then the per-language and per-class version, because the interaction is where the real failures live. A 12-class model across 5 languages is 60 cells, and the failures cluster: a specific class in a specific language, usually where that class's signal is lexical and the language's vocabulary coverage is thin. Aggregating over either axis hides it. You do not need 60 fully-powered test sets, but you do need a per-language test set for each language you claim to support — at minimum 100-200 items per class for your top two or three languages, and a smaller but real set, several hundred items total, for every other language you serve. An unmeasured language is an unsupported language, and that is the sentence to say when someone proposes launching without it.

Set per-language SLOs and make the gap a launch decision rather than a discovery. Options for Marathi at 0.61, stated as trade-offs: collect more Marathi labels, which is the real fix and the slowest; route Marathi to the LLM fallback, which costs 3.7× the tokens but works acceptably zero-shot and is often the right interim answer; lower the automation threshold for Marathi so more of it goes to human review, spending review capacity instead of accuracy; or do not launch Marathi automation and say so explicitly. All four are defensible. Shipping 0.61 behind a 0.84 headline is the only option that is not, and framing it that way is what the exercise is testing.

Domain adaptation: when the words are English but the meaning isn't

The same failure shape appears without any language change at all, because jargon is a low-resource language. A general-purpose model reads "the pod is crashlooping after the last deploy" or "patient presented with acute SOB" or "the NAV is stale on the T+1 reconciliation" as ordinary English and gets the tokenization right and the meaning wrong. Domain vocabulary was rare in pretraining, so it fragments into meaningless subwords — "crashlooping" becomes three pieces, "SOB" is either an abbreviation for shortness of breath or an insult depending entirely on domain context — and the model's representation of the term carries no domain semantics.

Three interventions, in increasing cost, and the honest guidance is that most teams should stop after the first.

Fine-tune a general encoder on your labeled domain data. This is the default and it is usually enough. A few thousand labeled in-domain examples teaches the classifier which surface patterns matter for your task, even where the underlying token representations are mediocre.

Continue pretraining on unlabeled in-domain text before fine-tuning — domain-adaptive pretraining. Take a few hundred thousand to a few million unlabeled documents from your domain, run masked-language-model training for a few GPU-hours, then fine-tune on labels. Typical gain on a specialized domain is a few points of F1, concentrated exactly on the jargon-heavy examples, and its real advantage is that it needs no labels — unlabeled in-domain text is usually abundant where labels are not. Extend the tokenizer vocabulary with your top few thousand domain terms in the same pass and the gain is larger, because "crashlooping" becomes one token that can carry a meaning.

Use a domain-pretrained model if a credible one exists for your field — biomedical, legal, financial and code domains all have them. Free gain when the domain matches. But check whether it was pretrained in your languages too, because a domain-specific model is very often English-only, and then you have traded the multilingual problem for the domain problem. That trade-off is frequently the actual decision in an Indian-market product and naming it is a strong signal.

The general rule to state, because it unifies this lesson: both a low-resource language and a specialized domain are the same phenomenon — text underrepresented in pretraining — and they respond to the same three levers, which are more in-domain data, a tokenizer that knows your vocabulary, and per-slice evaluation so you find out which slice is failing. Aggregate metrics hide low-resource slices by construction, whether the slice is a language, a dialect, a jargon domain, or a rare class. That sentence is the whole topic, and it is the one to end on.

How to say it in the interview

"Two separate things are happening. The cost tripled because tokenizers are trained predominantly on English, so the same sentence in Devanagari or Tamil script costs several times more tokens — that is a billing and context-budget problem before it is a quality problem. And a 0.84 aggregate is hiding an uneven spread, because multilingual models do not degrade smoothly across languages; I would expect something like 0.91 on English and 0.6 on Marathi, which means one market is being served badly and the aggregate is designed not to show it."

Checklist

• Explained the cost increase as tokenizer inflation, before discussing quality
• Asked for tokens-per-request split by detected language
• Named silent truncation as the worse consequence of inflation, not the cost
• Logged truncation rate per language before interpreting any per-language score
• Rejected the detect-language-then-route architecture because of code-switching
• Treated code-switched text as its own evaluation slice with its own metrics
• Proposed transliteration normalization as a cheap, high-leverage fix
• Decomposed the aggregate score per language and showed the arithmetic
• Stated that degradation tracks pretraining data share, not speaker count
• Set a per-language SLO and named the four defensible responses to a gap
• Treated jargon as a low-resource language with the same three levers
• Checked whether a domain-pretrained model covers the required languages

Related lessons

• Text Classification: When a Fine-Tuned Encoder Beats an LLM
• Building the Labeled Dataset
• Evaluation Beyond Accuracy