Cruxly › AI engineering › LLM Foundations

Tokenization: BPE, Vocabulary Size, and Why Tokens Are the Unit of Everything

build a BPE merge table by hand, then price a vocabulary choice

Prompt: "Our bill went up 30 percent when we moved to the new model version, with the same prompts. Our Tamil users complain the bot forgets the start of long chats. A teammate says the model cannot count the letters in a word. One concept explains all three. What is it, how is the vocabulary built, and how would you estimate tokens for a 10-page PDF?"

40 minutes first. Run byte-pair encoding by hand on five words, then compute what a vocabulary size costs and saves. The trap is treating a token as a word or a fixed number of characters.

BPE builds the vocabulary from pair counts, and you can run it by hand

Byte-pair encoding turns frequency statistics into a vocabulary: count adjacent pairs, merge the most frequent, repeat. Training the tokenizer happens once, on a corpus, before the model is trained; the result is an ordered list of merges. Take a toy corpus with word frequencies: hug x10, pug x5, pun x12, bun x4, hugs x5. Start with characters as symbols. The base vocabulary is the 7 characters b g h n p s u.

round 1 pair counts:  (u,g)=20  (p,u)=17  (u,n)=16  (h,u)=15  (g,s)=5 ...
  (u,g) appears in hug x10 + pug x5 + hugs x5 = 20      -> merge into "ug"
round 2 pair counts:  (u,n)=16  (h,ug)=15  (p,u)=12 ...
  (u,n) appears in pun x12 + bun x4 = 16                 -> merge into "un"
round 3 pair counts:  (h,ug)=15  (p,un)=12  (p,ug)=5  (ug,s)=5
  (h,ug) appears in hug x10 + hugs x5 = 15               -> merge into "hug"

After three merges the vocabulary has 7 + 3 = 10 entries and the words tokenize as hug, p ug, p un, b un, hug s. The corpus is 113 characters; it now needs 62 tokens, a 1.8x compression. A new word bugs becomes b ug s: it was never seen, but it decomposes into known pieces. A word with a character the base vocabulary lacks, say mug with no m, would fail in a character-level tokenizer; byte-level BPE (GPT-2 onward) starts from the 256 byte values, so any string in any script can always be encoded, at worst as raw bytes. That is also why odd Unicode never crashes the tokenizer but can cost many tokens.

Two details matter in practice. Merges are applied in learned order at encoding time, so tokenization is deterministic given the table. And the merge table reflects the training corpus: common English words get a single token; rarer scripts, rare names and rare code identifiers fragment into many. SentencePiece implements BPE and also the unigram model (start with a large vocabulary and prune pieces that hurt the corpus likelihood least); it treats whitespace as a normal symbol, so it works without pre-splitting on spaces, which suits languages without them. In an interview, "BPE or unigram" is a detail; "merges learned from frequency" is the point.

:::key A token is whatever the merge table says it is. It is not a word, a syllable or a fixed number of characters, and it differs between model families and sometimes between versions of the same family. :::

Vocabulary size is a trade: shorter sequences against a bigger matrix

Bigger vocabularies mean fewer tokens per text, and fewer tokens make attention, the KV cache and the bill smaller, at the price of a larger embedding and output matrix. Both sides have a number.

The cost side: the input embedding is V x d and the output (unembedding) matrix is another V x d unless tied. At d = 4096:

V =  32,000:  32,000 * 4096 = 131M   per matrix, 262M for both  (4% of a 6.7B model)
V = 128,256: 128,256 * 4096 = 525M   per matrix, 1.05B for both (about 13% of an 8B model)

The benefit side: Meta's Llama 3 announcement describes moving from a 32K vocabulary to 128K and says the new tokenizer yields up to 15 percent fewer tokens than Llama 2 for the same text. Fifteen percent fewer tokens is 0.85 of the KV cache and, because attention work is quadratic, about 0.85 squared = 0.72 of the attention FLOPs for the same document. It also buys the model more text per context window. The relationship is saturating: the first extra merges absorb very common strings, later ones only rare strings, so each added token earns less.

There is a second cost people forget: rare tokens are under-trained. A vocabulary entry that appears a few times in the pretraining data has an embedding the model barely learned, which is where glitch tokens come from (a string that was frequent in the tokenizer's corpus but rare or absent in the model's training data gives a vector that behaves unpredictably). So the senior answer is not "bigger is better"; it is "vocabulary size is chosen against the languages and code you expect, and then tokens-per-text is measured on your own data."

The multilingual consequence is the same mechanism. A tokenizer whose merges were learned mostly on English has few merges for other scripts, so the same meaning costs more tokens. As an illustration only (measure your own text): if 1,000 English words are about 1,300 tokens, and a Tamil rendering is 4.5x that, then at $2 per million input tokens:

English: 1,300 tokens   * $2 / 1M = $0.0026 per call
Tamil:   5,850 tokens   * $2 / 1M = $0.0117 per call   (4.5x the cost, and 4.5x the context used)

The Tamil user pays 4.5x for the same content and also exhausts the context window 4.5x sooner, which is why that user's bot "forgets the start of long chats" first. The broader treatment, including evaluation per language, is in Multilingual and Domain-Specific Text; the point here is that it is a tokenizer property, not a model-quality property.

Everything is metered in tokens, so count with the right tokenizer

Price, context window, rate limits and latency are all denominated in tokens, so a token count is the unit of account for the whole system. Output tokens usually cost more than input tokens and each is generated sequentially, which ties tokens to latency as well; the cost arithmetic is in Cost Attribution and Runaway Prevention and the latency arithmetic in TTFT, TPOT, and the Latency Budget.

Estimate with a rule of thumb, then measure. English prose runs around 1.3 tokens per word. A 10-page PDF of 500 words per page is 5,000 words, about 5,000 x 1.3 = 6,500 tokens. That is a planning number. It breaks for scanned PDFs (the OCR or vision path has its own cost), tables, code, non-English text and anything with long numbers or IDs. For a real budget, count a sample with the actual tokenizer and multiply.

Always count with the tokenizer of the model you will call. Another vendor's tokenizer is a different merge table and gives a different answer; a newer model from the same vendor can too. Anthropic's token-counting documentation says its count is an estimate that can differ by a small amount from billing, and that Claude 4.7 and later models use a newer tokenizer that produces approximately 30 percent more tokens for the same text than earlier models, with the exact increase depending on content, and it tells you to recount against the model you plan to use. That is the answer to "why did the bill change with the same prompt":

prompt: 1,300 tokens on the old tokenizer, +30% = 1,690 tokens on the new one
input cost at $2 / 1M tokens, 100,000 calls: $260.00 -> $338.00 (+$78.00)

Nothing in the prompt changed. The unit changed.

:::key Verified October 2026 Token-counting behaviour and the 30 percent figure are from Anthropic's token-counting documentation, retrieved October 2026. The multiplier is provider- and version-specific; re-verify before quoting it, and use the provider's count endpoint rather than a local library for anything you bill against. OpenAI's open-source tiktoken is accurate for OpenAI encodings only. :::

Budget the limits in tokens too. A context window is a token budget shared by system prompt, tools, history, retrieved text and the answer. If your output cap is too low the model is truncated mid-sentence and you pay for a useless answer; if the history grows by 1,500 tokens a turn, a 128,000-token window is gone in about 85 turns before you count anything else. Multi-Turn Conversation State and Context-Window Limits shows how to manage that.

Tokenization-caused failures, and chat templates

Name the failure by its cause: the model does not see characters. A model that sees straw + berry as two ids has no direct access to the letters inside them. Counting the r's in "strawberry" requires it to have memorised the spelling of each token from text about spelling, which works often and fails unpredictably. The exact split depends on the tokenizer, so treat any specific split as an example. The fix is to let code do character work: give the model a tool, or ask it to spell the word out letter by letter first.

The same cause explains these:

Arithmetic on long numbers. Digits are chunked inconsistently (1234567 may split as 123|456|7 or 1|234|567), so alignment of place values is not stable. Models do small sums well and drift on long ones; use a calculator tool for anything that must be exact.
Trailing whitespace. In most BPE tokenizers a word and the same word with a leading space are different tokens, so "Hello" and " Hello" are different inputs. A prompt that ends in a space ("Answer: ") forces the model to start from an unnatural position. Strip it unless you mean it.
Glitch tokens. Strings that exist in the vocabulary but barely appear in training can produce erratic output, because their embeddings were never learned well.
Unicode tricks. Visually identical text can tokenize differently (lookalike letters, zero-width characters), which matters for prompt-injection and for filters that match on strings. Normalise Unicode before you filter, and never rely on a keyword filter alone; Red-Teaming and Compliance as Code covers the layers.

Special tokens and chat templates are part of the model. A chat model is trained on conversations rendered into one token stream with reserved markers that mark who is speaking, for example a start-of-turn token, a role name, an end-of-turn token. The exact markers differ per model family and are applied by a template. If you fine-tune with one template and serve with another, the model sees a format it was not tuned on: it may ignore the system prompt, never emit its stop token, or run on past the end of its answer. The check is cheap: render one training example and one serving request through the same template function and diff them token by token.

fine-tune data rendered with template A  ->  serve with template B
symptom: answers run on, stop token never emitted, system prompt ignored

one template function, used by both training and serving
test: render a sample from each path and diff the token ids

If you call a hosted API you do not touch the template, but the lesson carries over to anything self-hosted or fine-tuned, as in LoRA and QLoRA: What Actually Changes During Fine-Tuning.

How to say it in the interview

"A tokenizer is a learned merge table: it starts from bytes or characters and repeatedly glues the most frequent adjacent pair into a new symbol until the vocabulary is full. The model never sees letters or words, only those ids, so everything is metered in tokens: price, context window, latency, and the model's blind spots on spelling and digits. I count with the provider's own tokenizer for the model I will call, because a new tokenizer can change the same text by tens of percent."

Checklist

Ran BPE by hand: pair counts, merge the most frequent, repeat
Explained byte-level fallback and why no input is 'unknown'
Stated the vocabulary trade with numbers: fewer tokens versus bigger V x d matrices
Mentioned under-trained rare tokens as the source of glitch behaviour
Counted tokens with the target model's own counter, never another vendor's tokenizer
Estimated a 10-page PDF with a stated rule (about 1.3 tokens per word) and flagged where it breaks
Explained Indic token inflation as a merge-table property that costs money and context
Moved letter counting and long arithmetic into code or tools
Stripped trailing whitespace and normalised Unicode before filters
Used one chat-template function for training and serving and diffed rendered examples

Practice this interactively