Tokenization

Definition: Tokenization is the process of converting raw text into a sequence of discrete units — tokens — that a machine learning model can consume as numerical input. A token can be a whole word, a piece of a word (a subword), a single character, or a raw byte, depending on the scheme in use.

Modern large language models almost universally rely on subword tokenization, which balances the small-but-brittle vocabularies of character-level schemes against the huge-but-sparse vocabularies of word-level schemes. Every downstream computation in a language model — embedding lookup, attention, generation, loss calculation — operates on integer token IDs, never on the text itself.

How It Works

Why Raw Text Can’t Feed a Neural Network

Neural networks are matrix-multiplication machines. They accept fixed-dimensional numeric vectors, not strings, so there has to be a deterministic mapping from text to numbers before anything resembling a Transformer Architecture can process it.

The naive options both fail in practice. Treat every unique word as a token, and English alone needs a vocabulary in the hundreds of thousands of entries once you count inflections, misspellings, names, and numbers — and any word not seen during vocabulary construction becomes an unrepresentable out-of-vocabulary (OOV) token, silently destroying information.

Go the other direction and tokenize by individual character, and the vocabulary shrinks to a few hundred symbols, but every sentence balloons into a long sequence, which is expensive: attention cost in a Transformer scales quadratically with sequence length, so a sequence 4x longer can mean roughly 16x more compute for self-attention.

Subword tokenization is the compromise that shipped in essentially every production LLM: a moderate vocabulary (typically 32,000-200,000 entries) built so common words stay as single tokens while rare or unseen words decompose into a handful of familiar pieces.

A Brief History

Tokenization wasn’t always this standardized. Early neural language models (pre-2015) mostly used word-level vocabularies with a hard cutoff — the top N most frequent words, everything else mapped to a single [UNK] token — which capped what a model could ever say or understand about rare words, names, or morphological variants.

Byte-Pair Encoding itself is older than its NLP application: it originated in 1994 as a general-purpose data compression algorithm. Sennrich, Haddow, and Birch repurposed it for neural machine translation in 2016, showing it solved the rare-word problem far more gracefully than word-level vocabularies or hand-built morphological rules.

From there, adoption moved fast: WordPiece had already been used internally at Google for speech recognition and later powered BERT (2018); GPT-2 (2019) popularized byte-level BPE specifically to guarantee zero OOV without a language-specific preprocessing step; and SentencePiece (2018) generalized the whole approach to be language-agnostic, removing the assumption that whitespace marks word boundaries. Every major LLM tokenizer since traces its lineage to one of these three lines of work.

The BPE Algorithm, Step by Step

Byte-Pair Encoding (BPE) is the dominant construction method, used — with variations — by GPT-family models, RoBERTa, and many others.

It is a greedy, frequency-driven merge algorithm run once, offline, over a large training corpus, well before the model itself ever sees a single training example.

The construction procedure:

Start with a base vocabulary of individual characters, or raw bytes in byte-level BPE — this guarantees zero OOV, since any Unicode string decomposes into a sequence of the 256 possible byte values.
Represent every word in the training corpus as a sequence of these base symbols.
Count every adjacent symbol pair across the whole corpus, and find the single most frequent pair.
Merge that pair into one new symbol, and add it to the vocabulary.
Repeat steps 3-4, re-counting after every merge, until the vocabulary reaches a target size $V$ chosen in advance — a hyperparameter of training, not something derived automatically from the data.

The result is a ranked list of merge rules, typically tens of thousands of them, stored alongside the vocabulary itself.

At inference time, the tokenizer applies the same merges, in the same learned order, to unseen text — greedily combining the highest-priority pair available at each step.

This is why BPE splits are described as “learned” rather than rule-based: nobody hand-writes “believ- is a valid prefix.” The corpus statistics discover it because “believ” recurs across “believe,” “believed,” “believable,” and “unbelievable,” so merging those characters together earns a high enough frequency rank to make the cut.

Tokenizer Variants: WordPiece and Unigram/SentencePiece

BPE is not the only construction algorithm, and the differences matter for anyone fine-tuning or comparing models across families:

WordPiece (used by BERT and its descendants) is structurally similar to BPE but scores candidate merges by a likelihood ratio — how much merging a pair increases the probability of the training corpus under a unigram language model — rather than raw frequency. In practice this tends to favor linguistically coherent subwords slightly more than pure frequency counting.
Unigram Language Model tokenization (popularized by Google’s SentencePiece library, used by T5, ALBERT, and many multilingual models) works backward from BPE: it starts with a large superset of candidate subwords, assigns each a probability, and iteratively removes the least useful ones until the vocabulary reaches the target size.
Because Unigram scores whole segmentations probabilistically rather than applying one fixed merge order, it can support multiple valid tokenizations of the same string, each with a defined probability — useful for subword regularization during training, where sampling alternate valid splits acts as a form of data augmentation.
SentencePiece is technically a tokenizer framework, not an algorithm — it can implement either BPE or Unigram internally, and its key innovation is treating the input as a raw stream of Unicode without assuming whitespace marks word boundaries. That makes it language-agnostic: it works the same way on space-delimited English and on Japanese or Thai, which don’t use spaces between words at all.

Algorithm	Merge/Prune Criterion	Segmentation	Notable Users
BPE	Highest adjacent-pair frequency	Single fixed merge order	GPT-family, RoBERTa
WordPiece	Highest likelihood-ratio score	Single fixed merge order	BERT, DistilBERT
Unigram	Lowest loss when symbol removed	Probabilistic, multiple valid splits	T5, ALBERT, SentencePiece-based models

Byte-Level Fallback and Unicode Coverage

A critical implementation detail: because text can contain emoji, rare CJK characters, or arbitrary Unicode nobody anticipated at training time, a robust tokenizer needs a guaranteed fallback that never fails.

Byte-level BPE solves this by operating on UTF-8 bytes rather than Unicode codepoints as the base alphabet.

Since every possible string decomposes into some sequence of the 256 byte values, there is no such thing as an unrepresentable input — worst case, an unfamiliar character falls back to being spelled out byte by byte, consuming more tokens but never failing outright.

This is a meaningful improvement over older word-level and even some subword schemes, which reserved a single [UNK] token for anything unrecognized — a lossy operation that destroys the original text and can’t be reversed during detokenization.

Vocabulary Size: The Central Trade-off

Vocabulary size $V$ is chosen at training time and never changes afterward without retraining the tokenizer — and by extension, usually the model too, since the embedding and output layers are indexed by vocabulary size.

A larger $V$ means shorter sequences for the same text (each token, on average, covers more characters), which reduces attention compute and lets more content fit in a fixed context window.

But it also means a larger embedding matrix and a larger final softmax layer, since both scale directly with $V$ : parameter count in those layers grows roughly as $V \times d$ , where $d$ is the model’s hidden dimension.

Production LLMs settle on values across a wide range: BERT used roughly 30,000, GPT-2 used about 50,000, and many contemporary multilingual models push past 100,000-250,000 to keep non-English languages from fragmenting into excessively long token sequences.

Tokenization Efficiency: A Quick Formula

A useful back-of-envelope metric is the compression ratio — characters per token — which measures how efficiently a tokenizer represents a given text:

$\text{compression ratio} = \frac{\text{characters in text}}{\text{tokens produced}}$

A well-matched English tokenizer typically achieves a ratio around 4 (roughly 4 characters per token). A poorly matched tokenizer applied to an unfamiliar language or domain can drop toward 1-2, meaning tokens barely compress the input at all — each token covers only a character or two, which is the numeric signature of the “token tax” described later in this article.

Common Tokenizer Implementations

In practice, almost nobody hand-writes a BPE trainer from scratch. A handful of libraries dominate production usage, and knowing them helps when debugging real token counts rather than illustrative ones:

tiktoken — OpenAI’s fast BPE implementation, widely used to count tokens client-side before sending API requests to GPT-family models.
Hugging Face tokenizers — a Rust-backed library supporting BPE, WordPiece, and Unigram, bundled with nearly every model on the Hugging Face Hub alongside its trained vocabulary files.
Google’s sentencepiece — the reference implementation for SentencePiece-style Unigram and BPE tokenization, used by T5, ALBERT, and many multilingual and on-device models.
Anthropic’s tokenizer — a byte-level BPE tokenizer used across the Claude model family, exposed via the API and SDKs so developers can count tokens before sending a request.

Each ships as a portable artifact — typically a vocabulary file plus a merge-rules or model file — that must travel with the model weights; swapping one without the other silently breaks tokenization.

Special Tokens and Vocabulary Construction

A production tokenizer’s vocabulary is not just merged subwords — it reserves IDs for special tokens that carry structural meaning rather than linguistic content.

Common examples include [BOS]/[EOS] (beginning/end of sequence), [PAD] (padding shorter sequences to a uniform batch length), [UNK] (a fallback for anything unrepresentable, rare in byte-level schemes), and [CLS]/[SEP] (classification and segment markers, common in BERT-style encoders).

These aren’t cosmetic bookkeeping — a model is trained to treat [EOS] as a hard signal to stop generating, and swapping or misplacing special tokens at inference time is a common, hard-to-diagnose cause of a model that won’t stop generating or that ignores conversation structure.

Increasingly, vocabularies also reserve tool-call and chat-role markers like <|im_start|> or <|assistant|> that let a single token sequence encode a multi-turn conversation with system, user, and assistant roles baked in structurally, rather than described in prose the model has to interpret freshly every time.

The Full Pipeline

The embedding-matrix step at the end is the handoff into Embeddings — everything downstream of it operates on dense vectors, never on text.

Nothing about this pipeline is unique to language generation — the same integer-ID-then-embedding-lookup pattern underlies how any categorical input becomes learnable, from user IDs in recommender systems to amino acids in protein language models.

Why It Matters

Cost is metered in tokens, not words or characters. Every commercial LLM API — OpenAI, Anthropic, Google — bills per input and output token, so tokenization efficiency directly determines the dollar cost of a prompt or completion.
Context windows are token-denominated. A “128K context window” means 128,000 tokens, not 128,000 words; because English averages roughly 0.75 words per token, that translates to something like 96,000 words of actual text — a number many users overestimate.
It explains a well-known LLM failure mode. Models notoriously struggle to count letters in a word (the classic “how many r’s in strawberry” failure) because they never see individual characters — they see whole subword tokens and have to infer spelling indirectly from training data patterns.
Vocabulary size trades off against model size and speed. The final embedding and output projection layers scale with vocabulary size $V$ — a bigger vocabulary means more parameters in the input/output layers and a more expensive softmax over the vocabulary at every generation step.
It determines effective sequence length. The same 500-word document might tokenize to 650 tokens under an efficient BPE vocabulary or 900+ under a poorly matched one, directly affecting whether it fits a given context budget.
Non-English languages often pay a “token tax.” Languages with less training-corpus representation get less efficient merges, so the same sentence can cost 2-5x more tokens in, say, Burmese or Amharic than in English — a real, measurable inequity in API pricing and effective context length.
It’s a training-inference consistency requirement. A model’s tokenizer is fixed at pretraining time and must be used identically at inference; feeding a model text tokenized with the wrong vocabulary silently produces garbage, since token ID 3255 means something completely different across two different vocabularies.
Code, math, and structured data tokenize unevenly. Whitespace-heavy indentation in Python, repeated symbols, and numeric sequences can tokenize inefficiently under text-oriented BPE vocabularies, which is why some coding-focused models train dedicated tokenizers over code corpora.
Retrieval and search pipelines depend on it upstream of embeddings. Any system built on Retrieval-Augmented Generation (RAG) or Cosine Similarity search first tokenizes documents before they’re embedded, so chunking strategies — how many tokens per chunk — are a direct design decision downstream of tokenization behavior.
It shapes what “fine-tuning” even means. Fine-Tuning a model on a specialized domain (legal, medical, chemical formulas) without adjusting the tokenizer means the model is still forced to spell out domain terms in inefficient subword fragments, wasting context and hurting learning efficiency.

Worked Example: Tokenizing a Sentence, Step by Step

Take the sentence: “Tokenizers handle unbelievably complex vocabulary efficiently.”

First, focus on one word — “unbelievably” — and trace an illustrative (approximate, not from a real trained vocabulary) sequence of BPE merges from raw characters down to final subwords:

Round 0 (raw chars):    u n b e l i e v a b l y
Round 1 (merge "l+y"):  u n b e l i e v a b [ly]
Round 2 (merge "a+b"):  u n b e l i e v [ab] [ly]
Round 3 (merge "ab+ly"):u n b e l i e v [ably]
Round 4 (merge "e+v"):  u n b e l i [ev] [ably]
Round 5 (merge "i+ev"): u n b e l [iev] [ably]
Round 6 (merge "l+iev"):u n b e [liev] [ably]
Round 7 (merge "e+liev"):u n b [eliev] [ably]
Round 8 (merge "b+eliev"):u n [believ] [ably]
Round 9 (merge "u+n"):  [un] [believ] [ably]

Nine merge rounds later, “unbelievably” — one word — becomes three tokens: un, believ, ably. Note what happened: the algorithm never “knew” English morphology going in; it simply merged whatever adjacent pairs recurred most often across the training corpus, and those merges happened to converge on linguistically sensible units because prefixes and suffixes really do recur across many words.

Now tokenize the full sentence and map each resulting piece to an integer ID (illustrative IDs, not from any real model’s vocabulary — the point is the shape of the mapping):

Token	Token ID	Note
`Token`	3255	common enough to be a standalone token
`izers`	5867	suffix, reused across “tokenizers,” “organizers,” etc.
`handle`	4820	leading space is part of the token in byte-level BPE
`un`	8241	prefix, split point begins
`believ`	9102	stem shared across believe/believed/believable
`ably`	7734	suffix, closes out the word
`complex`	2004	common enough to stay whole
`vocabulary`	6611	domain-relevant word, frequent in NLP corpora, stays whole
`efficiently`	5590	common adverb, often kept whole
`.`	13	punctuation gets its own token

That’s 10 tokens for a 9-word sentence — roughly 1.1 tokens per word, typical for English under a well-trained BPE vocabulary. Note the leading-space convention: byte-level BPE (as used by GPT-family tokenizers) treats the space before a word as part of the following token, so "handle" at the start of a sentence and " handle" mid-sentence are different tokens entirely. This is a frequent source of confusion when debugging token counts or writing prompts that manipulate whitespace precisely.

Detokenization runs this whole process in reverse: the ID sequence [3255, 5867, 4820, 8241, 9102, 7734, 2004, 6611, 5590, 13] maps back through the vocabulary table to strings, and those strings concatenate directly (leading spaces and all) to reconstruct the exact original text, byte for byte. That exact reversibility is a hard requirement — a tokenizer that can’t losslessly detokenize its own output is unusable in production, since it would corrupt the model’s generated text on the way back out.

Tokenization, Language, and Fairness

Tokenizer behavior is not a neutral technical detail — it has direct, measurable consequences that extend past raw compute cost.

Vocabulary construction runs once, over one training corpus, and that corpus is overwhelmingly English- and Western-language-dominated for most widely deployed tokenizers.

The consequence: a sentence in a low-resource language can require several times more tokens than an equivalent English sentence, because the tokenizer never learned efficient merges for that language’s character combinations and instead falls back to near character-level splitting.

Language	Illustrative Sentence Meaning	Approx. Tokens (Common LLM Tokenizer)
English	“Thank you for your help.”	~6 tokens
Spanish	Same meaning	~8 tokens
Hindi	Same meaning	~14 tokens
Arabic	Same meaning	~16 tokens
Burmese	Same meaning	~22 tokens
Amharic	Same meaning	~24 tokens

The pattern in that table repeats across most production tokenizers: the less a language was represented in the tokenizer’s training corpus, the more tokens the same meaning costs. This has two compounding effects. First, cost: users paying per-token for API access to the same model pay meaningfully more for the same amount of communicated meaning if they write in an underrepresented language. Second, context budget: a 128K-token context window holds proportionally less content in an inflated-token-count language, meaning those users get a worse effective context window for the identical nominal limit.

Researchers studying AI Bias and Fairness increasingly treat tokenizer parity — measuring tokens-per-sentence across languages for a fixed meaning — as a first-class fairness metric, not an afterthought to be handled purely at the model-weights level.

There is also a numeric-representation dimension worth knowing. Naive tokenizers used to split multi-digit numbers inconsistently — sometimes “380” as one token, sometimes as “3”+“80” or “38”+“0” depending on corpus frequency — which measurably hurt arithmetic performance because the model saw “380” represented differently depending on surrounding context.

Several modern tokenizers now enforce consistent digit-level or fixed-chunk splitting for numbers specifically to reduce this inconsistency — a small design choice with an outsized effect on downstream math capability.

Tokenizer Training Data and Domain Drift

A tokenizer inherits blind spots from whatever corpus it was trained on, and that has consequences well beyond human-language fairness.

A vocabulary built primarily from general web text handles conversational English efficiently but fragments unfamiliar domains badly.

A chemical formula like a long IUPAC name, a legal citation string, or a block of Python with heavy indentation and symbol repetition all decompose into far more tokens per character than ordinary prose, because none of those patterns were common enough in the training corpus to earn efficient merges.

This is precisely why specialized models often train dedicated tokenizers: code-focused models train BPE over source-code corpora so that common code constructs — indentation levels, bracket pairs, camelCase and snake_case boundaries — become efficient single or few-token units instead of fragmenting character by character.

The same logic applies to models targeting biomedical text, legal text, or non-Latin scripts: a domain-matched tokenizer isn’t a cosmetic optimization, it changes how much of the context window is spent on structure versus content, and how efficiently the model can learn patterns in that domain during pretraining.

The practical implication for anyone deploying a model outside its original training distribution: before assuming a performance gap is a model-capability problem, it’s worth checking whether the tokenizer itself is inflating token counts and starving the model of effective context on that specific input distribution.

This is also why some teams building domain-specific products choose to extend an existing vocabulary with a small set of domain tokens rather than retraining a tokenizer from scratch — a cheaper middle ground that recovers much of the efficiency gain without the cost of a full retrain of the embedding and output layers.

Tokenization Beyond Text

Text is where tokenization started, but the same core idea — mapping raw, high-dimensional input into a bounded vocabulary of discrete units a Transformer can attend over — now extends well past language.

Vision Transformers tokenize images by patches. An image is sliced into a grid of fixed-size squares (commonly 16x16 pixels), each patch flattened and linearly projected into a vector that plays the same architectural role a word token plays in text — this is how Computer Vision models feed pixel data through the same attention machinery built for language.

  224x224 image
  +----+----+----+----+----+----+----+
  | P1 | P2 | P3 | P4 | P5 | P6 | P7 |
  +----+----+----+----+----+----+----+
  | P8 | P9 |P10 |P11 |P12 |P13 |P14 |
  +----+----+----+----+----+----+----+
  |          ... more patch rows ...  |
  +----+----+----+----+----+----+----+
        |  each patch flattened + projected
        v
  [ vec1, vec2, vec3, ... vecN ]  -->  fed into Transformer, exactly
                                        like a sequence of word tokens

Audio models tokenize sound into discrete codes. Neural audio codecs learn a fixed codebook of sound-fragment “tokens,” turning a continuous waveform into a discrete sequence a Transformer can generate autoregressively, the same way it generates text tokens one at a time.
Vision-language and video models mix modalities in one sequence. A single input to a multimodal model can interleave text tokens, image-patch tokens, and even video-frame tokens in one sequence, letting one attention mechanism reason jointly across modalities that used to require entirely separate architectures.
The tradeoffs carry over directly. More patches or audio codes per second means longer sequences and quadratically more attention cost, exactly mirroring the vocabulary-size-versus-sequence-length tradeoff at the heart of text tokenization — the specific unit changes, but the underlying compute economics don’t.

This convergence is a large part of why “tokenization” as a concept generalized from an NLP preprocessing detail into a foundational idea across modern AI: any modality can plug into a Transformer-based architecture once someone designs a sensible tokenizer for it.

It also means the pitfalls generalize. A poorly designed image-patch size or audio codec granularity produces the same symptoms as a poorly matched text vocabulary — bloated sequence length, wasted compute, and degraded downstream quality — just measured in patches or audio frames instead of subword tokens.

Comparison

Scheme	Vocabulary Size	Handles Unseen Words?	Sequence Length	Typical Users
Word-level	Very large (100K-1M+)	No — hard OOV failures	Shortest	Legacy NLP systems, pre-2018
Character-level	Tiny (~100-300)	Yes, trivially	Longest (4-6x word-level)	Some character-aware CNN/RNN models, niche use
Subword / BPE	Moderate (32K-100K)	Yes — decomposes into known pieces	Moderate	GPT-family, RoBERTa, most modern LLMs
SentencePiece / Unigram	Moderate (32K-250K)	Yes — probabilistic decomposition	Moderate	T5, ALBERT, many multilingual models

The practical takeaway: subword schemes (BPE and Unigram/SentencePiece) won the field because they’re the only options that simultaneously keep vocabulary size bounded, guarantee no unrepresentable input, and keep sequence lengths short enough for attention’s quadratic cost to stay tractable.

When Each Scheme Still Shows Up

Despite subword tokenization’s dominance, the alternatives aren’t purely historical:

Word-level tokenization persists in some classic NLP pipelines (bag-of-words models, older sentiment classifiers) where interpretability of each token as a whole word matters more than vocabulary coverage.
Character-level tokenization resurfaces in tasks specifically about spelling or character manipulation — spell-checkers, some OCR post-processing, and research probing whether models can be taught character-level reasoning despite subword training.
BPE remains the default choice for new general-purpose LLMs trained primarily on English and code, where its simplicity and fast, deterministic segmentation are hard to beat.
SentencePiece/Unigram is preferred whenever multilingual coverage or non-whitespace-delimited languages are a first-class requirement, since it doesn’t assume any language-specific word-boundary convention.

Real-World Use Cases

API pricing and billing systems for every major LLM provider (OpenAI, Anthropic, Google, Cohere) meter usage in tokens, with published per-1K or per-1M token rates for input and output separately.
Prompt-length validators in developer tools and IDE plugins count tokens client-side (using libraries like tiktoken) before sending a request, to warn developers when a prompt will exceed a model’s context window.
Chunking logic in Retrieval-Augmented Generation (RAG) pipelines splits source documents into token-bounded chunks (e.g., 512 or 1,024 tokens per chunk) before embedding, since embedding models have fixed maximum input token lengths.
Code-completion tools (GitHub Copilot-style products) often use tokenizers trained specifically on source code corpora, since generic text tokenizers split code punctuation and indentation inefficiently.
Multilingual translation systems rely on SentencePiece-style tokenizers precisely because they don’t assume whitespace-delimited word boundaries, letting a single vocabulary serve dozens of typologically different languages.
Content moderation and filtering systems sometimes exploit or defend against tokenization quirks, since adversarial inputs can be crafted to split flagged words into token sequences that evade naive keyword-matching filters.
Search and autocomplete engines use subword tokenization to match partial words and handle typos gracefully, since a misspelled word still decomposes into mostly-familiar subword pieces rather than becoming a total OOV failure.
Speech-to-text and text-to-speech pipelines frequently use tokenization schemes tuned for phoneme- or byte-level representation, an adjacent application of the same merge-based vocabulary-construction idea outside pure text.
Cost-optimization middleware in enterprise LLM deployments actively rewrites prompts to reduce token counts (stripping redundant whitespace, abbreviating boilerplate instructions) before sending requests, since token count is the direct cost lever.
Model migration tooling has to account for the fact that swapping from one model family to another (say, from a GPT-family model to a Llama-family model) often means swapping the entire tokenizer, invalidating any token-ID-based caching or fine-tuned embeddings tied to the old vocabulary.

Common Pitfalls

Assuming 1 token equals 1 word. In English it’s closer to 0.75 words per token on average; the ratio gets worse (more tokens per word) for morphologically rich or non-Latin-script languages.
Ignoring the leading-space convention. In byte-level BPE, "cat" and " cat" are different tokens with different IDs — string manipulation that adds or strips leading whitespace can silently change tokenization and, in edge cases, model behavior.
Assuming token boundaries align with meaningful linguistic units. A model doesn’t “know” that believ is a stem in any explicit sense — it only knows the statistical co-occurrence patterns of that token ID, which is why models can struggle with tasks that require explicit character- or morpheme-level reasoning.
Forgetting that unusual words fragment badly. Rare proper nouns, brand names, and technical jargon not well-represented in the training corpus can explode into many small, awkward tokens, consuming disproportionate context budget and sometimes degrading generation quality.
Mixing tokenizers across a pipeline. Using one tokenizer to prepare training data and a different (even slightly different) one at inference silently corrupts the input — token ID 500 in vocabulary A might mean something entirely unrelated in vocabulary B.
Miscounting tokens for cost estimation. Manually approximating token counts (e.g., “characters divided by 4”) is unreliable enough to cause real budget surprises; production systems should use the actual tokenizer library for the target model.
Treating context window limits as word or character limits. Users regularly overestimate how much text fits in a stated context window because they mentally convert tokens to words at the wrong ratio, especially for non-English text.
Overlooking digit and number tokenization inconsistency. Numbers can tokenize inconsistently across contexts in some vocabularies, quietly hurting a model’s arithmetic reliability — a subtle root cause behind some “the model can’t do basic math” complaints.
Assuming special tokens are invisible or free. Chat templates, system prompts, and role markers ([BOS], <|im_start|>, etc.) consume real tokens and real context budget even though they carry no user-visible content.
Fine-tuning without considering vocabulary fit. Adapting a general-purpose model to a specialized domain (legal, medical, chemistry) without evaluating whether its tokenizer handles domain vocabulary efficiently can leave significant, avoidable performance on the table.

Example

A startup builds a customer-support chatbot on top of a general-purpose LLM API and gets billed far more than expected in its first month.

Investigating, the team discovers their system prompt — a long block of formatting instructions and few-shot examples — is being sent on every single request, and because much of it is boilerplate written with heavy indentation and repeated punctuation, it tokenizes far less efficiently than expected: what they assumed was “about 200 words, maybe 250 tokens” turns out to be 480 tokens once run through the actual tokenizer. Multiplied across tens of thousands of daily requests, that gap alone accounts for a meaningful fraction of their bill.

Digging further, they notice a second problem: a large share of their support tickets come from users writing in Vietnamese and Thai, and those messages consistently tokenize at 2-3x the token count of English messages carrying equivalent meaning, because the underlying model’s tokenizer was trained on a predominantly English and Western-European corpus. Non-English users are quietly consuming more of the context window and costing more per ticket, purely as a side effect of tokenizer construction rather than anything about the conversations themselves.

The fixes are entirely at the tokenization layer, not the model layer. They trim the system prompt to remove redundant formatting that inflated token count without adding information, switch to a model whose tokenizer was trained on a more balanced multilingual corpus for their non-English traffic, and add a client-side token counter (using the model provider’s official tokenizer library) to their request pipeline so cost estimates before shipping a feature match actual billed usage.

None of this touched model weights, prompts’ semantic content, or the chatbot’s underlying logic — it was purely about understanding what the tokenizer was doing to their text before it ever reached the model.

Six months later, the same team runs into a related issue when they add a code-generation feature: users pasting Python snippets into the chatbot notice unusually high latency and cost on requests involving deeply nested code. Profiling the tokenizer output shows why — their general-purpose tokenizer splits repeated indentation into many separate whitespace tokens rather than compressing it efficiently, since the training corpus that built the vocabulary was mostly prose, not source code.

Switching the code-handling path to a model with a code-aware tokenizer resolves the latency and cost spike without any change to the feature’s logic. That reinforces the same lesson from a different angle: token-level inefficiency is invisible until someone actually inspects the tokenizer’s output, and it can masquerade as a model or infrastructure problem when the real cause sits one layer earlier, in how the text was chunked into tokens in the first place.

Referenced by

Dig deeper

Spot an error in this note?