Tokenizer
What is a tokenizer?
A tokenizer is the component that converts raw text into the sequence of tokens a language model actually reads. Most modern models use a sub-word scheme such as byte-pair encoding (BPE), which breaks text into common chunks — frequent words stay whole, rare words split into pieces. The reverse step turns the model's token outputs back into text.
Tokenizers are model-specific. The same sentence can become a different number of tokens depending on which tokenizer is used, which is why token counts aren't directly comparable across providers.
Why tokenizers matter in 2026
They matter because code tokenizes less efficiently than prose, and that directly affects an AI coding agent's cost. Identifiers, punctuation, whitespace and rare symbols common in source code often split into more tokens than ordinary English, so a file of code can cost more tokens than its character count suggests. Knowing this is why measuring real token usage (with the right tokenizer) beats eyeballing character counts when you're trying to cut your bill.
Tokenizers also change between model generations of the same provider, which is the 2026 lesson. Anthropic's pricing docs state that Claude 4.7 and later models (so today's Opus 5.5, Sonnet 5.5 and Fable 5.1) use a newer tokenizer that produces approximately 30% more tokens for the same text, while Sonnet 4.6 and earlier use the previous one; the exact increase depends on the content. A per-token price that looks flat across an upgrade can still mean a bigger bill, because the count went up. Re-measure your token budgets whenever you switch model generations.
When the tokenizer detail doesn't matter
- For rough estimates — the ~4-characters-per-token rule of thumb is fine when you just need a ballpark.
- Within a single model — if you stay on one model version, the absolute tokenizer differences are constant and you can reason in relative terms. Switching model generations, even within one provider, resets that baseline.