Tokenizers and why your word count lies

You paste a document into a prompt, your editor says 2,000 words, and you figure that’s a safe fit under a 128k context window with room to spare. Then the API throws a context length error, or your bill for that one call is higher than you expected. Word count didn’t lie to you exactly. It just answered a question nobody asked. The model doesn’t read words. It reads tokens, and the exchange rate between the two is not fixed.

Your word processor and your API bill disagree

Word count is a human convenience. It splits on whitespace and calls it a day. Tokenizers don’t care about whitespace as a boundary, they care about statistically common chunks of characters. That’s a different unit of measurement entirely, and every LLM provider bills, and limits context windows, in tokens, not words.

This gap is invisible until it costs you something: a truncated RAG chunk, a prompt that silently gets cut off mid-instruction, or an invoice that doesn’t match your mental model of “I sent roughly 2,000 words.” If you build anything on top of an LLM API, you need to know what a tokenizer is actually doing to your text, not just that “tokens exist.”

What a tokenizer actually does

Modern LLMs almost all use some flavor of subword tokenization, most commonly byte pair encoding (BPE) or a close variant. The idea is simple: instead of a vocabulary of whole words (which would need to be enormous to cover every word, misspelling, and language) or individual characters (which would make sequences absurdly long), you build a vocabulary of frequently occurring character chunks.

The tokenizer is trained on a large corpus. It starts with individual characters and repeatedly merges the most frequent adjacent pair into a new token, building up a vocabulary of typically 50,000 to 200,000 entries depending on the model family. Common whole words like “the” or “and” end up as single tokens because they appear constantly. Rarer words get split into pieces. “Tokenization” itself might become “token” + “ization.” A word the model has genuinely never seen intact, like a made-up brand name or a typo, gets chopped down to individual characters or byte fragments if nothing else matches.

This is why token count and word count diverge in ways that aren’t predictable from word count alone. A sentence full of common short words tokenizes efficiently. A sentence full of rare technical terms, unusual proper nouns, or non-English text does not.

Why English text looks efficient and other things don’t

For plain English prose, the commonly cited rule of thumb (OpenAI publishes this for their own tokenizers) is roughly 4 characters per token, which works out to around three tokens for every four words. That ratio holds reasonably well for ordinary sentences because so much of English maps onto whole-word or near-whole-word tokens in these vocabularies.

That ratio falls apart fast outside of plain English. Tokenizer vocabularies are built from training corpora that skew heavily toward English and a handful of other high-resource languages. Text in languages with different scripts, or languages that weren’t well represented in the tokenizer’s training data, gets split much more finely, sometimes down to near-byte-level fragments. The practical effect: the same sentence, translated, can cost meaningfully more tokens in one language than another, even though a human word count would look similar. If you’re building a product for a non-English market, this isn’t a footnote, it directly changes your context budget and your per-request cost.

Numbers and punctuation-heavy text behave oddly too. Long digit strings, unusual formatting, and dense symbols often tokenize less efficiently than plain prose, because they don’t match the common merge patterns the tokenizer learned.

Code is its own animal

If you’re feeding source code into a model, whether for a coding assistant or a RAG pipeline over a codebase, word count is close to meaningless as an estimate. Code has different statistical structure than prose: lots of repeated symbols, camelCase and snake_case identifiers, indentation, brackets. Tokenizers built primarily on natural-language corpora don’t always split code efficiently, which is part of why some model providers have shipped tokenizers specifically tuned to also perform well on code. The practical upshot for you as a builder is the same either way: don’t reuse your prose token estimate for code. Measure it separately.

Why this matters for context windows

Context window sizes are always specified in tokens. When you’re deciding how much conversation history, how many retrieved documents, or how much of a codebase you can stuff into a single call, you need to reason in tokens from the start, not convert from a word count after the fact and hope the multiplier holds. The 4-characters-per-token approximation is fine for a rough sanity check on plain English, but it is not accurate enough to build a hard limit against. Push it too close to the edge and you’ll get truncation, and truncation inside a prompt is often silent: the model just doesn’t see the part that got cut, and it won’t tell you that happened.

This gets worse with system prompts, tool definitions, and few-shot examples stacked on top of user content, since all of that eats into the same token budget. A prompt template that looked fine at 500 words during testing can behave differently once it’s wrapped in a longer system prompt or combined with a large retrieved document.

Why this matters for RAG chunking

Anyone building a retrieval pipeline runs into this directly. Chunk size is a token decision dressed up as a word decision. If you chunk documents by word count or by character count without checking actual token count, your chunks will vary in token size depending on the content: a chunk of dense jargon or a chunk in a different language can blow past what you budgeted, while a chunk of simple prose comes in well under. That inconsistency matters because you’re usually packing multiple chunks into a fixed context budget alongside the query and the system prompt. Chunking by actual token count, using the same tokenizer the target model uses, gives you predictable packing. Chunking by word or character count gives you a guess that’s right some of the time.

How to actually count tokens

The only reliable way to know your token count is to run the actual tokenizer for the model you’re calling, not a generic estimate. OpenAI’s tokenizers are available through the tiktoken library and let you count tokens for a given encoding before you ever send a request. Anthropic and other providers expose their own tokenization behavior as well, and it’s worth checking the current documentation for whichever model you’re targeting, since encodings do change between model generations. Don’t assume last year’s encoding still applies to this year’s model, and don’t assume two different providers’ tokenizers agree with each other on the same input.

If you’re building anything that touches billing estimates, context budgeting, or chunk sizing, the token counter should be a real dependency in your pipeline, not a mental shortcut applied once during development and never revisited.

The takeaway

Word count is a fine metric for humans skimming a document. It is not the unit your model bills in, limits in, or truncates in. Tokenizers split text based on learned statistical patterns, not whitespace, which means the same word count can represent very different token counts depending on language, vocabulary, and whether you’re looking at prose or code. If you’re shipping anything on top of an LLM API, measure tokens with the actual tokenizer for the model you’re using. The 4-characters-per-token rule of thumb is fine for a gut check. It is not fine as the number your context window math or your cost projections actually rely on.

Want more breakdowns like this on the tools and infrastructure behind real LLM apps? Head back to the AI Tool Gazette homepage for the rest of our explainers.