A token is a chunk of text that a language model reads and writes as a single unit, usually a subword piece. For English, one token is roughly 4 characters, so a model sees "tokenization" not as eleven letters but as two or three pieces like "token" and "ization". That gap between how you count words and how the model counts tokens causes two things engineers hit often: surprising API bills and a model that cannot reliably spell or count letters.
Most explanations stop at the definition. This one follows the subword mechanism to its cost consequences: why counting words under-predicts your bill, why code and non-English text cost more, and why a sonnet-writing model still miscounts the letters in "strawberry".
The short answer: a token is a subword piece
A token is the smallest unit a model processes, and it is typically smaller than a word and larger than a single character. OpenAI's own rule of thumb is that one token corresponds to about 4 characters of English text, or roughly three quarters of a word, so 100 tokens land near 75 words. Common words become a single token. Rare or long words get split into several.
A tokenizer turns text into a list of integer IDs. The model never sees letters. It sees numbers that stand for tokens, predicts the next number, and the tokenizer maps it back to text. GPT-4's tokenizer, cl100k_base, has a vocabulary of about 100,000 distinct tokens, each mapped to one integer ID. (The exact count is 100,277.)
Mental model: a token is not a word and not a letter. It is a frequent chunk of characters. Frequent words are one token; uncommon words, names, and most non-English text shatter into several.
Why models split into subwords instead of words
Subword splitting solves the open-vocabulary problem: language has effectively infinite words, but a model needs a fixed, finite vocabulary. The method that won is Byte Pair Encoding (BPE), adapted for neural language models by Rico Sennrich, Barry Haddow, and Alexandra Birch in 2016. Their insight was that rare words can be built from smaller, reusable pieces, so the model can read and produce words it has never seen by composing known subwords.
BPE builds its vocabulary by counting. It starts from individual characters, then repeatedly merges the most frequent adjacent pair into a new token. "e" and "r" appearing together often becomes "er". Do this thousands of times and you get a vocabulary where common sequences are single tokens and rare ones stay split. Modern tokenizers use byte-level BPE, which starts from raw bytes, so any text in any script can be represented without an unknown-word fallback.
- Pure word tokens: a fixed dictionary cannot cover new words, typos, or rare names, so it breaks on real text.
- Pure character tokens: every sentence becomes a very long sequence, which is slow and wastes the model's limited context.
- Subword tokens (BPE): common words stay compact, rare words decompose into reusable pieces, and nothing is ever unrepresentable.
How many tokens is a word?
Count words to estimate cost and the estimate comes in low, every time. APIs bill per token, not per word, and tokens outnumber words because punctuation, spaces, capitalization, and word fragments each consume tokens. A safe back-of-envelope conversion is the OpenAI ratio: about 100 tokens for every 75 words, or roughly 1.3 tokens per English word.
The ratio is not fixed. Numbers, URLs, code, emoji, and unusual formatting push it higher because they fragment into many small tokens. A single emoji can cost several tokens. A long hexadecimal string or a messy JSON blob can cost far more tokens than its word count suggests. The only exact way to know is to run the text through the model's actual tokenizer, such as tiktoken for OpenAI models, rather than guessing from word count.
For budgeting, count tokens with the model's own tokenizer on a representative sample, then multiply. Estimating from word count alone will under-predict, especially for code, structured data, and non-English text.
Why non-English text and code cost more tokens
The contrarian part: paying less is not about writing less. Two prompts of identical word count can differ several-fold in token cost depending on language, code, and formatting. The unit on the invoice is the token, and the token does not track your word count.
The same sentence costs more tokens in some languages than in others, and the difference is large. Tokenizers are trained mostly on English-heavy data, so English chunks into efficient tokens while other languages fragment. A study of OpenAI's commercial API across 22 languages found that some non-Latin-script languages cost several times more tokens than English to express the same content, with Telugu needing up to about 5 times more tokens than English for the same information.
You can see the mechanism in a single word. The Spanish word "manzanas" (apples) does not get one token. A GPT tokenizer splits it into pieces like " man", "zan", and "as", three tokens for one common word, because the English-trained vocabulary lacks a compact token for it. Multiply that across a document and a Spanish prompt can cost meaningfully more tokens than its English translation.
Code pays a similar tax for a different reason. Indentation, brackets, snake_case and camelCase identifiers, and operators fragment into many tokens. Whitespace alone matters: deeply nested code spends tokens on leading spaces on every line. This is why prompts full of source code or structured data eat through a context budget faster than prose of the same visual length.
| Text type | Roughly how it tokenizes | Cost implication |
|---|---|---|
| Common English prose | ~1.3 tokens per word, ~4 chars per token | Cheapest, closest to the 100 tokens to 75 words rule |
| Non-Latin-script languages | Words fragment into many subword pieces | Up to about 5x more tokens than English for the same meaning |
| Source code | Indentation, identifiers, and symbols split heavily | Higher token count than its line count suggests |
| Numbers, IDs, emoji | Long digit strings and emoji split into pieces | Token count far exceeds character intuition |
Why subword tokenization breaks spelling and counting
A model fails at counting letters because it cannot see letters. It sees tokens. The classic example is counting the r's in "strawberry": GPT-4o's tokenizer splits that word into chunks like "st", "raw", and "berry", so the model is reasoning over three opaque pieces, not ten individual characters. It has no built-in view of the letters hidden inside each token, which is why character-level questions trip it up. This is the same root cause behind the well-known question of why AI cannot count the r's in strawberry, broken down token by token in the dedicated strawberry post.
Research on counting confirms it is a tokenization problem, not a reasoning gap. When letters are grouped into multi-character tokens, accuracy on counting tasks collapses toward random as strings get longer. Separate the same letters with spaces or delimiters so each becomes its own token, and accuracy jumps dramatically. The model can count fine when the tokenizer stops hiding the items being counted.
The same blind spot explains spelling mistakes, reversing strings, and some arithmetic slips. Tokens are statistical chunks of frequent text, not meaning-aware units, so any task that depends on the internal structure of a token is working against the representation. It also clarifies why tokenization is a distinct concept from how a model represents meaning: tokens are surface-level pieces, while the deeper semantic representation lives in vector embeddings, a separate layer entirely.
Quick trick: if a model miscounts letters or misspells, insert spaces between characters (s t r a w b e r r y). That forces one token per letter and usually fixes the count.
How token limits and context windows relate to tokens
A context window is measured in tokens, not words or characters. It is the maximum number of tokens a model can hold in view at once, covering both your input and its output. When people say a model has a 128K context window, they mean 128,000 tokens, which is why the same window holds fewer words of dense code or non-English text than of plain English prose.
This is also why a long chat eventually starts forgetting earlier messages: once the running conversation exceeds the token budget, older tokens fall out of the window. The window is a fixed working span, not a memory. A context window is not memory. A bigger window delays the cutoff but never removes it, and reasoning-heavy responses quietly spend extra tokens you may not be counting.
Estimating cost from tokens in practice
To estimate cost, convert text to tokens first, then apply the per-token price for input and output separately. Start from the 100 tokens to 75 words ratio for English, adjust upward for code and non-English text, and remember that providers usually charge a different rate for output tokens than input tokens.
- Count input tokens with the model's real tokenizer (for example tiktoken), not by counting words.
- Estimate output tokens from how long the response should be, since output is often priced higher than input.
- Add the tokens consumed by your system prompt and any retrieved context; they bill on every single call.
- Account for reasoning models that generate hidden intermediate tokens, which add to output cost beyond the visible answer.
Reasoning models catch teams off guard here. Models that think before answering produce internal steps that count as billable output even when you never see them, so the hidden cost of reasoning tokens can dominate the bill on hard prompts. Estimate from total tokens generated, not from the length of the final answer.
How to reduce token usage and cost
For any prompt you send repeatedly, tokens are the unit to optimize. A system prompt that runs on every request is paid for on every request, so trimming it compounds across thousands of calls. The savings come from removing tokens, not from removing words, and those are not the same thing.
- Tighten repeated system prompts and instructions; every retained token bills on each call.
- Prefer English for fixed scaffolding when content allows, since it tokenizes most efficiently.
- Minify or strip pretty-printing from JSON and code you pass in, because whitespace is tokens.
- Stop re-sending unchanged context each turn when a memory layer or retrieval can supply it on demand.
Re-sending the same background every turn is the most common quiet expense. This is where an external memory layer like MemX fits: instead of stuffing prior facts, preferences, and history back into every prompt and paying for those tokens on each call, MemX persists them outside the context window and supplies only what a given request needs. The window stays focused on the current task, and you stop re-billing for context the model already received once.
01What is a token in an LLM?
A token is the smallest unit of text a language model processes, usually a subword piece rather than a full word or single letter. The model converts text into a list of integer token IDs, predicts the next ID, and converts it back to text. In English, one token averages about 4 characters.
02How many tokens is a word?
On average, one English word is about 1.3 tokens, so 100 tokens equal roughly 75 words. Common words are a single token, while rare or long words split into several. Code, numbers, emoji, and non-English text use more tokens per word, sometimes several times more.
03Why does non-English text cost more tokens?
Tokenizers are trained mostly on English data, so English chunks into efficient tokens while other scripts fragment into many pieces. Research on commercial models found some non-Latin-script languages cost several times more tokens than English to say the same thing, which directly raises API cost.
04Why can't LLMs count letters in a word like strawberry?
Because the model sees tokens, not letters. A tokenizer splits strawberry into chunks like st, raw, and berry, so the individual letters are hidden inside opaque tokens. Studies show counting accuracy collapses with grouped tokens but recovers when letters are separated by spaces.
05Is a context window measured in tokens or words?
A context window is measured in tokens. A 128K window means 128,000 tokens of combined input and output, not words. Because dense code and non-English text use more tokens per word, the same window holds fewer words of that content than of plain English prose.
