Tokens (in Large Language Models)

What a Token Is

A token is the smallest unit of text that a large language model (LLM) actually processes. Although humans read sentences as words, an LLM never sees raw words or letters directly. Before any text reaches the model, a component called the tokenizer converts it into a sequence of tokens, and each token is then mapped to an integer ID that the neural network can compute on.

A single token is not the same as a single word. Depending on the text, a token can be a whole common word (such as "the"), a fragment of a longer or rarer word (such as "tokenization" split into "token" and "ization"), a single character, a space, or a punctuation mark. Numbers, emoji, and code symbols are also broken into tokens. Everything an LLM reads as input and everything it produces as output is measured in these units.

Because tokens are the model's true unit of currency, almost every practical concern about LLMs, including context limits, pricing, latency, and memory, is expressed in tokens rather than words or characters.

• A token is the atomic unit an LLM reads and generates, not a word or a letter.
• Each token corresponds to an integer ID drawn from the model's fixed vocabulary.
• Tokens can represent whole words, subword fragments, characters, spaces, or punctuation.

Tokens vs Words vs Characters: The Rough Conversion

For English text, OpenAI publishes a widely cited rule of thumb: 1 token is roughly 4 characters, 1 token is about three quarters of a word, and 100 tokens equal approximately 75 words. Put differently, a typical English word averages a little over one token, so a 1,000-word document lands near 1,300 to 1,500 tokens.

These ratios are estimates, not guarantees. The exact count depends on the specific tokenizer and the content itself. Common words often map to a single token, while long, technical, or invented words are split into several subword pieces. Whitespace, capitalization, and punctuation all influence the result, so the only reliable way to know an exact count is to run the text through the target model's tokenizer.

• English rule of thumb: about 4 characters per token and roughly 0.75 words per token.
• 100 tokens is approximately 75 words; a short paragraph is often 60 to 120 tokens.
• Exact counts vary by tokenizer; common words are one token, rare words split into several.

How Tokens Are Produced: Tokenization and Subword Units

Tokens are created by a process called tokenization. Modern LLMs use subword tokenization, which sits between splitting text into whole words and splitting it into single characters. Subword methods break text into frequent, data-driven fragments, which keeps the vocabulary at a manageable size while still handling rare or previously unseen words by composing them from smaller pieces.

The dominant technique is Byte Pair Encoding (BPE), adapted for neural machine translation by Rico Sennrich, Barry Haddow, and Alexandra Birch in their 2015 paper "Neural Machine Translation of Rare Words with Subword Units" (arXiv:1508.07909, later published at ACL 2016). BPE starts from individual characters and repeatedly merges the most frequent adjacent pair into a new symbol, learning a vocabulary of common fragments. Related approaches include WordPiece and SentencePiece. OpenAI's tiktoken library implements BPE encodings such as cl100k_base (used by GPT-4 and GPT-3.5-turbo) and o200k_base (used by GPT-4o).

The full mechanics of building and applying a tokenizer are covered on a dedicated tokenization page; the key point here is that subword tokenization is what makes a fixed vocabulary able to encode essentially any input.

• Subword tokenization balances vocabulary size against the ability to encode rare words.
• Byte Pair Encoding (BPE) merges frequent character pairs into reusable subword units.
• Common methods include BPE, WordPiece, and SentencePiece; tiktoken ships cl100k_base and o200k_base.

Input Tokens, Output Tokens, and the Context Budget

Every interaction with an LLM involves two token streams. Input tokens (also called prompt tokens) are everything sent to the model: the system instructions, prior conversation, retrieved documents, and the current question. Output tokens (or completion tokens) are everything the model generates in response.

Both streams share a single ceiling called the context window, which is the maximum number of tokens the model can attend to at once. The window is a combined budget: input and output draw from the same pool. If a model has a 128,000-token window and the prompt already consumes 120,000 tokens, only about 8,000 tokens remain for the response. The token unit is the subject of this page; the budget itself is covered in depth on the context window page.

• Input tokens are the prompt; output tokens are the generated completion.
• The context window caps input and output combined, not each separately.
• A large prompt directly reduces how much room is left for the answer.

Why Token Counts Drive Pricing and Latency

Commercial LLM APIs bill by the token, and they almost always price input and output separately. Output tokens typically cost several times more than input tokens, commonly in the range of three to five times higher, because generating each new token is a sequential step that demands fresh computation, whereas input tokens can be processed in parallel. As of 2026, published per-token rates differ by vendor and model tier, so cost estimates should always be checked against current pricing pages.

Latency follows a similar logic. Reading a long prompt is comparatively fast because input tokens are handled together, but producing output is inherently step-by-step: the model generates one token, appends it, and predicts the next. Longer responses therefore take proportionally longer to return. In long-running conversations the input also grows turn by turn as history accumulates, which can quietly inflate cost even when each individual reply is short.

• APIs charge per token, with input and output priced separately.
• Output tokens usually cost several times more than input tokens (often 3x to 5x).
• Latency scales with output length because tokens are generated sequentially.

Token Efficiency Across Languages

Tokenizers are trained predominantly on English and other Latin-script text, so they encode English very efficiently and many other languages far less so. The same meaning expressed in a non-Latin script often requires substantially more tokens, which raises both cost and latency for those users and consumes the context window faster.

Reported disparities are large. Peer-reviewed studies and vendor analyses indicate that Arabic commonly needs on the order of two to three times more tokens than equivalent English, Korean often needs roughly one and a half to three times more, and some Indic languages such as Telugu can reach several times the English token count for equivalent content. Chinese, Japanese, and Korean (CJK) scripts and morphologically rich languages tend to fragment into more subword pieces, with one 2025 analysis reporting that non-Latin and morphologically complex languages often incur token inflation around three to five times that of English. Because billing and context limits are denominated in tokens, this tokenization gap means non-English users can effectively pay more for the same information.

• Tokenizers favor English; many languages need more tokens for the same meaning.
• Arabic, Korean, several Indic languages, and CJK scripts often see 1.5x to 5x more tokens.
• Higher token counts raise cost and latency and shrink the usable context window.

Tokens, Memory, and Truncation

An LLM has no built-in long-term memory across the limits of its context window. Within a single session it appears to remember earlier turns only because that history is resent as input tokens on every request. Once the accumulated conversation approaches the context limit, something has to give.

Applications handle this by truncating or compressing history. Common strategies include dropping the oldest turns, summarizing earlier exchanges into a shorter token footprint, or retrieving only the most relevant past content on demand rather than replaying everything. Each approach is fundamentally a token-budgeting decision. AI memory and second-brain tools such as MemX sit on top of this constraint by storing information externally and retrieving only the relevant pieces when asked, so the model receives a compact, relevant set of tokens instead of an ever-growing transcript.

• Models recall earlier turns only because that history is resent as input tokens.
• When history nears the context limit, applications truncate or summarize it.
• External memory and retrieval keep the token footprint small by sending only relevant content.

Practical Tips for Estimating and Managing Tokens

For rough planning, the English heuristics are enough: multiply word count by about 1.3 to estimate tokens, or divide token budgets by 0.75 to estimate words. For anything cost-sensitive or close to a limit, use the model's actual tokenizer, such as OpenAI's tiktoken or an interactive tokenizer playground, to get an exact figure.

Keeping prompts lean pays off on both cost and latency. Trim redundant instructions, avoid resending unchanged context, summarize long histories, and set sensible maximum-output limits so responses do not run longer than needed.

• Estimate quickly with words times 1.3, but verify with a real tokenizer for accuracy.
• Use tiktoken or a vendor tokenizer tool to count exact tokens for a target model.
• Reduce cost and latency by trimming prompts, summarizing history, and capping output length.

Common Misconceptions About Tokens

Several intuitions about tokens are wrong in ways that lead to budgeting and design mistakes. The most common is treating one token as one word; in reality a token is usually a subword fragment, and counts vary by language and content.

Other misconceptions concern where tokens come from and how the budget works. Tokenization is deterministic for a given tokenizer, not random, and different models can split the same sentence differently. Input and output share one context window rather than having independent limits, and spaces and punctuation consume tokens just like letters do.

• A token is not a word; it is typically a subword piece, and ratios differ across languages.
• Tokenization is deterministic per tokenizer, and different models tokenize the same text differently.
• Input and output share one context window; whitespace and punctuation also cost tokens.