Tokenization (and Tokenizers)

What Tokenization Is and Where It Sits in the LLM Pipeline

Tokenization is the first step in every large language model (LLM) pipeline. A language model cannot operate on characters or words directly: it operates on integers. Tokenization is the process of cutting a raw text string into a sequence of discrete units called tokens, each of which maps to an integer ID in a fixed vocabulary. Those IDs are then looked up in an embedding table to produce the vectors the neural network actually consumes.

The component that performs this step is the tokenizer. It bundles two things: a learned vocabulary (the set of known token strings and their IDs) and an encoding procedure that deterministically converts any input string into IDs and back again. Encoding turns text into IDs; decoding reverses the mapping to reconstruct text from the IDs the model generates.

Tokenization sits before the model and is shared between input and output. The same vocabulary that encodes a prompt also defines the units the model predicts one at a time during generation. Because the vocabulary is fixed at training time, the tokenizer is effectively part of the model's interface: swapping it requires retraining.

• Input path: raw text to tokens to integer IDs to embedding vectors.
• Output path: predicted token IDs decoded back into text.
• The tokenizer and its vocabulary are frozen at training time and cannot be changed without retraining.

Subword Tokenization and Why It Beat Word and Character Approaches

Early systems tokenized on whole words or single characters. Both extremes have serious drawbacks. Word-level vocabularies are huge, cannot represent any word they did not see in training (the out-of-vocabulary, or OOV, problem), and waste capacity on rare inflections. Character-level vocabularies are tiny and never hit OOV, but they produce very long sequences and force the model to relearn spelling from scratch, which is slow and costly given that attention scales with sequence length.

Subword tokenization is the compromise that won. It keeps frequent words whole, splits rarer words into reusable fragments such as roots, prefixes, and suffixes, and falls back to bytes or characters for anything unseen. This gives a compact, fixed vocabulary that can still encode any string, captures morphology so the model generalizes across related words, and keeps sequences far shorter than character-level encoding.

• Word-level: large vocabulary, brittle to unseen words (OOV).
• Character-level: tiny vocabulary, no OOV, but very long sequences.
• Subword: fixed vocabulary, no true OOV, sequences shorter than character-level, morphology captured automatically.

Byte Pair Encoding (BPE): Iterative Merging of Frequent Pairs

Byte Pair Encoding is the most widely used subword algorithm. It was adapted from a data-compression technique to neural machine translation by Sennrich, Haddow, and Birch in their paper Neural Machine Translation of Rare Words with Subword Units, first posted in 2015 and published at ACL in 2016, which made open-vocabulary translation practical.

BPE training starts with a base vocabulary of individual symbols (characters, or raw bytes in byte-level BPE). It then repeatedly scans the corpus, finds the most frequent adjacent pair of tokens, and merges that pair into a single new token, recording the merge as a rule. Early merges create two-character tokens; as training continues, longer and longer subwords emerge. Merging stops when the vocabulary reaches a target size. At encode time the learned merge rules are reapplied in order.

Byte-level BPE, used by GPT-2, operates on the 256 possible byte values rather than Unicode characters. Because every possible byte is in the base vocabulary, the tokenizer can represent any input, including emoji and arbitrary scripts, with no genuine OOV case.

• Greedy and frequency-driven: each step merges the single most frequent adjacent pair.
• Merge rules are ordered and replayed deterministically at encode time.
• Byte-level BPE starts from 256 bytes, guaranteeing any string is encodable.

WordPiece and Unigram: How They Differ From BPE

WordPiece, used by BERT, looks almost identical to BPE but changes the merge criterion. Instead of merging the most frequent pair, it merges the pair that most increases the likelihood of the training data, scoring candidates by how much more often two tokens co-occur than chance would predict. This favors merges that are informative rather than merely common, at higher computational cost.

The Unigram language model, introduced by Kudo (2018), works in the opposite direction. Rather than building up from characters, it starts from a large candidate vocabulary and prunes it down: it estimates a probability for each subword, then iteratively removes the units whose removal least hurts the overall likelihood of the corpus, stopping at the target size. Unigram is probabilistic and can represent a word in multiple ways, which also enables regularization techniques during training.

In short, BPE is bottom-up and frequency-based, WordPiece is bottom-up and likelihood-based, and Unigram is top-down and likelihood-based.

• BPE: merge the most frequent adjacent pair (bottom-up, greedy).
• WordPiece: merge the pair that most raises training-data likelihood (bottom-up).
• Unigram: start large and prune low-value subwords by likelihood (top-down, probabilistic).

SentencePiece: Treating Raw Text as a Character Stream

SentencePiece, described by Kudo and Richardson (2018), is not a fourth merge algorithm but a tokenizer implementation and toolkit. It can train either BPE or Unigram models, and its key contribution is operating directly on raw, untokenized text rather than relying on a language-specific pre-tokenizer that assumes spaces separate words.

SentencePiece treats the entire input as a stream of Unicode characters, escaping whitespace as a visible meta symbol (U+2581, shown as an underscore). Because spaces are encoded as ordinary symbols, tokenization is lossless and fully reversible: the original text, including its exact spacing, can always be reconstructed from the tokens. This matters for languages such as Chinese, Japanese, and Thai that do not delimit words with spaces, which is why SentencePiece became a standard for multilingual models.

• A toolkit that can train BPE or Unigram, not a separate algorithm.
• Consumes raw text with no language-specific pre-tokenizer.
• Encodes whitespace explicitly (U+2581) for fully reversible, lossless tokenization.

Vocabulary Size and Merges (As of 2026)

Vocabulary size is the central design knob. A larger vocabulary packs more text into each token, shortening sequences but enlarging the embedding table and the output softmax. A smaller vocabulary keeps the model compact but lengthens sequences. There is a direct trade-off between vocabulary size and sequence length.

Concrete reference points, as of 2026: GPT-2 uses byte-level BPE with a vocabulary of 50,257 tokens (256 byte tokens, 50,000 learned merges, and one end-of-text token), averaging roughly four English characters per token. OpenAI's tiktoken library exposes successive encodings: r50k_base/gpt2 (about 50k), cl100k_base (about 100k, used by GPT-4 and GPT-3.5-Turbo), and o200k_base (about 200k), the encoding used by GPT-4o. The o200k_base vocabulary roughly doubles cl100k_base and improves compression notably for non-English scripts. For monolingual English models a common range is 30,000 to 50,000 tokens, while modern multilingual models trend toward 100k to 200k and beyond.

• GPT-2: 50,257 tokens (256 bytes + 50,000 merges + 1 special), as of 2026.
• cl100k_base: roughly 100k, used by GPT-4 and GPT-3.5-Turbo, as of 2026.
• o200k_base: roughly 200k, used by GPT-4o, with better non-English compression, as of 2026.
• Larger vocabularies shorten sequences but enlarge the embedding and softmax layers.

Why Tokenization Causes Spelling, Math, and Non-English Quirks

Many surprising LLM failures trace back to tokenization rather than reasoning. Because a model sees tokens, not letters, it has no direct view of spelling. A word that is a single token offers no internal character structure, so tasks like counting letters or reversing a string become unexpectedly hard.

Arithmetic suffers similarly. If numbers are split inconsistently (for example, a multi-digit number tokenized as one chunk in one place and split differently elsewhere), the model cannot rely on a stable place-value structure, which contributes to errors on otherwise simple calculations. Some tokenizers, such as the Llama tokenizer, deliberately split digits individually to mitigate this.

Tokenization can also create glitch tokens: rare strings that received a vocabulary slot but were almost never seen in training, leaving their embeddings poorly trained. The widely cited SolidGoldMagikarp case, surfaced in early 2023 by researchers Jessica Rumbelow and Matthew Watkins, showed such tokens producing erratic model behavior.

• Spelling and letter-counting tasks are hard because tokens hide character structure.
• Inconsistent digit splitting undermines place value and harms arithmetic.
• Rare, undertrained vocabulary entries (glitch tokens) can trigger erratic outputs.

Downstream Effects on Cost, Context Limits, and Language Fairness

Tokenization is not just an implementation detail: it shapes economics and access. API pricing and context windows are denominated in tokens, not characters or words, so a more efficient tokenizer makes the same text cheaper to process and lets more of it fit inside a fixed context limit.

This creates a fairness gap across languages. Tokenizers trained predominantly on English encode English text efficiently but fragment other scripts into many more tokens per equivalent meaning. The result is that speakers of underrepresented languages pay more, hit context limits sooner, and may receive lower-quality outputs for the same content. Improved multilingual compression, such as the gains in o200k_base over cl100k_base for non-Latin scripts as of 2026, partly narrows but does not eliminate this gap.

• Cost and context windows are measured in tokens, so token efficiency is money and capacity.
• English-centric vocabularies inflate token counts for other languages.
• Higher token counts mean higher cost, earlier context truncation, and potential quality loss.

Choosing or Inspecting a Tokenizer in Practice

For application developers, the practical task is usually inspecting an existing tokenizer rather than training one. Counting tokens before sending a request lets you predict cost and stay within context limits. OpenAI's tiktoken library returns the exact encoding for a given model, and the Hugging Face tokenizers library covers BPE, WordPiece, and Unigram for open models. Visual tools that show how a string splits make tokenization quirks easy to spot.

When training a model from scratch, the main decisions are which algorithm to use (BPE, WordPiece, or Unigram, often via SentencePiece for raw multilingual text), the target vocabulary size, and how to handle digits and whitespace. The right vocabulary size depends on the language mix and the sequence-length versus model-size trade-off.

Retrieval and AI-memory systems are also affected: tools that store and search documents must account for tokenizer behavior when chunking text and estimating how much retrieved context fits inside a model's window.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
token_ids = enc.encode("Tokenization splits text into subwords.")
print(token_ids)
print([enc.decode([t]) for t in token_ids])
print("token count:", len(token_ids))

• Use tiktoken for OpenAI models and Hugging Face tokenizers for open models.
• Count tokens up front to forecast cost and avoid context overflow.
• When training, choose algorithm, vocabulary size, and digit/whitespace handling based on your language mix.