Retrieval & Context

Chunking Strategies

Chunking strategies are the rules that split long documents into smaller passages before embedding and indexing them for retrieval. The strategy fixes the size, boundaries, and overlap of each chunk, which sets the ceiling on what a retrieval system can ever return.

What document chunking is and why it sets the retrieval ceiling

Chunking is the step in a retrieval pipeline that splits a long source document into smaller passages, called chunks, before each passage is converted to a vector embedding and stored in an index. A chunking strategy is the specific procedure that decides where one chunk ends and the next begins, how large each chunk is, and whether neighboring chunks share any text. Retrieval-augmented generation (RAG) and semantic search both depend on this step because they retrieve whole chunks, not arbitrary spans, so a chunk is the smallest unit a query can ever match.

The strategy sets an upper bound on retrieval quality that no downstream component can recover. If a fact is split across two chunks, neither chunk contains the complete statement, and a query that needs the whole fact will match a partial passage at best. If a chunk mixes several unrelated topics, its embedding averages those topics into a single vector that represents none of them well, which lowers the similarity score for any specific query. Reranking, query rewriting, and a larger context window all operate on whatever the chunker produced, so errors introduced here propagate to every later stage.

Chunk boundaries also interact with the embedding model. Most embedding models have a fixed maximum input length, and text past that limit is truncated rather than embedded. A chunk longer than the model's maximum is therefore represented only by its opening tokens, with the remainder silently discarded. Matching chunk size to the model's limit is part of the strategy, not an afterthought.

A chunk is the atomic unit of retrieval: queries match whole chunks, never sub-spans within them.
A fact split across two chunks cannot be retrieved intact by a single matched passage.
Mixing topics in one chunk dilutes its embedding, lowering similarity for any specific query.
Chunking happens once at index time; fixing a bad strategy means re-embedding the entire corpus.

Fixed-size versus recursive chunking

Fixed-size chunking splits text at a constant length, for example every 500 tokens, ignoring sentence and paragraph boundaries. It is simple and fast, and it guarantees that every chunk fits the embedding model's input limit, but it routinely cuts sentences and even words in half. A chunk can end mid-clause, leaving the start of an idea in one chunk and its conclusion in the next.

Recursive chunking improves on this by trying a prioritized list of separators in order and only falling back to a coarser split when a piece is still too large. LangChain's RecursiveCharacterTextSplitter is the widely used reference implementation. Its default separator list is ["\n\n", "\n", " ", ""], so it first tries to keep paragraphs whole, then lines, then words, and only as a last resort splits between individual characters. The result respects natural boundaries far more often than a fixed cut while still honoring a target size, which is why recursive splitting is the common default for general text.

Both methods are size-driven rather than meaning-driven. Recursive splitting prefers cleaner boundaries, but it still decides where to cut based on a character or token budget, not on whether the surrounding text is about the same topic. For prose with regular structure this is usually enough; for documents where topics shift inside a single paragraph it can still place a boundary in the wrong place.

python

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,      # max size measured by length_function (default: len)
    chunk_overlap=50,    # characters shared between consecutive chunks
    separators=["\n\n", "\n", " ", ""],  # tried in order, coarsest last
)

chunks = splitter.split_text(document_text)
# Each chunk is at most ~500 chars and prefers paragraph/line/word breaks.

Recursive splitting with LangChain. chunk_size and chunk_overlap are measured by length_function, which is len (characters) by default; use from_tiktoken_encoder to measure in tokens instead.

Fixed-size splitting cuts at a constant length and can break sentences or words mid-token.
Recursive splitting tries separators in priority order: paragraph, line, word, then character.
LangChain's default separators are ["\n\n", "\n", " ", ""], preferring the largest intact unit.
Both are size-driven, so neither guarantees a boundary falls at a genuine topic change.

Semantic and document-structure chunking

Semantic chunking sets boundaries by meaning rather than length. A common recipe embeds each sentence, computes the cosine distance between consecutive sentence embeddings, and starts a new chunk wherever that distance exceeds a threshold, for example the 95th percentile of all distances in the document. A large jump in embedding distance signals a topic shift, so the split lands at a natural seam. Implementations often add a small buffer of neighboring sentences to each embedding so that a single short sentence does not trigger a spurious break.

Document-structure chunking uses the explicit layout of a file instead of inferred meaning. Markdown headers, HTML tags, code-block fences, table rows, and PDF section markers all give boundaries that the author already chose. Splitting on these keeps a heading attached to its body and keeps a code block or table intact, which matters because a code snippet cut in half is rarely useful when retrieved. For structured formats this often outperforms both size-based and semantic methods because the structure already encodes topic boundaries.

These approaches cost more than size-based splitting. Semantic chunking requires embedding every sentence at index time, which adds compute and latency to ingestion, and its threshold needs tuning per corpus. Structure-based chunking depends on the document actually having reliable markup, which clean Markdown provides but scanned PDFs often do not. The right choice depends on the format and the value of boundary precision for the corpus.

Semantic chunking embeds sentences and splits where consecutive cosine distance crosses a percentile threshold.
A buffer of neighboring sentences stabilizes the embedding so single short lines do not force a break.
Structure-based chunking splits on headers, tags, code fences, or table rows the author already defined.
Semantic chunking adds index-time embedding cost; structure chunking needs reliable markup to work.

Chunk size and overlap tradeoffs

Chunk size trades precision against context. Small chunks produce focused embeddings, so a matched chunk is tightly about the query, but a small chunk may omit context the model needs to use the fact correctly. Large chunks carry more context per retrieval, yet their embeddings average more content and match queries less sharply, and they consume more of the prompt budget per retrieved passage. There is no universal best size; typical RAG pipelines use roughly 200 to 1000 tokens per chunk and tune within that range against an evaluation set.

Overlap is the amount of text repeated between consecutive chunks, often expressed as a fraction of chunk size such as 10 to 20 percent. Overlap exists to protect facts that would otherwise be cut by a boundary: a sentence that ends one chunk also begins the next, so a query needing that sentence can match either side. The cost is redundancy, since overlapping text is embedded and stored more than once, inflating index size and the chance of returning near-duplicate passages.

Size and overlap must also respect the embedding model's input limit. A model like all-MiniLM-L6-v2 accepts up to 256 word pieces and truncates anything longer, so a chunk measured in characters can silently exceed the model's token capacity and lose its tail. Larger models such as OpenAI's text-embedding-3-small accept up to 8191 tokens, which removes the truncation risk but does not remove the precision cost of oversized chunks. Measuring chunk size in the same units the model counts, tokens, avoids this trap.

n = ceil( (L - o) / (s - o) ),  for s > o

Estimated number of chunks n for a document of length L using chunk size s and overlap o (same units, e.g. tokens). It is an estimate because separator-aware splitters land on real boundaries rather than at exact stride offsets. Each chunk after the first advances by the stride s minus o, so the effective step is (s - o). Overlap must be smaller than size or the window never moves forward.

Small chunks give sharper matches but may drop needed context; large chunks blur the embedding.
Typical chunk sizes run about 200 to 1000 tokens, tuned per corpus against an eval set.
Overlap of roughly 10 to 20 percent guards boundary-crossing facts at the cost of stored redundancy.
Chunk size must stay within the embedding model's max input or the tail is silently truncated.

Tokens versus characters and how to measure size

Chunk size can be measured in characters or in tokens, and the two are not interchangeable. A token is the unit a tokenizer emits, often a sub-word fragment, while a character is a single letter or symbol. For English text one token averages roughly four characters, but the ratio varies with language, code, punctuation, and whitespace, so a chunk capped at 500 characters and a chunk capped at 500 tokens are different sizes.

This distinction matters because both embedding models and the generation model count tokens, not characters. An embedding model's maximum input is stated in tokens, and a chunk that looks safely small in characters can still exceed that token limit and be truncated. The prompt budget that retrieved chunks must fit into is also measured in tokens. By default LangChain's splitter measures length with the Python len function, which counts characters, so without changing the length function the configured size does not correspond to the model's token count.

The practical fix is to measure chunk size with the same tokenizer the model uses. LangChain provides RecursiveCharacterTextSplitter.from_tiktoken_encoder, which counts length in tiktoken tokens so that chunk_size and chunk_overlap are expressed in the units the model actually sees. This keeps every chunk inside the embedding model's input limit and makes prompt-budget arithmetic exact rather than approximate.

A token is a sub-word unit; one English token averages about four characters, but the ratio varies.
Embedding limits and prompt budgets are counted in tokens, so character-based sizing can mislead.
LangChain measures length with len (characters) by default, not with the model's tokenizer.
Use from_tiktoken_encoder so chunk_size and chunk_overlap are measured in the model's tokens.

Key takeaways

Chunking splits documents into the passages a retrieval system indexes and returns; the chunk is the smallest retrievable unit.
The strategy sets a ceiling on retrieval quality that reranking and larger context windows cannot recover.
Recursive splitting respects paragraph, line, and word boundaries, while semantic and structure-based methods split on meaning or markup.
Smaller chunks match more sharply but carry less context; typical sizes run about 200 to 1000 tokens.
Overlap of roughly 10 to 20 percent protects facts that fall on a boundary at the cost of stored redundancy.
Measure size in the model's tokens, not characters, so chunks stay within the embedding model's input limit.

Frequently asked questions

Fixed-size chunking cuts text at a constant length and ignores sentence or paragraph boundaries, so it can break words and clauses in half. Recursive chunking tries a prioritized list of separators, paragraph then line then word, and only falls back to a finer split when a piece is still too large. Recursive splitting respects natural boundaries far more often while still honoring a target size, which is why it is the common default for general prose.

A common starting point is a chunk of a few hundred tokens with an overlap of about 10 to 20 percent of that size, then tuning against an evaluation set. Typical pipelines use roughly 200 to 1000 tokens per chunk. Smaller chunks give sharper matches but less context, so the best value depends on the corpus and the question types, and it should be measured rather than assumed.

By default LangChain's splitter measures length with the Python len function, which counts characters, so the configured size does not match the model's token count. Use RecursiveCharacterTextSplitter.from_tiktoken_encoder to measure length with a tiktoken tokenizer instead. That makes chunk_size and chunk_overlap correspond to the tokens the embedding and generation models actually count.

Yes. Most embedding models have a fixed maximum input length and silently truncate text beyond it. For example, all-MiniLM-L6-v2 accepts up to 256 word pieces, while OpenAI's text-embedding-3-small accepts up to 8191 tokens. A chunk larger than the limit is represented only by its opening tokens, so the rest of its content never reaches the index.

Semantic chunking embeds every sentence and splits where consecutive embeddings diverge, which adds compute and latency at index time and needs a tuned threshold. It is worth that cost when topics shift inside paragraphs and clean boundaries materially improve retrieval. For documents with reliable structure such as Markdown headers or code fences, splitting on that structure is often simpler and just as effective.

Retrieval returns whole chunks, so a chunk is the smallest unit a query can match. If a fact is split across two chunks neither contains it intact, and if a chunk mixes topics its embedding averages them and matches poorly. Because chunking happens before embedding and indexing, every later stage operates on whatever the chunker produced, so a poor strategy caps the accuracy of the entire pipeline.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free