Your RAG quality has a hard ceiling, and you set it before the model reads a single word. Not the model, not the prompt: your chunking. Get the split wrong and the best embedding model in the world still answers from garbage. The fix is one default you can ship today: recursive chunking at 400 to 512 tokens with 10 to 20 percent overlap, then tune by symptom. That default handles most prose, and it beats splitting blindly on a fixed token count because it respects paragraph and sentence boundaries first. The size and method you pick shape every vector your retriever ever searches.
Quick takeaways: 1) Default: recursive splitting, 400 to 512 tokens, 10 to 20 percent overlap. 2) Chunk by structure and meaning, not raw token count. 3) Smaller chunks favor precision, larger chunks favor recall. 4) Overlap prevents answers from getting cut at a boundary. 5) Keep tables, code, and markdown sections intact. 6) Late chunking and contextual retrieval fix context loss without rewriting your splitter from scratch.
What is chunking in RAG, and why does it decide retrieval quality?
A chunk is a slice of a document that you embed and store as one searchable unit in a RAG pipeline. Retrieval-augmented generation works by embedding your query, finding the nearest chunks in a vector index, and pasting those chunks into the prompt. The model only answers from what retrieval hands it. If the sentence that holds the answer lands in a chunk that scores poorly, or gets split across two chunks so neither one is complete, the model never sees it.
Embedding models compress a chunk into a single vector. A chunk that mixes three unrelated topics produces a muddy vector that matches nothing well. A chunk that ends mid-thought produces a vector that misses the point it was making. The split you choose shapes every vector, and every vector shapes every search.
Chunking is the one decision you make before the model ever runs. Get the split wrong and the best embedding model in the world still answers from garbage.
One line to remember: the split you choose shapes every vector, and every vector shapes every search.
One line to remember: garbage chunks in, garbage answers out. Your retriever can only find what your splitter kept whole.
Four methods cover almost every pipeline. Fixed-size splits text every N tokens regardless of meaning. Recursive splits on a priority list of separators, paragraphs first, then lines, then spaces, and only falls back to a hard cut when a piece is still too big. Sentence-based keeps whole sentences together. Semantic groups adjacent sentences by embedding similarity so a chunk breaks where the topic shifts.
Fixed-size: simple, fast, and the most likely to cut a sentence in half
Fixed-size chunking is deterministic and trivial to implement. The cost is that it cuts wherever the counter hits the limit, often mid-sentence or mid-idea, which fragments meaning. Use it only as a baseline or when documents have no usable structure.
Recursive: the sane default for most text
Recursive character splitting preserves document structure far better than fixed-size at almost no extra cost. It tries to keep paragraphs whole, then sentences, then words, before resorting to a hard cut. Chunks vary in length, which rarely hurts retrieval.
Semantic: smarter splits, but you pay for every cut
Semantic chunking embeds sentences and breaks the text where similarity between neighbors drops, producing context-aware chunks that track topic boundaries. It tends to improve relevance on dense, mixed-topic documents. The price is compute: you run an embedding pass just to decide where to cut. Reach for it when recursive splitting leaves topics tangled.
| Method | How it splits | Best for | Main weakness |
|---|---|---|---|
| Fixed-size | Every N tokens, ignores meaning | Baselines, structureless text | Cuts sentences and ideas mid-way |
| Recursive | Priority separators: paragraph, line, word | Most prose and docs (default) | Chunk lengths vary |
| Sentence | Keeps whole sentences together | Q&A, transcripts, short facts | Can split a multi-sentence idea |
| Semantic | Breaks where embedding similarity drops | Dense, mixed-topic documents | Extra embedding compute to split |
Pick your method in four questions
- Does the document have structure such as markdown, code, or tables? Use structure-aware splitting.
- Are answers single facts or definitions? Use sentence-based or small chunks of 200 to 300 tokens.
- Are topics densely mixed in one document? Use semantic chunking.
- None of the above? Use recursive at 400 to 512 tokens and ship it.
How to pick a size: recall versus precision
Most RAG systems work well with chunks between 200 and 500 tokens. Smaller chunks sharpen precision: each vector represents one tight idea, so matches are clean, but a single chunk may lack the surrounding context needed to answer. Larger chunks raise recall: more context lives in each chunk, but the vector blurs across multiple ideas and dilutes relevance.
Match chunk size to answer shape. If your questions are answered by a single fact or definition, lean smaller (around 200 to 300 tokens). If answers require reasoning across a few connected sentences, lean larger (around 400 to 512). Test both on a fixed question set before you commit.
Why overlap matters and how much to use
Overlap repeats a slice of text at chunk boundaries so an idea split across two chunks survives in at least one of them. Aim for 10 to 20 percent of the chunk size. A 500-token chunk gets roughly 50 to 100 tokens of overlap. Too little and answers that straddle a boundary vanish; too much and you bloat the index and retrieve near-duplicate chunks that crowd out variety.
Symptoms your chunking is wrong, and the fix for each
Forget the theory. Match the bug you are seeing to the fix below, and bookmark this part.
- Retrieval misses obvious answers: chunks are too large and the vector is diluted. Reduce size or switch to semantic splitting so each chunk holds one idea.
- Answers cut off mid-sentence or reference something not shown: your split lands inside an idea. Increase overlap, or move from fixed-size to recursive so cuts respect boundaries.
- Context bleed, the model blends two unrelated facts: chunks span multiple topics. Switch to semantic chunking or split on structural markers like headings.
- Right document, wrong section retrieved: chunks are too small to carry enough signal. Increase size or add the section heading to each chunk as a prefix.
- Tables and code come back as gibberish: a generic splitter is shredding them. Use structure-aware splitting that keeps these blocks intact.
Structure-aware chunking for code, tables, and markdown
Respect the document's own structure instead of imposing a token grid on top of it. Markdown carries headings that mark natural topic boundaries, so split on heading levels and attach the heading path to each chunk for context. Code has functions and classes that lose meaning when cut in the middle, so split on those units. Tables must stay whole, because a row is meaningless without its column headers.
Never let a splitter cut across a structural unit that only makes sense whole. A function, a table, a list, a fenced code block, and a heading section are all atomic. Recursive splitters can take a custom separator list, so feed them the markers your format uses before they fall back to plain text rules.
Late chunking and contextual retrieval, in plain English
Both techniques attack the same problem: a chunk loses the context that surrounded it. Late chunking flips the usual order. Instead of splitting first and embedding each piece in isolation, it runs the whole document through a long-context embedding model, then pools the token vectors into chunk embeddings afterward. Each chunk vector carries information from the full document, and the method needs no extra training or LLM call.
Contextual retrieval takes a different route, and the payoff is large. Anthropic reported that contextual embeddings cut the top-20 retrieval failure rate by about 35 percent; combined with BM25 keyword search the reduction reached about 49 percent; and adding a reranking step pushed it to roughly 67 percent. The mechanism: before embedding, an LLM writes a short context blurb for each chunk that situates it in the document, and prepends that blurb to the chunk. The cost is an LLM pass over every chunk at index time.
Pick by constraint. Late chunking is cheaper because it skips the per-chunk LLM call and works with existing embedding models. Contextual retrieval preserves meaning more completely but costs more compute, and its gains stack with hybrid search and reranking. Neither replaces a sensible base splitter; they sit on top of one.
A default recipe you can ship today
- Start with recursive splitting at 400 to 512 tokens, 10 to 20 percent overlap, for general prose.
- Feed the splitter your format's separators first: markdown headings, code unit boundaries, table edges. Keep atomic blocks whole.
- Prefix each chunk with its section heading or document title so isolated chunks keep their bearings.
- Build a fixed set of 20 to 50 real questions with known answers, and measure retrieval before and after every change.
- If retrieval still misses, add hybrid search (vectors plus BM25) and a reranker before you rebuild the splitter.
- Only then reach for semantic chunking, late chunking, or contextual retrieval, and re-measure against the same question set.
Key takeaway: chunk by structure and meaning, not by token count alone. Ship recursive splitting at 400 to 512 tokens with 10 to 20 percent overlap, keep tables and code intact, then diagnose by symptom and layer on late chunking or contextual retrieval only where measurements say you need them. Chunking is the one decision you make before the model ever runs: get it right and everything downstream gets easier.
MemX bridge: if you do not want to own a chunking pipeline at all, MemX (memx.app) is an external AI memory layer that handles it for you. It captures your documents, screenshots, photos, voice notes, and forwarded messages, indexes them for retrieval, and lets you ask any model questions against that context. The splitting, overlap, and structure handling happen behind the scenes, so you query your own knowledge without tuning a single chunk size.
01What chunk size should I use for RAG?
Start at 400 to 512 tokens. That sits inside the 200 to 500 range that works for most systems. Go smaller (200 to 300) when answers are single facts, larger when answers need reasoning across several sentences. Test both on a fixed question set before committing.
02How much overlap should RAG chunks have?
Use 10 to 20 percent of the chunk size. A 500-token chunk gets roughly 50 to 100 tokens of overlap. Less and ideas split across a boundary disappear; more and you bloat the index with near-duplicate chunks that crowd out variety.
03Why does my RAG retrieval miss obvious answers?
Usually the chunks are too large, so each vector blurs across multiple ideas and matches the query weakly. Shrink the chunk size or switch to semantic chunking so each chunk holds one idea. Adding hybrid search and a reranker also recovers misses.
04What is the best chunking method for RAG?
Recursive character splitting is the best default for prose: it respects paragraph and sentence boundaries instead of cutting on a token grid. Use semantic chunking for dense mixed-topic documents, and structure-aware splitting for code, tables, and markdown.
05What is the difference between late chunking and contextual retrieval?
Late chunking embeds the whole document first, then pools token vectors into chunks, so each chunk keeps document context with no extra LLM call. Contextual retrieval has an LLM write a context blurb per chunk before embedding, which costs more but preserves meaning more fully.
