Retrieval-Augmented Generation (RAG)

What Retrieval-Augmented Generation Is and the Problem It Solves

Retrieval-Augmented Generation (RAG) is an architecture that combines a pretrained language model with an external, searchable knowledge source. Instead of relying only on facts baked into model weights during training, a RAG system retrieves relevant text at inference time and supplies it to the model as context, so generation is conditioned on fresh, specific evidence.

The term was introduced by Patrick Lewis and colleagues at Facebook AI Research in the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Their framing distinguishes parametric memory (knowledge stored implicitly in the model's weights) from non-parametric memory (an explicit, indexed corpus such as Wikipedia). RAG couples the two.

This solves several limitations of a standalone large language model (LLM). A pretrained model has a fixed knowledge cutoff, cannot see private or proprietary data, and tends to fabricate plausible-sounding but incorrect details when asked about things it does not know. RAG addresses all three by making the authoritative source available at query time and steering the model to answer from it.

• Parametric memory: facts compressed into model weights, fixed at training time.
• Non-parametric memory: an external corpus indexed for search, updatable without retraining.
• Core promise: grounded, current, and source-attributable answers over data the base model never saw.

The Core Pipeline: Ingest, Chunk, Embed, Store, Retrieve, Generate

A RAG system has two phases. An offline indexing phase prepares the knowledge base, and an online query phase answers user questions. Most production systems follow the same sequence of stages.

During indexing, source documents are ingested and split into smaller passages called chunks. Each chunk is converted into a numerical vector (an embedding) by an embedding model, and the vectors are written to a vector store or vector database alongside the original text. At query time the user's question is embedded with the same model, the store returns the chunks whose vectors are nearest to the query vector, those chunks are assembled into a prompt, and the LLM generates an answer conditioned on them.

def rag_answer(query, store, embed, llm, k=5):
    q_vec = embed(query)
    # nearest-neighbor search over stored chunk embeddings
    chunks = store.search(q_vec, top_k=k)
    context = "\n\n".join(c.text for c in chunks)
    prompt = (
        f"Use only the context to answer.\n"
        f"Context:\n{context}\n\n"
        f"Question: {query}"
    )
    return llm.generate(prompt)

• Ingest: load documents from files, databases, web pages, or APIs and normalize them to text.
• Chunk: split each document into retrievable passages of bounded length.
• Embed: map each chunk to a dense vector with an embedding model.
• Store: index the vectors (and metadata) in a vector database for fast nearest-neighbor search.
• Retrieve: embed the query and fetch the top-k most similar chunks.
• Generate: insert the retrieved chunks into the prompt and have the LLM answer from them.

Chunking Strategies and Why They Matter

Chunking is consequential because the chunk is the unit of retrieval: the model can only see what a chunk contains. Chunks that are too large dilute relevance and waste context budget, while chunks that are too small fragment ideas and lose the surrounding context needed to interpret them.

Common strategies trade off simplicity against fidelity to document structure. Fixed-size chunking splits on a token or character count with some overlap between adjacent chunks to avoid cutting sentences awkwardly. Recursive or structure-aware splitting respects natural boundaries such as paragraphs, headings, and list items. Semantic chunking groups sentences by embedding similarity so each chunk covers a single coherent topic.

• Fixed-size with overlap: predictable, cheap, but blind to meaning and structure.
• Recursive / structure-aware: splits on headings and paragraphs to preserve logical units.
• Semantic chunking: groups sentences by similarity so a chunk maps to one idea.
• Metadata enrichment: attach titles, section paths, dates, and source IDs to enable filtering and citation.

Retrieval, Reranking, and Assembling Context for the Model

Dense retrieval encodes the query and each chunk into the same vector space and ranks chunks by similarity, an approach formalized by Dense Passage Retrieval (Karpukhin et al., 2020). It captures semantic matches that keyword search misses, but it can overlook exact terms, rare names, and identifiers.

Because first-pass retrieval optimizes for speed over a large index, many systems retrieve a generous candidate set and then rerank it with a more expensive cross-encoder that scores each query-document pair jointly. Reranking pushes the truly relevant passages to the top before they reach the model, which matters because LLMs attend unevenly to long contexts: Liu et al. (2023) documented a "lost in the middle" effect where evidence placed in the middle of a long prompt is used far less reliably than evidence at the start or end.

• Bi-encoder retrieval: fast approximate nearest-neighbor search over the whole index.
• Cross-encoder reranking: precise but costly scoring applied only to the candidate shortlist.
• Context assembly: deduplicate, order by relevance, respect the model's context limit, and keep source references for citation.

How RAG Reduces Hallucination and Adds Fresh or Private Knowledge

RAG reduces hallucination by changing the task the model faces. Rather than recalling a fact from weights, the model is asked to read supplied passages and answer from them, which is closer to reading comprehension than to open-ended recall. When the prompt instructs the model to rely on the retrieved context and to say when the answer is not present, fabrication drops and answers become attributable to specific sources.

RAG also decouples knowledge from the model. Updating an answer means updating the index, not retraining the model, so new documents become answerable immediately. The same mechanism lets a general-purpose LLM operate over private corpora (internal wikis, contracts, support tickets, personal notes) that were never part of its training data, without exposing that data to a training run.

RAG mitigates hallucination but does not eliminate it. The model can still misread context, blend retrieved facts with parametric priors, or answer confidently when retrieval returned nothing relevant.

• Grounding: answers cite retrieved passages, enabling verification and source attribution.
• Freshness: re-index to reflect new facts without any model retraining.
• Privacy and specialization: query proprietary or personal data with an off-the-shelf model.

RAG vs Fine-Tuning vs Long Context Windows

RAG, fine-tuning, and long context are complementary tools for giving a model knowledge, not interchangeable ones. Fine-tuning adjusts model weights on curated examples; it is well suited to teaching style, format, tone, or a narrow skill, but it is a poor fit for facts that change, because every update requires retraining, and it offers no citation trail.

Long context windows let a model read very large inputs in a single prompt, and as of 2026 several frontier models advertise context windows of around one million tokens, with some larger. This can replace retrieval when the relevant material is small enough to paste in full. At corpus scale, however, stuffing everything into context is slower and far more expensive per query than retrieving a few relevant passages, and the lost-in-the-middle effect means more context does not guarantee better use of it.

In practice teams combine the approaches: RAG supplies the facts, fine-tuning shapes behavior and output format, and a large context window provides the headroom to fit retrieved evidence plus instructions.

• RAG: best for large, changing, or private knowledge that needs source attribution.
• Fine-tuning: best for teaching durable style, format, or a specialized skill, not volatile facts.
• Long context: best when the relevant material is small enough to include directly; costly at full-corpus scale.
• These are layers, not rivals; most production systems use more than one.

Advanced Patterns: Hybrid Retrieval, Agentic RAG, and Graph RAG

Hybrid retrieval combines dense (semantic) search with sparse keyword search such as BM25, then fuses the two ranked lists. Dense search captures meaning while sparse search nails exact terms, names, and codes, so the combination typically recalls relevant passages that either method alone would miss.

Agentic RAG wraps retrieval in a reasoning loop. Instead of a single fixed lookup, an LLM agent decides whether to retrieve, reformulates queries, calls tools or other indexes, and can critique its own retrieved evidence and search again if it is insufficient. This adapts retrieval to question difficulty at the cost of more model calls and higher latency.

Graph RAG indexes a knowledge graph rather than (or alongside) flat passages. Microsoft's GraphRAG (Edge et al., 2024) uses an LLM to extract entities and relationships into a graph, then precomputes summaries of clustered communities, which improves answers to broad, corpus-wide questions that no single retrieved chunk can cover.

• Hybrid search: fuse dense vectors with BM25 to balance semantic recall and exact-match precision.
• Agentic RAG: an agent loops over query rewriting, tool use, and self-correction.
• Graph RAG: traverse entity-relationship graphs for multi-hop and global summarization questions.

Operating RAG in Production: Indexing, Latency, and Cost

Beyond the core pipeline, production RAG systems are shaped by how the corpus changes and how tightly latency and cost are constrained. Corpora that grow continuously need incremental indexing so new documents become searchable without rebuilding the whole index, plus metadata filtering by time and source so queries can be scoped before vectors are compared. RAG also underpins many AI memory and second-brain applications, where a personal corpus of notes and files is indexed for semantic recall.

Latency and cost are driven by the number of model calls and the amount of context assembled. Reranking, agentic loops, and large retrieved contexts each add expense, so teams tune top-k, candidate-set size, and reranker depth against quality targets rather than maximizing retrieval indiscriminately.

• Incremental indexing: add or update vectors as the corpus changes, without full rebuilds.
• Metadata filtering: scope retrieval by time, source, or access control before similarity search.
• Cost and latency tuning: balance top-k, reranking depth, and context size against quality.

Common Failure Modes and How to Evaluate a RAG System

RAG can fail at retrieval or at generation, and good evaluation separates the two. If the relevant chunk is never retrieved, no prompting fixes the answer; if it is retrieved but the model ignores or contradicts it, the problem is generation. Evaluation frameworks therefore score retrieval and generation independently.

RAGAS (Es et al., 2023) popularized reference-light, LLM-graded metrics for this purpose: faithfulness measures whether the answer's claims are supported by the retrieved context, answer relevance measures whether the answer addresses the question, and context-focused metrics such as context precision and context recall measure whether retrieval surfaced the right passages and ranked them highly. Tracking these separately turns vague quality complaints into a specific, fixable stage of the pipeline.

• Retrieval failures: poor chunking, weak embeddings, missing hybrid search, or no reranking.
• Generation failures: ignoring context, blending in parametric priors, or answering when nothing relevant was retrieved.
• Context-ordering failures: relevant evidence buried mid-prompt and underused (lost in the middle).
• Evaluate with faithfulness, answer relevance, and context precision/recall to isolate the failing stage.