Prompting & Reasoning

In-Context Learning

In-context learning (ICL) is the ability of a large language model to perform a new task purely from instructions and example demonstrations placed in its prompt at inference time, with no changes to its weights. The model conditions on the prompt and predicts the next token; the apparent learning happens inside a single forward pass.

What In-Context Learning Means

In-context learning (ICL) describes how a large language model (LLM) carries out a task it was never explicitly trained for, using only the instructions and examples written into its prompt. The phrase was popularized by the GPT-3 paper, Brown et al., "Language Models are Few-Shot Learners" (2020), which showed that a 175-billion-parameter autoregressive model could classify text, translate, or solve arithmetic when the task was "specified purely via text interaction with the model," with no gradient updates or fine-tuning.

The word "learning" here is easy to misread. No parameters change. The model's weights are identical before and after it reads the prompt. What changes is the conditioning: the prompt sets the context against which the next token is predicted. The demonstrations steer the model toward a region of its learned distribution that fits the task, so the behavior looks like rapid adaptation even though nothing about the network has been updated.

This makes ICL a property that emerges from large-scale pretraining rather than a separate training step. A model acquires the capacity for ICL during pretraining on text, then exhibits it at inference whenever a prompt supplies enough signal about the desired input-to-output mapping.

ICL performs a task from prompt instructions and examples alone, with no weight updates.
Brown et al. (2020) introduced the framing with GPT-3, a 175-billion-parameter model.
The model conditions on the prompt within a single forward pass; nothing is saved between requests.
ICL is a capability that emerges from large-scale pretraining, not an extra training phase.

Conditioning, Not Gradient Updates

An autoregressive LLM defines a probability distribution over the next token given everything before it. In ordinary text generation, "everything before it" is just the running passage. In ICL, the prompt is deliberately constructed so that the preceding context contains task instructions and worked examples. The model then predicts the next token conditioned on that whole context.

Contrast this with gradient-based learning such as fine-tuning. There, training examples are used to compute a loss and adjust the weights through backpropagation; the model is permanently altered. In ICL the examples never produce a gradient and never touch the weights. They only appear in the input sequence, so their effect lasts exactly as long as they remain in the context and disappears the moment the request ends.

Because the influence lives entirely in the prompt, ICL is stateless across calls. Two separate API requests share no learned information unless the relevant examples are resent each time. This is the core distinction from any method that writes back into model parameters or external memory.

p(y | x, C) = Π over t of p_θ(y_t | y_<t, x, C),   where C = (x₁,y₁), ..., (x_k,y_k)

ICL as conditioning: the answer y to a query x is the next-token distribution under fixed weights θ, conditioned on k demonstrations C placed in the prompt. The product runs over the tokens of y, and θ never changes.

An LLM models the next token given prior context; ICL just fills that context with task examples.
Fine-tuning changes weights via backpropagation; ICL leaves the weights θ untouched.
Demonstrations influence output only while they remain in the prompt context.
ICL is stateless across separate requests unless examples are resent each time.

Zero-Shot, One-Shot, and Few-Shot

The GPT-3 paper laid out a spectrum based on how many demonstrations the prompt contains. Zero-shot gives only a natural-language instruction and the query, with no worked examples. One-shot adds a single example. Few-shot supplies several examples, typically a handful up to a few dozen, bounded by the context window. Brown et al. found that performance generally rose as the number of demonstrations increased, and that larger models benefited more from added examples.

These settings are points on a continuum, not discrete algorithms. The same model and the same forward-pass mechanism handle all of them; the only variable is how much demonstration text sits in the prompt. A zero-shot prompt leans entirely on the model's pretrained association between an instruction and a behavior, while a few-shot prompt also shows the exact input-output format expected.

Constructing a few-shot prompt is mechanical: lay out example pairs in a consistent format, then append the new query in the same format and let the model complete it.

python

# Build a few-shot ICL prompt for sentiment classification
demos = [
    ("The plot dragged and I nearly left.", "negative"),
    ("A warm, funny, beautifully acted film.", "positive"),
    ("Forgettable but harmless.", "neutral"),
]

query = "The pacing was perfect and the ending floored me."

lines = [f"Review: {text}\nSentiment: {label}" for text, label in demos]
prompt = "\n\n".join(lines) + f"\n\nReview: {query}\nSentiment:"

# The model completes after 'Sentiment:' using only these in-prompt examples.
print(prompt)

A few-shot ICL prompt: consistent example formatting teaches the label space and output shape, then the query is appended in the same format for the model to complete.

Zero-shot: instruction and query only, no demonstrations.
One-shot: exactly one demonstration before the query.
Few-shot: several demonstrations, capped by the context window.
Brown et al. found more demonstrations usually help, with larger models gaining more.

Why It Works: Induction Heads and Meta-Optimization

The mechanism behind ICL is an active research question rather than a settled fact. One influential line comes from Anthropic's interpretability work. Olsson et al., "In-context Learning and Induction Heads" (2022), identify induction heads: attention circuits that copy information forward and search back for earlier occurrences of the current token to predict what followed before. The authors document a training phase change where induction heads form and ICL ability sharply improves at the same time, arguing the two are causally linked.

A second line proposes that transformers implicitly run an optimization process in-context. Von Oswald et al., "Transformers learn in-context by gradient descent" (2022), construct weights for a linear self-attention layer that exactly reproduce a gradient-descent step on a regression loss, suggesting the forward pass can emulate gradient-based learning. Garg et al., "What Can Transformers Learn In-Context?" (2022), show empirically that transformers trained on input-output pairs can in-context fit linear functions, sparse linear functions, two-layer networks, and decision trees, competitive with task-specific estimators.

These accounts are complementary rather than contradictory. Pattern-completion circuits like induction heads may underpin the copying and analogy that ICL relies on, while the meta-optimization view explains how a fixed network can behave like a learner on structured tasks. Neither claim should be overstated: much of the evidence comes from small models or simplified function classes, and the picture for full-scale LLMs is still incomplete.

Induction heads (Olsson et al. 2022) copy and pattern-match across the sequence; their formation coincides with ICL ability.
Von Oswald et al. (2022) show a linear attention layer can replicate a gradient-descent step.
Garg et al. (2022) show transformers can in-context fit linear functions, decision trees, and more.
Mechanism evidence often comes from small or synthetic settings, so conclusions remain partial.

Sensitivity and Surprising Failures

ICL is powerful but brittle. Outputs can shift with the order of demonstrations, the choice of label words, and the exact prompt formatting, which is why prompt engineering remains a practical discipline. Small wording changes that a human would treat as equivalent can move accuracy substantially.

A striking result complicates the intuition that examples teach the correct mapping. Min et al., "Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?" (2022), found that randomly replacing the labels in the demonstrations barely hurt performance across a range of classification and multiple-choice tasks, consistently over 12 models including GPT-3. Their reading is that demonstrations help mainly by showing the label space, the input distribution, and the expected output format, rather than by supplying correct input-to-label pairings.

This does not mean labels never matter, and later work has refined the conditions under which correctness helps. The takeaway is narrower and useful: a large share of ICL's gain comes from format and distribution cues, so well-structured demonstrations matter as much as perfectly labeled ones.

ICL is sensitive to demonstration order, label wording, and prompt format.
Min et al. (2022) found random labels barely hurt accuracy across 12 models including GPT-3.
Demonstrations appear to convey label space, input distribution, and output format strongly.
Correct labels still matter in some settings; the result is a nuance, not a blanket rule.

Practical Limits

The hardest constraint on ICL is the context window: every instruction and demonstration competes for a fixed token budget, so the number of examples that fit is bounded. Larger windows allow more demonstrations but raise cost and latency, since attention work grows with sequence length.

Position within the context also matters. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023), found that models use information best when it sits near the beginning or end of the prompt and degrade noticeably when the relevant content is buried in the middle, even for models built for long contexts. Packing many demonstrations into a long prompt can therefore bury useful signal in a low-attention region.

Because ICL holds nothing between calls, repeated tasks resend the same examples each request, which is expensive at scale and is one reason teams turn to retrieval-augmented generation or fine-tuning when a behavior must persist cheaply. ICL is best when a task is occasional, varied, or fast-changing and a few good examples in the prompt are enough.

Demonstrations compete for a fixed context window, capping how many examples fit.
Lost-in-the-middle (Liu et al. 2023): middle-of-prompt content is used least reliably.
ICL keeps no state between calls, so examples are resent each request, raising cost at scale.
For persistent or high-volume behavior, fine-tuning or retrieval often beats large few-shot prompts.

Key takeaways

In-context learning performs new tasks from prompt instructions and examples alone, with no weight updates.
The GPT-3 paper (Brown et al. 2020) introduced ICL using a 175-billion-parameter model.
The learning is conditioning inside one forward pass; weights θ never change and nothing persists between calls.
Zero-shot, one-shot, and few-shot are points on a spectrum set only by how many demonstrations the prompt holds.
Induction heads and in-context meta-optimization are leading but still partial explanations of why ICL works.
ICL is brittle to order and format, and bounded by the context window and lost-in-the-middle effects.

Frequently asked questions

It is when a large language model does a new task just from instructions and a few examples written into its prompt, without anyone retraining it. You show it the pattern in the prompt, and it follows that pattern for your query. The model's internal weights stay exactly the same.

Fine-tuning changes the model's weights using training examples and backpropagation, so the change is permanent and applies to every future request. In-context learning changes nothing in the model; the examples live only in the prompt and influence output solely while they are present. ICL is instant and disposable, while fine-tuning is durable but requires a training run.

Because the model adapts its behavior to the task from examples, which looks like learning. Mechanistically it is conditioning: the prompt steers the next-token distribution toward outputs that fit the demonstrated pattern. Some research even shows transformers can emulate a gradient-descent step inside the forward pass, which is why the learning analogy is more than loose metaphor.

Less than expected. Min et al. (2022) found that replacing the labels in demonstrations with random ones barely hurt accuracy across 12 models including GPT-3. Demonstrations seem to help mainly by showing the label space, input style, and output format, though correct labels still matter in some tasks.

They differ only in how many examples the prompt contains. Zero-shot gives an instruction and the query with no examples, one-shot adds a single example, and few-shot supplies several. The same model and mechanism handle all three; more examples usually help, up to the limit of the context window.

The context window caps how many demonstrations fit, and longer prompts cost more and run slower. Models also use the middle of a long prompt least reliably, a finding called lost-in-the-middle (Liu et al. 2023). Because ICL keeps no state between calls, examples must be resent every request, so persistent or high-volume tasks often favor fine-tuning or retrieval.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free