Prompting & Reasoning

Few-Shot Learning

By Aditya Kumar Jha, Engineer

Few-shot learning is the ability to learn a new task or recognize a new class from only a handful of labeled examples. In modern large language models it most often means in-context learning: placing a few input-output demonstrations directly in the prompt so the model performs the task with no weight updates.

What few-shot learning means

Few-shot learning refers to learning a new task, or recognizing a new category, from only a small number of labeled examples, typically a few per class rather than the thousands a conventional supervised model expects. The term is used in two related but distinct senses. The first is the classical machine learning sense, where a model is explicitly trained to generalize from tiny labeled sets, usually through meta-learning. The second is the large language model sense popularized by GPT-3, where a few demonstrations are written into the prompt and the model performs the task immediately without any change to its weights.

The two senses share a goal, generalizing from scarce supervision, but differ in mechanism. Classical few-shot learning still updates parameters during a meta-training phase and sometimes adapts at test time. In-context few-shot learning involves no gradient steps at all once the model is trained, the demonstrations act only as conditioning text. Reading any claim about few-shot learning requires knowing which sense is intended, because the engineering implications are different.

Few-shot sits on a spectrum defined by how many examples are shown. Zero-shot provides only a task description, one-shot provides a single demonstration, and few-shot provides several, often two to a few dozen. The boundaries are conventional rather than precise, but the trend is consistent: more high-quality demonstrations usually help up to a point, after which the prompt grows long and returns diminish.

  • Two senses: classical meta-learning from few labels, and LLM in-context learning from prompt demonstrations.
  • Classical few-shot updates weights during meta-training; in-context few-shot updates none.
  • It sits between zero-shot (task description only) and full fine-tuning (many labeled examples).
  • The 'shot' count is the number of labeled demonstrations available for the new task.

The classical N-way K-shot setup

In classical few-shot classification the standard evaluation protocol is the N-way K-shot episode. Each episode samples N previously unseen classes and provides K labeled examples per class as a support set, so the support set holds N times K labeled points in total. The model must then classify query examples drawn from those same N classes. A common configuration is 5-way 1-shot, five new classes with a single example each, which is deliberately hard.

Meta-learning trains a model across many such episodes so that the learning procedure itself generalizes to new episodes with new classes. This is sometimes phrased as learning to learn: rather than learning a fixed classifier, the model learns an algorithm that turns a small support set into a working classifier. Benchmarks like Omniglot and miniImageNet are built around this episodic protocol.

Methods fall into broad families. Metric-based methods learn an embedding space where simple distance comparisons classify query points. Matching Networks (Vinyals et al., 2016, arXiv:1606.04080) map a support set and a query to a label using a learned similarity. Prototypical Networks (Snell et al., 2017, arXiv:1703.05175) average each class's support embeddings into a prototype and assign each query to the nearest prototype. Optimization-based methods instead learn good initial weights: MAML (Finn et al., 2017, arXiv:1703.03400) trains parameters so that a few gradient steps on a new task's small dataset yield strong performance.

S = { (xᵢ, yᵢ) }, i = 1..N·K with N classes, K examples each
The support set S in an N-way K-shot episode contains N times K labeled examples; the model classifies separate query examples from the same N classes.
cₖ = (1/K) Σ_{(xᵢ,yᵢ)∈Sₖ} f(xᵢ) ; ŷ = argminₖ d( f(x), cₖ )
Prototypical Networks: each prototype cₖ is the mean embedding of class k's support examples, and a query x is assigned to the class whose prototype is nearest under distance d.
  • An N-way K-shot episode has N new classes and K labeled examples each, N times K support points total.
  • Meta-learning trains across many episodes so the learning procedure generalizes to unseen classes.
  • Metric methods (Matching, Prototypical Networks) classify by distance in a learned embedding space.
  • Optimization methods like MAML learn an initialization that adapts in a few gradient steps.

Few-shot prompting in large language models

The LLM sense of few-shot learning comes from the GPT-3 paper, Language Models are Few-Shot Learners (Brown et al., 2020, arXiv:2005.14165, submitted 28 May 2020). It introduced a 175 billion parameter autoregressive model and showed that scaling lets a model perform many tasks when given only a task description and a few demonstrations in its context, with, in the authors' words, no gradient updates or fine-tuning. The demonstrations are specified purely as text the model reads.

This mechanism is called in-context learning. The model is conditioned on a sequence that contains several worked input-output pairs followed by a new input, and it predicts the output by continuing the pattern. No parameters change. The same frozen model can be steered to translation, classification, or arithmetic simply by swapping which demonstrations appear in the prompt. The paper distinguishes three regimes by demonstration count: zero-shot gives only the instruction, one-shot gives a single example, and few-shot gives several.

Few-shot prompting helps most when the task is hard to describe but easy to show, when a specific output format is required, or when the label set is unusual. It helps less when the model already handles the task zero-shot, in which case demonstrations mostly add token cost. The probability the model assigns to an output can be read as conditioning on both the new input and the demonstrations placed before it.

p( y | x, D ) where D = (x₁,y₁), (x₂,y₂), ..., (x_k,y_k)
In-context few-shot prediction: the model conditions on the new input x and the k demonstrations D placed earlier in the prompt; only the context changes, never the parameters.
text
Classify the sentiment as positive or negative.

Review: The plot dragged and I nearly left.
Sentiment: negative

Review: A warm, funny, beautifully acted film.
Sentiment: positive

Review: Forgettable but harmless.
Sentiment:
A two-shot prompt template. Two labeled demonstrations set the label space and format; the final unlabeled review is what the model completes.
  • GPT-3 (175B parameters, arXiv:2005.14165) framed in-context demonstrations as few-shot learning.
  • In-context learning conditions a frozen model on prompt demonstrations; no weights change.
  • Regimes by example count: zero-shot, one-shot, few-shot.
  • Few-shot prompting earns its token cost most when a task needs a shown format or unusual labels.

Demonstration selection and ordering effects

In-context few-shot performance is sensitive to which demonstrations are chosen and how they are ordered. Lu et al. (2022, arXiv:2104.08786, Fantastically Ordered Prompts) showed that simply reordering the same set of demonstrations can move accuracy from near state of the art to near random guessing. The effect persists across model sizes, and a good ordering found for one model does not reliably transfer to another, so ordering cannot be treated as a fixed property of the examples.

Selection matters alongside ordering. Demonstrations that resemble the test input, for example chosen by semantic similarity, often outperform random ones. A counterintuitive finding from Min et al. (2022, arXiv:2202.12837) is that the correctness of demonstration labels matters less than expected: randomly corrupting the labels barely hurt performance across the models tested. What carried the signal was the label space, the input distribution, and the overall format. In other words, demonstrations teach the model what kind of answer to produce and in what shape, more than they teach the specific input-to-label mapping.

These results have practical consequences. Building a few-shot prompt is partly an empirical search over example choice, count, and order, validated on held-out data rather than assumed. Format consistency across demonstrations and the test query is important, and adding more examples is not always better once the prompt becomes long or noisy.

  • Reordering identical demonstrations can swing accuracy from strong to near-random (Lu et al., 2022).
  • Good orderings do not transfer between models, so ordering must be tuned per model.
  • Min et al. (2022) found correct labels matter less than the label space, input distribution, and format.
  • Choosing demonstrations similar to the test input often beats random selection.

Few-shot prompting versus fine-tuning

Few-shot prompting and fine-tuning are two ways to specialize a model, and the core difference is whether parameters change. Few-shot prompting performs no gradient update: the task is conveyed entirely through context at inference time, so the same deployed model serves every task. Fine-tuning performs gradient updates on labeled data, producing new weights specialized to one task or domain. The GPT-3 paper's contribution was showing that for many tasks the prompting route reaches useful quality without the labeled dataset and training run that fine-tuning needs.

The tradeoffs are concrete. Few-shot prompting needs no training infrastructure and adapts instantly to a new task by editing text, but it consumes context window on every request, can be sensitive to example choice and order, and is capped by what the base model already knows. Fine-tuning costs data and compute up front and yields a separate model to host, but it removes per-request demonstration tokens, can encode patterns too large to fit in a prompt, and often reaches higher accuracy on a narrow task. Parameter-efficient methods such as LoRA reduce that cost by training a small set of added weights instead of the full model.

In practice the two are complementary. Few-shot prompting is the fast path for prototyping and for tasks that change often, while fine-tuning suits stable, high-volume tasks where per-request token cost and consistency justify a training investment. Many systems start with few-shot prompts and graduate the proven, frequent tasks to fine-tuned models.

  • Few-shot prompting changes no weights; fine-tuning updates parameters via gradient descent.
  • Prompting adapts instantly and needs no training run but spends context tokens every request.
  • Fine-tuning costs data and compute up front but removes demonstration tokens and can raise accuracy.
  • Parameter-efficient tuning like LoRA narrows the cost gap by training only small added weights.

Strengths, limits, and when to use it

Few-shot learning is valuable wherever labeled data is scarce or expensive. In the classical setting that means recognizing rare classes, such as new species, defect types, or characters from a handful of examples, where collecting large datasets is impractical. In the LLM setting it means standing up a working task in minutes by writing a few examples, which is well suited to rapid iteration and to long-tail tasks that do not justify a fine-tuning pipeline.

The limits are real. Classical few-shot methods can degrade when test classes differ sharply from the meta-training distribution, since the learned embedding or initialization may not transfer. In-context few-shot inherits the base model's knowledge ceiling and its sensitivity to prompt construction, and it cannot reliably teach genuinely new facts that the model never saw during pretraining. Both senses can be brittle when the few available examples are unrepresentative of the broader task.

A reasonable decision rule: try zero-shot first, since if it already works the demonstrations are wasted tokens. If the task needs a shown format, an unusual label set, or pattern that is hard to state in words, add a few well-chosen, consistently formatted demonstrations and validate the choice and order on held-out inputs. If the task is high volume, stable, and demands accuracy or low latency beyond what prompting reaches, move to fine-tuning.

  • Best when labels are scarce, the task is easy to show but hard to describe, or iteration must be fast.
  • Classical methods can fail when test classes drift from the meta-training distribution.
  • In-context few-shot is capped by the base model's knowledge and is sensitive to prompt construction.
  • Decision rule: zero-shot first, then few-shot with validated examples, then fine-tune for stable high-volume tasks.

Key takeaways

  • Few-shot learning generalizes a task or class from only a handful of labeled examples.
  • The classical N-way K-shot setup gives N new classes with K examples each and is trained by meta-learning.
  • Metric methods (Matching, Prototypical Networks) and optimization methods (MAML) are the main classical families.
  • GPT-3 reframed few-shot as in-context learning: demonstrations in the prompt, no weight updates.
  • Demonstration choice and order strongly affect in-context accuracy and must be validated empirically.
  • Versus fine-tuning, few-shot prompting changes no parameters but spends context tokens each request.

Frequently asked questions

It is learning to do a new task, or recognize a new category, from just a few labeled examples instead of thousands. In large language models it usually means writing a few input-output examples into the prompt so the model copies the pattern, without retraining the model.
The difference is the number of demonstrations provided. Zero-shot gives only a task description with no examples, one-shot gives a single example, and few-shot gives several. The GPT-3 paper introduced these three regimes for prompting a frozen language model.
It is the standard evaluation setup in classical few-shot classification. An episode has N new classes with K labeled examples per class, so the support set holds N times K examples, and the model must classify separate query examples from those same N classes. A 5-way 1-shot task means five new classes with one example each.
No. Few-shot prompting performs no gradient update, the task is conveyed entirely through examples in the context at inference time. Fine-tuning updates the model's weights on labeled data, producing a new specialized model. Prompting adapts instantly but uses context tokens per request, while fine-tuning costs training up front but can reach higher accuracy.
Yes, often a lot. Lu et al. (2022) showed that reordering the same demonstrations can shift accuracy from near state of the art to near random guessing, and good orderings do not transfer between models. Example selection and consistent formatting also matter, so few-shot prompts should be validated on held-out inputs.
Surprisingly, they matter less than expected. Min et al. (2022) found that randomly corrupting demonstration labels barely hurt performance, and that the label space, input distribution, and overall format carried most of the signal. Demonstrations mainly teach the model what kind of answer to produce and in what shape, but using correct labels is still the safe default.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free