Models & Evaluation

Speculative Decoding

Speculative decoding is an inference-acceleration method that speeds up autoregressive LLM generation without changing the output distribution: a small fast draft model proposes several future tokens, the large target model verifies them in one parallel forward pass, and a rejection-sampling rule keeps the longest correct prefix.

What speculative decoding is

Speculative decoding is an inference-time optimization for autoregressive large language models. Standard decoding generates text one token at a time: producing K tokens requires K sequential forward passes through the full model, and each pass waits on the previous one. Speculative decoding breaks part of that sequential dependency by letting a small, fast draft model guess several upcoming tokens, then letting the large target model check all of those guesses at once. When the guesses are good, multiple tokens advance per target-model call instead of one.

The technique was introduced in two near-simultaneous 2022 to 2023 papers. Yaniv Leviathan, Matan Kalman, and Yossi Matias at Google Research published "Fast Inference from Transformers via Speculative Decoding" (submitted November 30, 2022, arXiv:2211.17192), and Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper at DeepMind published "Accelerating Large Language Model Decoding with Speculative Sampling" (submitted February 2, 2023, arXiv:2302.01318). Both report roughly 2x to 3x decoding speedups on large models with no change to model weights or architecture.

The defining property is that speculative decoding is exact. The text it produces is distributed identically to ordinary sampling from the target model alone, up to numerical precision. It is a latency optimization, not an approximation, so it does not trade away quality for speed. The draft model only affects how fast the answer arrives, never which answer is statistically possible.

A draft model proposes candidate tokens; a target model verifies them.
Multiple tokens can be accepted per target-model forward pass.
Output is provably distributed identically to sampling from the target alone.
It requires no retraining or architecture change to the target model.

Why parallel verification is nearly free

The speedup rests on a hardware fact: autoregressive decoding of a single token is memory-bandwidth-bound, not compute-bound. At batch size one, each step loads the entire weight matrix and key-value cache from memory and performs comparatively little arithmetic, so the accelerator sits idle waiting on memory transfers. A forward pass that scores one token and a forward pass that scores several tokens in a batch take roughly the same wall-clock time, because both are limited by how fast weights stream from memory rather than by how many multiply-adds run.

Speculative decoding exploits this gap. The draft model cheaply produces k candidate continuations, and the target model verifies all k positions in a single batched forward pass that costs about the same as one ordinary decoding step. If j of those k tokens are accepted, the system advanced j tokens for the price of one target call plus the cheaper draft calls. The draft model is typically far smaller than the target, so its sequential cost is a small fraction of a target step.

The net win therefore depends on two things: how often the draft model agrees with the target (its acceptance rate) and how cheap the draft model is relative to the target. A draft that is fast but frequently wrong wastes target verification on rejected tokens; a draft that is accurate but nearly as expensive as the target removes the cost advantage. The best results come from a draft that is both cheap and well-aligned with the target on the easy parts of the sequence.

T_spec  approx  (1/E[tau]) * (c_draft * k + c_target)

Per-accepted-token cost: one target verification of cost c_target plus k draft steps of cost c_draft, amortized over the expected number of accepted tokens E[tau]. Speedup over baseline is roughly E[tau] when c_draft is small relative to c_target.

Single-token decoding is bottlenecked by memory bandwidth, not arithmetic.
Verifying k tokens in parallel costs roughly the same as one normal step.
Speedup grows with the draft's acceptance rate and shrinks with its cost.
A fast but inaccurate draft can erase the benefit through wasted verification.

The acceptance rule that preserves the distribution

The exactness guarantee comes from a modified rejection sampling rule applied at each drafted position. Let p(x) be the target model's probability for token x at a given position and q(x) be the draft model's probability for that same token. After the draft proposes a token x sampled from q, the algorithm accepts x with probability min(1, p(x)/q(x)). If the target finds the token at least as likely as the draft did, it is always accepted; if the target finds it less likely, it is accepted in proportion to that ratio.

When a token is rejected, the algorithm does not simply discard the step. It resamples a replacement token from a corrected, normalized residual distribution proportional to max(0, p(x) - q(x)). This correction is what makes the procedure provably equivalent to sampling directly from p. The combined effect of accept-with-probability-min and resample-on-reject is that every token in the output has exactly the marginal distribution it would have under standard target-only sampling.

Verification proceeds left to right and stops at the first rejection. Every drafted token before the first mismatch is kept, the first mismatch is replaced by the corrected resample, and any drafted tokens after it are thrown away because they were conditioned on a now-invalid prefix. The target's parallel forward pass also yields a free bonus token from the position just past the last accepted draft token, so even an entirely rejected draft still advances at least one token per round.

accept x ~ q  with prob  min(1, p(x) / q(x)) ;  else resample x' ~ norm(max(0, p - q))

The acceptance and correction rule. Applied per position, it guarantees the kept token is distributed exactly as p, the target distribution, regardless of how good or bad the draft q is.

Each drafted token is accepted with probability min(1, p(x)/q(x)).
Rejected tokens are resampled from the normalized residual max(0, p - q).
Verification keeps the longest correct prefix and corrects the first mismatch.
A free bonus token from the target makes worst-case progress at least one per round.

A minimal sketch of the loop

In pseudocode the outer loop repeats two phases until generation ends. First the draft model autoregressively extends the current context by k tokens, recording the draft probability q for each. Then the target model runs one parallel forward pass over those k positions to obtain the target probabilities p at every position, including the position one step beyond the last draft token.

The acceptance walk then compares p and q position by position using the rule above. It accepts a contiguous prefix, halts at the first rejection (resampling a corrected token there), and uses the target's extra distribution to emit a bonus token if the whole draft was accepted. The accepted tokens are appended to the context and the loop restarts. Because both models share the same tokenizer and the draft is conditioned on the same prefix, no realignment of token boundaries is needed.

Production frameworks implement richer variants of this loop. They cache key-value tensors across rounds so neither model recomputes the shared prefix, batch draft trees instead of a single linear draft, and tune k dynamically based on recent acceptance rates. The core accept-or-correct logic, however, remains the rejection-sampling rule that keeps the output exact.

python

def speculative_step(ctx, draft, target, k):
    # 1. Draft k tokens autoregressively, recording q.
    drafted, q = [], []
    cur = ctx
    for _ in range(k):
        qx = draft.next_token_probs(cur)
        tok = sample(qx)
        drafted.append(tok); q.append(qx)
        cur = cur + [tok]

    # 2. Verify all k positions in one parallel target pass.
    p = target.probs_for_positions(ctx, drafted)  # length k + 1

    # 3. Accept the longest correct prefix; correct the first miss.
    out = []
    for i, tok in enumerate(drafted):
        if random() < min(1.0, p[i][tok] / q[i][tok]):
            out.append(tok)                       # accepted
        else:
            out.append(sample(residual(p[i], q[i])))  # corrected resample
            return ctx + out                      # stop at first reject
    out.append(sample(p[k]))                      # free bonus token
    return ctx + out

One round of speculative decoding. residual(p, q) normalizes max(0, p - q). The returned tokens are distributed exactly as target-only sampling.

Phase one drafts k tokens; phase two verifies them in a single target pass.
Acceptance is a left-to-right walk that stops at the first rejected token.
Key-value caching avoids recomputing the shared prefix each round.
Tuning k to the observed acceptance rate balances draft cost against hit rate.

Variants and how they differ

The original formulation uses a separate, smaller pretrained model as the draft. That works but forces practitioners to find and serve a second model whose tokenizer and behavior match the target. A family of follow-up methods removes or restructures that requirement. Medusa ("Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads", arXiv:2401.10774) attaches several extra lightweight decoding heads to the target itself so it predicts multiple future tokens in one pass, then verifies candidate continuations with tree attention, reporting about 2.2x to 3.6x speedups.

EAGLE ("EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty", arXiv:2401.15077) drafts at the level of the model's second-to-top-layer feature vectors rather than tokens, using a small autoregressive head plus the target's frozen output layer, and reports being faster than both Medusa and lookahead decoding on MT-bench. Lookahead decoding ("Break the Sequential Dependency of LLM Inference Using Lookahead Decoding", arXiv:2402.02057) is draft-model-free: it uses Jacobi-style parallel iteration to generate and cache n-grams from the target's own trajectory, then verifies them, needing no auxiliary model.

The simplest variant of all is prompt lookup, or n-gram, decoding. It replaces the draft model with plain string matching against the prompt: when the model is about to repeat a phrase already present in the input, the matching span is proposed as the draft and verified normally. This costs almost nothing and helps a great deal on summarization, code editing, and retrieval-augmented generation where the output echoes the input, but it gives no benefit when the output shares little text with the prompt.

Medusa adds extra decoding heads to the target and verifies with tree attention.
EAGLE drafts at the feature level and reuses the target's frozen output head.
Lookahead decoding is draft-free, using Jacobi iteration to propose n-grams.
Prompt-lookup decoding drafts by matching n-grams from the prompt, free but input-dependent.

Practical considerations

Speculative decoding mainly helps low-batch, latency-sensitive serving, which is exactly where decoding is most memory-bandwidth-bound. At high batch sizes the accelerator is already busy with arithmetic from many concurrent requests, so the spare capacity that verification exploits shrinks and the technique can even hurt throughput. Serving stacks therefore often enable it selectively, for small batches or interactive single-user sessions.

Acceptance rate is the metric that governs the payoff, and it depends on alignment between draft and target. A draft distilled or trained on the target's outputs tends to agree more often than an unrelated small model. Because acceptance varies by content, easy boilerplate and repeated phrases accept readily while novel or high-entropy text does not, the realized speedup fluctuates across a generation and across prompts rather than being a fixed constant.

Open-source frameworks expose these methods directly. vLLM, TensorRT-LLM, and similar serving systems ship speculative decoding with options for a separate draft model, EAGLE-style heads, or n-gram prompt lookup, letting operators pick the variant that matches their model pair and traffic. Correct implementations preserve the exactness guarantee, so enabling speculative decoding should change latency and cost without changing the text a user receives at a given sampling seed.

The benefit is largest at small batch sizes and interactive latencies.
Acceptance rate, and thus speedup, varies by prompt and by content difficulty.
Drafts trained or distilled on the target accept more often than unrelated models.
vLLM and TensorRT-LLM ship draft-model, EAGLE, and n-gram variants out of the box.

Key takeaways

Speculative decoding speeds up autoregressive generation by drafting several tokens with a small model and verifying them in one parallel target pass.
It is exact: a rejection-sampling rule makes the output distributed identically to sampling from the target model alone.
It works because single-token decoding is memory-bandwidth-bound, so verifying k tokens in parallel costs about the same as one normal step.
Each drafted token is accepted with probability min(1, p(x)/q(x)) and rejected tokens are resampled from the normalized residual of target minus draft.
The realized speedup depends on the draft's acceptance rate and on how cheap the draft is relative to the target.
Variants include Medusa decoding heads, EAGLE feature-level drafting, draft-free lookahead decoding, and prompt-lookup n-gram drafting.

Frequently asked questions

No. The acceptance and correction rule is designed so the kept tokens follow exactly the target model's distribution, up to numerical precision. It is a latency optimization, not an approximation, so at a fixed sampling seed it produces the same statistical output as ordinary decoding from the target.

The founding papers report roughly 2x to 3x decoding speedups on large models such as T5-XXL and Chinchilla 70B. The exact gain depends on the draft model's acceptance rate and its cost relative to the target, so it varies by model pair, prompt, and content. Repetitive or boilerplate text accepts more drafted tokens and speeds up more than novel high-entropy text.

At batch size one, a decoding step spends most of its time streaming the model weights and key-value cache from memory rather than doing arithmetic. Adding more tokens to the same forward pass reuses those already-loaded weights, so a batched verification of several positions finishes in roughly the same wall-clock time as a single-token step.

A draft model is a small, fast model that proposes candidate tokens for the larger target model to verify. A good draft is both cheap to run and well-aligned with the target so its proposals are accepted often. Drafts that are distilled from or trained on the target's outputs typically achieve higher acceptance rates than unrelated small models with the same tokenizer.

Classic speculative decoding uses a separate draft model. Medusa instead adds extra decoding heads onto the target itself, and EAGLE drafts at the level of internal feature vectors using a small head plus the target's frozen output layer. All three still verify candidates with the target and preserve the output distribution; they differ mainly in how the draft is produced.

Usually not much. Large batches already keep the accelerator busy with arithmetic from many requests, so there is little idle capacity for parallel verification to exploit, and the technique can reduce throughput. It is most effective for small-batch, latency-sensitive serving such as single-user interactive sessions.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free