AI Foundations

GPT and Decoder-Only Models

A decoder-only model is a Transformer that uses causal (masked) self-attention to predict the next token from left-to-right context alone. GPT is the canonical example of this autoregressive architecture, which became the dominant design for generative language models.

What a decoder-only model is

A decoder-only model is a Transformer built entirely from a stack of identical blocks, where each block contains causal (masked) multi-head self-attention, a position-wise feed-forward network, residual connections, and layer normalization. The defining property is the direction of information flow: every token can attend only to itself and to tokens that came before it, never to tokens that come after. This left-to-right constraint is what makes the model autoregressive, meaning it generates one token at a time, each one conditioned on the sequence produced so far.

The name comes from the original 2017 Transformer, which had two halves: an encoder that read the input with full bidirectional attention, and a decoder that produced the output one token at a time using masked attention. Decoder-only models keep just the second half and remove the cross-attention to a separate encoder. The input prompt and the generated continuation live in the same single stream of tokens, so there is no architectural distinction between reading and writing.

This is the architecture behind the GPT family and most large language models in wide use today. Its appeal is that one simple, uniform objective, predicting the next token, is enough to train a model that can later answer questions, translate, summarize, and write code, with no task-specific output heads.

Built from repeated blocks of causal self-attention plus a feed-forward network.
Each token attends only to itself and earlier tokens, never to future ones.
Prompt and generated text share a single token stream with no separate encoder.
Trained on one objective: predict the next token given the preceding context.

Causal masking: how the future is hidden

Self-attention normally lets every position look at every other position. To keep a model autoregressive, decoder-only Transformers add a causal mask that blocks each position from seeing anything to its right. In practice the attention scores for those forbidden positions are set to negative infinity before the softmax, so after normalization their weights become zero. The original Transformer paper describes implementing this masking inside scaled dot-product attention by masking out, setting to negative infinity, all values in the input of the softmax that correspond to illegal connections.

This single trick is what allows the entire sequence to be trained in parallel. During training the model sees the whole target sequence at once, but the mask guarantees that the prediction at each position depends only on earlier positions, exactly as it would during one-token-at-a-time generation. Without the mask, the model could trivially cheat by reading the answer it is meant to predict.

The mask is a fixed lower-triangular pattern that does not change during training; only the attention scores it gates are learned. This is why the same model behaves consistently whether it is processing a long prompt all at once or emitting the next token in a chat reply.

p(x) = ∏_{t=1}^{n} p(x_t | x_1, …, x_{t-1})

Autoregressive factorization: the probability of a sequence is the product of next-token probabilities, each conditioned only on the tokens before it.

Attention(Q, K, V) = softmax( (Q Kᵀ / √d_k) + M ) V,   where M_{ij} = 0 if j ≤ i else −∞

Causal-masked scaled dot-product attention. The mask M adds negative infinity to scores for future positions (j > i), so they receive zero weight after softmax.

A causal mask zeroes out attention to all future positions in each layer.
Forbidden scores are set to negative infinity before the softmax, giving zero weight.
Masking lets the full sequence train in parallel while staying autoregressive.
The mask is a fixed lower-triangular pattern, not a learned parameter.

Training objective: next-token prediction

Decoder-only models are trained with a left-to-right language modeling objective: given a stretch of text, predict the next token at every position, and minimize the cross-entropy between the predicted distribution and the actual next token. Because the causal mask makes every position a valid prediction target, a single sequence of length n yields n training signals at once, which makes the objective remarkably efficient to scale.

This objective requires no human labels. Any corpus of text is simultaneously the input and the supervision, which is why these models are described as self-supervised or, in the GPT-1 paper's terms, generatively pre-trained. GPT-1 in 2018 demonstrated that pre-training a decoder-only Transformer this way on unlabeled text, then fine-tuning on specific tasks, produced large gains across natural language understanding benchmarks.

The same pre-training objective scales smoothly to far larger models and datasets. GPT-2 in 2019 showed that a sufficiently large model trained only to predict the next token could perform many tasks zero-shot, and GPT-3 in 2020 showed that scaling further enabled in-context learning, where the model picks up a task from examples placed in the prompt with no gradient updates at all.

The loss is cross-entropy on next-token prediction at every position.
Training is self-supervised: raw text serves as both input and label.
One length-n sequence produces n prediction targets, aiding scaling.
The objective stays identical from small models up to the largest GPT-scale ones.

Three Transformer shapes: decoder, encoder, encoder-decoder

Transformers come in three broad shapes that differ in their attention pattern. Decoder-only models like GPT use causal attention and are tuned for generation. Encoder-only models like BERT use full bidirectional attention, letting every token see the entire input at once, which is well suited to understanding tasks such as classification, named-entity recognition, and extractive question answering, but not to open-ended generation.

Encoder-decoder models like T5 combine both: an encoder reads the input bidirectionally, and a separate decoder generates the output autoregressively while cross-attending to the encoder's representation. This sequence-to-sequence design maps naturally onto tasks with a clear input-output split, such as translation or summarization, and was the form of the original 2017 Transformer.

The trade-off is generality versus specialization. Bidirectional encoders build richer representations of a fixed input but cannot generate text token by token. Encoder-decoder models add machinery and a hard boundary between input and output. Decoder-only models collapse everything into one causal stream, which turned out to be the most flexible foundation for general-purpose generative models.

Decoder-only (GPT): causal attention, autoregressive generation.
Encoder-only (BERT): bidirectional attention, strong for understanding tasks.
Encoder-decoder (T5): bidirectional read plus autoregressive write via cross-attention.
Decoder-only unifies prompt and output into a single causal token stream.

Why decoder-only won for generative LLMs

Several factors pushed the field toward decoder-only architectures for general-purpose language models. Simplicity is one: a uniform stack with a single objective is easy to scale and reason about, with no encoder-decoder boundary to design around. Decoder-only models also exhibit in-context learning cleanly, because any task description and examples can be placed in the same prompt the model is already conditioned on, with no architectural change.

Inference efficiency is another major reason. Because each token attends only to earlier tokens, the keys and values computed for previous tokens never change as generation proceeds. They can be stored in a KV cache and reused, so generating the next token costs roughly one forward pass over a single new position rather than recomputing attention over the whole sequence. This makes left-to-right generation practical even for very long outputs.

Finally, the pre-training objective and the deployment behavior match exactly. The model is trained to continue text and is used to continue text, so the same masked, autoregressive mechanism serves both phases. Combined with favorable scaling behavior, where larger decoder-only models trained on more data reliably improve, these properties made the GPT-style architecture the default choice for modern large language models.

A single uniform stack and objective scale and reason about more easily.
In-context learning falls out naturally: the task lives in the same prompt.
Causal attention makes the KV cache valid, so generation reuses past computation.
Training and inference use the identical autoregressive mechanism end to end.

How generation works token by token

Generation runs the model forward repeatedly. The prompt is encoded in one pass, producing a probability distribution over the vocabulary for the next token; a token is sampled or chosen, appended to the sequence, and fed back in to produce the following distribution. This loop continues until a stop token appears or a length limit is reached. Each step is a deterministic forward pass over the network followed by a sampling decision governed by settings like temperature.

The KV cache is what keeps this loop affordable. Rather than re-encoding the entire growing sequence at every step, the model stores the key and value vectors for all previous tokens and computes attention only for the single new token against that cache. Causal masking is precisely what makes this correct: since earlier tokens never attend to later ones, their cached keys and values are final and never need recomputation.

The code below shows a minimal greedy generation loop using a causal mask. It illustrates the structure, building a lower-triangular mask, scoring attention, and appending the argmax token, rather than a production implementation, which would add the KV cache, sampling, and batching.

python

import torch
import torch.nn.functional as F

def causal_self_attention(x, W_q, W_k, W_v):
    # x: (seq_len, d_model)
    q, k, v = x @ W_q, x @ W_k, x @ W_v
    d_k = q.shape[-1]
    scores = (q @ k.transpose(-2, -1)) / d_k ** 0.5  # (seq, seq)
    # lower-triangular mask: position i may not see j > i
    seq = x.shape[0]
    mask = torch.triu(torch.ones(seq, seq), diagonal=1).bool()
    scores = scores.masked_fill(mask, float('-inf'))  # block the future
    weights = F.softmax(scores, dim=-1)
    return weights @ v

def greedy_generate(prompt_ids, step_fn, max_new_tokens=20, eos_id=None):
    ids = list(prompt_ids)
    for _ in range(max_new_tokens):
        logits = step_fn(ids)          # (vocab,) for the last position
        next_id = int(logits.argmax())  # pick the most likely next token
        ids.append(next_id)
        if next_id == eos_id:
            break
    return ids

Causal-masked self-attention (future positions set to negative infinity before softmax) and a minimal greedy next-token generation loop.

Generation is a loop: predict a distribution, pick a token, append, repeat.
A stop token or length limit ends the loop.
The KV cache stores past keys and values so only the new token is computed.
Causal masking guarantees cached values stay valid as the sequence grows.

Key takeaways

A decoder-only model is a Transformer that uses causal (masked) self-attention so each token sees only itself and earlier tokens.
Causal masking sets future-position attention scores to negative infinity before the softmax, enforcing left-to-right, autoregressive behavior.
These models train on a single self-supervised objective: predict the next token, with cross-entropy loss at every position.
Encoder-only models (BERT) read bidirectionally for understanding; encoder-decoder models (T5) map input to output; decoder-only models (GPT) unify both for generation.
GPT-1 (2018), GPT-2 (2019), and GPT-3 (2020) showed this architecture scaling from generative pre-training to zero-shot ability to in-context learning.
Decoder-only won for generative LLMs because of simplicity, smooth scaling, in-context learning, and KV-cache-friendly autoregressive generation.

Frequently asked questions

Decoder-only is the architecture class: any Transformer that uses causal self-attention and generates text left to right. GPT is the best-known family of decoder-only models, introduced by OpenAI starting with GPT-1 in 2018. So GPT is a specific lineage of decoder-only models, not a separate kind of architecture.

The causal mask prevents each position from attending to tokens that come after it, which keeps the model autoregressive. This lets the full sequence be trained in parallel while ensuring every prediction depends only on earlier context, matching how the model later generates one token at a time. Without it, the model could trivially read the token it is supposed to predict.

BERT is an encoder-only model with full bidirectional attention, so every token sees the entire input at once, which is strong for understanding tasks like classification and extractive question answering. A decoder-only model uses causal attention and can only see leftward context, which makes it suited to generation. BERT cannot generate text token by token the way a decoder-only model does.

They use one simple, uniform stack and a single next-token objective that scales cleanly to very large models and datasets. In-context learning falls out naturally because the task and its examples live in the same prompt, and causal attention makes the KV cache valid, so autoregressive generation stays efficient. Training and inference also use the identical mechanism end to end.

It runs forward to produce a probability distribution over the next token, samples or selects one, appends it to the sequence, and repeats until a stop token or length limit. A KV cache stores the keys and values of earlier tokens so only the new token is computed at each step. Causal masking guarantees those cached values never need recomputation.

Because each token attends only to earlier tokens, the keys and values for past tokens are final once computed and never change as generation continues. Storing them in a KV cache lets the model attend the new token against cached state instead of re-encoding the whole sequence each step. This is what keeps long autoregressive generation affordable.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free