Transformer (Neural Network Architecture)

What is a transformer?

A transformer is a deep learning architecture that turns an input sequence (such as a sentence) into an output sequence by repeatedly applying an operation called self-attention. It was introduced in the paper "Attention Is All You Need," submitted to arXiv on June 12, 2017 by eight researchers at Google: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.

The central claim of that paper, and the source of its title, is that attention mechanisms alone are sufficient to model sequences, with no need for the recurrence or convolution that previous architectures relied on. The original model was an encoder-decoder built for machine translation, with roughly 100 million parameters (a 65-million-parameter base configuration and a 213-million-parameter large configuration). As of 2026 the paper is among the ten most-cited works of the 21st century, with more than 173,000 citations, because the architecture became the foundation for large language models (LLMs), vision models, and multimodal systems.

• Origin: "Attention Is All You Need," Vaswani et al., Google, June 2017.
• Core idea: replace recurrence and convolution entirely with self-attention.
• Original task: sequence-to-sequence machine translation.
• Legacy: the base architecture behind GPT, BERT, ViT, and most modern foundation models.

The self-attention mechanism, explained simply

Self-attention lets every token in a sequence look at every other token and decide how much each one matters for its own representation. Instead of reading a sentence strictly left to right, the model weighs all tokens against each other at once and builds a context-aware representation of each token.

The mechanism works through three learned projections of each token, commonly described with a library analogy. The Query (Q) represents what a token is looking for. The Key (K) represents what each token offers. The Value (V) is the actual content retrieved once a match is found. For a given token, the model compares its query against the keys of all tokens, producing attention scores; those scores are scaled and normalized into weights, then used to compute a weighted sum of the value vectors.

Concretely, the score between a query and a key is their dot product, divided by the square root of the key dimension to keep values numerically stable. This scaled dot-product attention is the core operation repeated throughout the network. Because it compares every token with every other token directly, self-attention can model relationships between distant tokens just as easily as adjacent ones.

• Query: what a token is searching for.
• Key: what each token advertises about itself.
• Value: the information returned when a query matches a key.
• Scores are dot products of queries and keys, scaled by the square root of the key dimension, then normalized into attention weights.

Core components of a transformer

A transformer block is built from a small set of repeating components. Understanding them clarifies how the architecture turns raw tokens into rich representations.

Token embeddings convert discrete tokens (words or subwords) into continuous vectors. Because self-attention has no inherent notion of order, the architecture adds positional encoding to those embeddings so the model knows where each token sits in the sequence. The original paper used fixed sine and cosine functions of different frequencies; many later models use learned positional embeddings or rotary methods instead.

Multi-head attention runs several attention operations in parallel, each with its own learned projections, so different heads can focus on different relationships (for example, syntax versus long-range reference). Their outputs are concatenated and projected back together. Each block also contains a position-wise feed-forward network, a small two-layer neural network applied to every token independently. Residual connections and layer normalization wrap these sublayers to stabilize training in deep stacks.

• Embeddings: map tokens to dense vectors.
• Positional encoding: inject word-order information (the original paper used sine and cosine functions).
• Multi-head attention: parallel attention heads capturing different relationship types.
• Feed-forward layers: per-token transformations applied after attention.
• Residual connections and layer normalization: stabilize deep networks.

Encoder, decoder, and decoder-only variants

The 2017 transformer had two stacks. The encoder reads the whole input and produces context-rich representations using bidirectional attention, where each token can attend to tokens on both sides. The decoder generates output one token at a time using masked (causal) self-attention so it can only see tokens already produced, plus cross-attention into the encoder's output. Later research split these into three influential families.

Encoder-only models, such as BERT, stack only encoders and read context in both directions. They excel at understanding tasks like classification, retrieval, and named-entity recognition. Decoder-only models, such as the GPT family, stack only decoders and use masked self-attention to predict the next token, which makes them naturally suited to text generation. Encoder-decoder models retain both stacks and remain strong for tasks like translation and summarization.

Most general-purpose LLMs as of 2026 are decoder-only, because next-token prediction over large corpora scales cleanly and supports open-ended generation, instruction following, and chat.

• Encoder-only (BERT-style): bidirectional, built for understanding and classification.
• Decoder-only (GPT-style): causal/masked attention, built for generation; dominant for modern LLMs.
• Encoder-decoder (original transformer): both stacks, strong for translation and summarization.

Why transformers replaced RNNs and LSTMs

Before transformers, sequence modeling relied on recurrent neural networks (RNNs) and their gated variant, long short-term memory (LSTM) networks. These process tokens one step at a time, carrying a hidden state forward. That sequential dependency has two costs: it limits parallelism during training, and it makes learning long-range dependencies hard because information must survive many sequential steps.

Transformers remove recurrence, so all tokens in a sequence are processed simultaneously. This parallelism maps efficiently onto modern GPU and TPU hardware and shortens training time for large datasets. Self-attention also gives every token a direct path to every other token, so the distance between two related tokens does not degrade the signal the way it does in a recurrent chain.

Those two properties, parallel computation and direct long-range connections, are why transformers displaced RNNs and LSTMs as the default for natural language processing and then for many other domains.

• RNNs and LSTMs are sequential, which limits parallelism and weakens long-range memory.
• Transformers process all tokens in parallel, exploiting GPU and TPU hardware.
• Self-attention provides direct connections between any pair of tokens regardless of distance.

Transformers beyond text: vision and multimodal models

The transformer is domain-agnostic: any input that can be turned into a sequence of vectors can be fed to it. The Vision Transformer (ViT), introduced in "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (Dosovitskiy et al., arXiv preprint October 2020, published at ICLR 2021), applies this idea to images. It splits an image into fixed-size patches, linearly projects each patch into a vector, adds positional information, and feeds the resulting sequence into a standard transformer encoder.

ViT showed that, given enough pre-training data, a pure transformer can match or exceed convolutional neural networks on image classification while often using fewer training resources. The same recipe extends to multimodal models that jointly process text, images, audio, or video by mapping each modality into a shared sequence of tokens, which is how modern systems answer questions about an image or transcribe and reason over audio.

• Vision Transformer (ViT): treats image patches as tokens for a transformer encoder.
• Multimodal models map text, images, audio, and video into a shared token space.
• The same architecture supports semantic search and retrieval across mixed media.

Scaling, compute cost, and known limitations

Transformers tend to improve as model size, data, and compute increase together, a regularity that motivated the move toward very large models. That scalability comes with real costs and constraints.

The most discussed limitation is that standard self-attention has time and memory complexity quadratic in sequence length, because every token attends to every other token. Doubling the context roughly quadruples the attention compute, which makes very long inputs expensive. During autoregressive generation, a key-value (KV) cache also grows with sequence length and consumes memory per layer and per token. Engineering responses include IO-aware exact attention such as FlashAttention (Dao et al., 2022), which reorders memory access to compute the same result faster and with less memory, plus various sparse and linear-attention approximations that trade some quality for efficiency.

Beyond compute, transformers have a finite context window, can produce fluent but incorrect output (hallucination), and require large datasets and energy to train. These remain active areas of research as of 2026.

• Self-attention cost is quadratic in sequence length, limiting practical context size.
• The KV cache grows with sequence length during generation, adding memory pressure.
• FlashAttention (2022) computes exact attention more efficiently via IO-aware tiling.
• Open limitations include finite context windows, hallucination, and high training cost.

Where transformers sit in the LLM stack

A modern LLM is essentially a large decoder-only transformer plus the machinery around it. The transformer provides the core mapping from a sequence of input tokens to a probability distribution over the next token. Surrounding layers turn that core into a usable system: a tokenizer converts text to tokens, the pre-trained transformer supplies broad knowledge, and post-training steps such as supervised fine-tuning and reinforcement learning from human feedback align behavior.

At inference, the same transformer weights drive techniques like prompting, retrieval-augmented generation, and tool use. Retrieval and memory systems sit outside the model, feeding relevant context into the transformer's context window at run time. Understanding the transformer therefore clarifies both what these systems can do and where their constraints, such as context length and attention cost, originate.

• The transformer is the core sequence model inside an LLM, not the entire system.
• Tokenization, pre-training, and alignment wrap the architecture into a usable model.
• Prompting, retrieval, and tool use operate on top of the same transformer weights.