AI Foundations

Recurrent Neural Network (RNN)

A recurrent neural network (RNN) is a neural network that processes a sequence one element at a time while maintaining a hidden state that carries information from earlier steps forward. The recurrence lets the same weights reuse past context to inform the current output, which makes RNNs suited to text, speech, and other ordered data.

What a Recurrent Neural Network Is

A recurrent neural network is a class of neural network built to handle sequential data, where the order of inputs matters and earlier inputs influence the meaning of later ones. Unlike a feedforward network that treats each input independently, an RNN reads a sequence one step at a time and keeps a hidden state, a vector that summarizes everything seen so far. At each step the network combines the current input with the previous hidden state to produce a new hidden state, then optionally emits an output for that step.

The defining feature is the recurrent connection: the hidden state at one time step feeds back as an input to the next. This loop gives the network a form of short-term memory, allowing it to model dependencies across the sequence without needing a fixed-size input. The same set of weights is applied at every time step, a property called weight sharing, so an RNN can process sequences of any length with a fixed number of parameters.

RNNs were a dominant approach for language modeling, machine translation, speech recognition, and time-series prediction through the early and mid 2010s. They process input in a strictly sequential order, which matches the structure of language and audio but also limits how fast they can train, a constraint that later motivated the move to attention-based architectures.

Processes ordered sequences one element at a time rather than as a fixed-size whole.
Maintains a hidden state vector that carries context from earlier steps forward.
Reuses the same weights at every time step, so it handles variable-length input.
Suited to text, speech, and time-series data where order carries meaning.

The Recurrence Relation and Hidden State

The core computation of a simple RNN is a recurrence relation that updates the hidden state at each time step t. The new hidden state h_t is a nonlinear function of the previous hidden state h_{t-1} and the current input x_t. A typical formulation applies a weight matrix to each, adds a bias, and passes the sum through an activation function such as tanh.

Because h_t depends on h_{t-1}, which depended on h_{t-2}, the hidden state at any step is in principle a function of every input that came before it. This is how the network carries information across time. An output for each step can be produced from h_t through an additional output weight matrix, for example to predict the next token in a sequence.

The minimal forward step below shows the recurrence in code. The matrices W_h, W_x, and the bias b are learned during training and shared across all time steps. Running this cell in a loop over the elements of a sequence produces the full hidden-state trajectory.

h_t = tanh(W_h h_{t-1} + W_x x_t + b)

The simple RNN recurrence: the new hidden state h_t is a nonlinear function of the previous hidden state h_{t-1} and the current input x_t, using shared weights W_h and W_x and bias b.

python

import numpy as np

def rnn_cell(x_t, h_prev, W_x, W_h, b):
    # One forward step of a simple RNN cell.
    # x_t: current input, h_prev: previous hidden state.
    h_t = np.tanh(W_x @ x_t + W_h @ h_prev + b)
    return h_t

# Run the cell over a sequence to build the hidden-state trajectory.
h = np.zeros(4)                       # initial hidden state
for x_t in sequence:                  # sequence is a list of input vectors
    h = rnn_cell(x_t, h, W_x, W_h, b)

A minimal RNN cell forward step and the loop that applies it across a sequence, carrying the hidden state h from one step to the next.

h_t is computed from the previous hidden state and the current input at each step.
The nonlinearity, often tanh, lets the network model complex sequential patterns.
W_h, W_x, and b are learned and identical at every time step.
Looping the cell over a sequence yields a hidden state that encodes all prior inputs.

Backpropagation Through Time and the Gradient Problem

RNNs are trained with backpropagation through time (BPTT), a variant of standard backpropagation. The network is conceptually unrolled across the time steps of a sequence into a deep computational graph, and gradients of the loss are propagated backward through every step. Because the same weights appear at each step, their gradient contributions from all time steps are summed.

Unrolling over many steps creates a very deep chain of multiplications, and this is where simple RNNs break down. Bengio, Simard, and Frasconi showed in 1994, in IEEE Transactions on Neural Networks, that learning long-term dependencies with gradient descent is difficult. Sepp Hochreiter had analyzed the same effect. When the relevant factors are repeatedly multiplied, the gradient tends to shrink toward zero over many steps, the vanishing gradient problem, or grow without bound, the exploding gradient problem.

A vanishing gradient means the network cannot learn associations between events separated by many time steps, because the error signal fades before it reaches the early inputs. An exploding gradient destabilizes training. Exploding gradients can be controlled with gradient clipping, but the vanishing case is harder and limited what plain RNNs could learn over long ranges, which is the problem gated architectures were designed to solve.

BPTT unrolls the network across time and backpropagates the loss through every step.
Long unrolled chains cause gradients to vanish toward zero or explode toward infinity.
Bengio et al. (1994) formally showed gradient descent struggles with long-term dependencies.
Vanishing gradients block learning of distant associations; clipping tames exploding ones.

Gated Variants: LSTM and GRU

The Long Short-Term Memory (LSTM) network, introduced by Sepp Hochreiter and Jurgen Schmidhuber in Neural Computation in 1997, addressed the vanishing gradient problem directly. An LSTM adds a separate cell state that runs through the sequence with mostly linear updates, a path the original paper described as a constant error carousel through which error can flow without vanishing. Multiplicative gate units learn to control that flow.

The LSTM uses three gates, small neural layers with sigmoid outputs between 0 and 1 that act as soft switches. The forget gate decides what to remove from the cell state, the input gate decides what new information to write, and the output gate decides what to expose as the hidden state. By learning to keep the cell state stable when needed, an LSTM can bridge time lags of well over a thousand steps, which the original paper demonstrated.

The Gated Recurrent Unit (GRU), proposed by Kyunghyun Cho and colleagues in 2014, is a streamlined alternative. It merges the cell and hidden state and uses two gates, an update gate and a reset gate, giving it fewer parameters than an LSTM while often performing comparably. Both LSTM and GRU became the standard recurrent building blocks for sequence tasks before attention-based models took over.

LSTM (Hochreiter and Schmidhuber, 1997) adds a gated cell state to preserve gradient flow.
Forget, input, and output gates are sigmoid switches that read and write the cell state.
GRU (Cho et al., 2014) uses update and reset gates with fewer parameters than an LSTM.
Gating let recurrent networks learn dependencies across hundreds or thousands of steps.

Sequence-to-Sequence, Attention, and the Bottleneck

For tasks that map one sequence to another, such as translation, RNNs were arranged in an encoder-decoder, or sequence-to-sequence, structure. The encoder RNN reads the input sequence and compresses it into a single fixed-length context vector, and the decoder RNN generates the output sequence from that vector. This worked but had a clear weakness: forcing an entire input sentence through one fixed-size vector created an information bottleneck that hurt performance on long inputs.

Bahdanau, Cho, and Bengio introduced an attention mechanism in 2014 to remove this bottleneck. Instead of relying on one context vector, the decoder learns to look back over all of the encoder hidden states and form a weighted combination of them at each output step, focusing on the source positions most relevant to the word being produced. The attention weights also yield interpretable soft alignments between input and output.

Attention substantially improved encoder-decoder translation, especially on longer sentences, and it introduced the idea that a model could selectively read from a full set of representations rather than a single summary. That idea, applied without any recurrence at all, became the foundation of the architecture that followed.

Encoder-decoder RNNs compress an input sequence into one fixed-length context vector.
That single vector becomes a bottleneck that degrades quality on long inputs.
Bahdanau et al. (2014) added attention so the decoder reads all encoder states, weighted.
Attention weights act as interpretable soft alignments between source and target.

Why Transformers Replaced RNNs

In 2017, Vaswani and colleagues at Google published Attention Is All You Need (arXiv:1706.03762), introducing the Transformer. The paper proposed an architecture that relies entirely on attention mechanisms, dispensing with recurrence and convolutions. Without a recurrent loop, the model has no inherent notion of order, so it adds positional encodings to inform each token of its place in the sequence.

The decisive advantage is parallelism. An RNN must process a sequence step by step because each hidden state depends on the previous one, so computation cannot be parallelized along the sequence length. A Transformer computes attention over all positions at once, letting the whole sequence be processed in parallel during training. This made far larger models and datasets practical, and it produced higher translation quality with less training time than the recurrent and convolutional models that preceded it.

Transformers became the basis for most modern natural language processing, including large language models. RNNs and their gated variants remain useful in constrained or streaming settings where strict sequential, step-by-step processing or a small memory footprint matters, but for general sequence modeling at scale, the parallelizable Transformer displaced the recurrent approach.

The Transformer (Vaswani et al., 2017) removed recurrence and used attention alone.
Positional encodings supply order information that recurrence had provided implicitly.
RNNs are inherently sequential; Transformers process the whole sequence in parallel.
Parallelism enabled larger models and made Transformers the standard for modern NLP.

Key takeaways

An RNN processes sequences one step at a time, carrying a hidden state across time steps.
The recurrence h_t = tanh(W_h h_{t-1} + W_x x_t + b) reuses shared weights at every step.
Training uses backpropagation through time, which exposes vanishing and exploding gradients.
LSTM (1997) and GRU (2014) use gates to preserve gradient flow over long-range dependencies.
Attention (Bahdanau et al., 2014) removed the fixed-vector bottleneck in encoder-decoder RNNs.
The Transformer (2017) replaced RNNs for most NLP because it parallelizes over the sequence.

Frequently asked questions

It is a neural network that reads a sequence one item at a time and keeps a running summary, called a hidden state, of everything it has seen so far. At each step it combines the new item with that summary to update its understanding. This memory makes it well suited to ordered data like text, speech, and time series.

When an RNN is trained by backpropagation through time over a long sequence, the error signal is multiplied through many steps and tends to shrink toward zero before reaching the early inputs. This prevents the network from learning associations between events that are far apart in the sequence. Bengio, Simard, and Frasconi (1994) formally showed why gradient descent struggles with such long-term dependencies.

Both add gates, small sigmoid layers that act as soft switches controlling what information is kept, written, or read. An LSTM (Hochreiter and Schmidhuber, 1997) maintains a separate cell state with a near-linear update path that lets gradients flow without vanishing. A GRU (Cho et al., 2014) achieves a similar effect with two gates and fewer parameters.

An RNN processes a sequence sequentially, with each hidden state depending on the previous one, so it cannot parallelize along the sequence. A Transformer (Vaswani et al., 2017) uses attention to relate all positions at once, so the whole sequence is processed in parallel and order is supplied by positional encodings. That parallelism is why Transformers train faster and scale better, and why they replaced RNNs for most NLP.

Transformers have replaced RNNs for most large-scale natural language processing, including large language models. RNNs and gated variants like LSTM and GRU still appear in settings where strict step-by-step streaming processing or a small memory footprint is valued, such as some on-device and real-time applications. For general sequence modeling at scale, the Transformer is now the default.

Backpropagation through time (BPTT) is how RNNs are trained. The network is unrolled across the time steps of a sequence into a deep graph, and the loss is backpropagated through every step. Because the same weights are reused at each step, their gradients from all steps are summed before the weights are updated.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free