Language modeling is the task of learning a probability distribution over sequences of tokens, typically by predicting the next token given the preceding context. This single objective is the statistical foundation on which every large language model is built.
What Is a Language Model?
A language model is a system that assigns a probability to a sequence of tokens (words, sub-words, or characters). Formally, given a sequence x = (x1, x2, ..., xt), a language model estimates the joint probability P(x1, x2, ..., xt). Using the chain rule of probability, this joint distribution is factorized into a product of conditional probabilities, where each token's probability depends only on the tokens that came before it.
Because the chain rule decomposes the whole sequence into a series of next-token predictions, learning a language model reduces to learning one conditional distribution: the probability of the next token given the preceding context. A model that can accurately predict the next token can, by repeated application, assign a probability to any sequence and generate new sequences.
This framing is statistical rather than linguistic. The model does not encode grammar rules by hand. Instead it learns, from large amounts of text, which continuations are likely. The idea traces back to Claude Shannon's 1948 information-theoretic treatment of English as a stochastic process.
- Input: a context of preceding tokens.
- Output: a probability distribution over the entire vocabulary for the next token.
- Joint sequence probability is the product of these per-step conditional probabilities (chain rule).
The Next-Token Prediction Objective
The core training objective is next-token prediction. At each position the model produces a vector of scores (logits) over its vocabulary, and a softmax function converts those scores into a proper probability distribution that sums to one. Training adjusts the model's parameters to maximize the probability assigned to the actual next token in the training data, which is equivalent to minimizing the cross-entropy (negative log-likelihood) loss.
This is a form of self-supervised learning: no human labels are required, because the next token in any text already serves as the correct answer. Any corpus of raw text is therefore a free source of billions of training examples. This property is what allowed language modeling to scale to internet-sized datasets.
At inference time the model uses the same distribution to generate text. It can take the most likely token (greedy decoding), sample from the distribution, or use strategies such as temperature scaling, top-k, or nucleus (top-p) sampling to balance coherence against diversity.
- Objective: maximize log-likelihood of the observed next token (minimize cross-entropy).
- Self-supervised: the data labels itself, so no manual annotation is needed.
- Decoding strategies (greedy, sampling, top-p) shape the generated output from the same learned distribution.
Autoregressive vs. Masked Language Modeling
There are two dominant variants of the language modeling objective. Autoregressive (or causal) language modeling predicts each token using only the tokens to its left, processing text strictly left to right. This is the objective behind the GPT family and most generative models, and it is naturally suited to text generation because generation also proceeds left to right.
Masked language modeling (MLM), introduced at scale by BERT in 2018, instead hides a random subset of tokens and trains the model to reconstruct them using context from both directions. This bidirectional view produces richer representations for understanding tasks such as classification, named-entity recognition, and question answering, but a vanilla masked model is not designed to generate fluent text one token at a time.
In short, autoregressive models excel at natural language generation while masked models excel at natural language understanding. Most systems marketed today as large language models use the autoregressive objective.
- Autoregressive / causal (GPT-style): left-to-right, strong at generation.
- Masked (BERT-style): bidirectional context, strong at understanding and representation.
- Both are language modeling; they differ in which tokens the model is allowed to see.
From N-Grams to Neural to Transformer Models
Early language models were n-gram models: they estimated the probability of the next word from counts of how often short word sequences (n words long) appeared in a corpus. N-grams are simple and fast but suffer from data sparsity and the curse of dimensionality, because most plausible word sequences are never observed in training and longer contexts become impossible to count reliably.
A turning point came with Bengio et al. (2003), 'A Neural Probabilistic Language Model,' published in the Journal of Machine Learning Research. It used a feed-forward neural network to jointly learn distributed word representations (embeddings) and the probability function, letting the model generalize to unseen sequences by placing similar words near each other in vector space. Recurrent neural networks and LSTMs later extended this to longer contexts.
The decisive architectural shift was the Transformer, introduced in 'Attention Is All You Need' (Vaswani et al., 2017). By replacing recurrence with self-attention, Transformers process all positions in parallel and capture long-range dependencies efficiently. Nearly every modern large language model is a Transformer trained with a language modeling objective.
- N-gram: count-based, limited context, severe sparsity.
- Neural (Bengio 2003, then RNN/LSTM): learned embeddings, better generalization.
- Transformer (2017): self-attention, parallel training, the basis of today's LLMs.
Perplexity and How Models Are Evaluated
The standard intrinsic metric for a language model is perplexity. Perplexity is defined as the exponentiated average negative log-likelihood the model assigns to a held-out test sequence. Intuitively it measures how surprised the model is by real text: a perplexity of N means the model is, on average, as uncertain as if it were choosing uniformly among N equally likely options at each step.
Lower perplexity indicates a better fit to the data. Because perplexity depends on the tokenizer and the specific test set, comparisons are only meaningful when those are held constant. The Hugging Face documentation notes that perplexity is the exponentiated average negative log-likelihood and is equivalent to the exponentiation of the cross-entropy between the data and the model's predictions.
Perplexity measures predictive fit, not usefulness. Modern models are therefore also judged on downstream benchmarks (reasoning, coding, knowledge, safety) and increasingly on human or model-based preference evaluations, since a low-perplexity model is not automatically a helpful or truthful one.
- Perplexity = exp(average negative log-likelihood) on held-out text.
- Lower is better, but it is comparable only with the same tokenizer and test set.
- Benchmarks and preference evaluations complement perplexity for real-world quality.
How Language Modeling Scales Into LLMs
Large language models are language models scaled up in three coordinated dimensions: the number of parameters, the size of the training dataset, and the amount of compute used for training. Kaplan et al. (2020), 'Scaling Laws for Neural Language Models,' found that test loss falls as a smooth power law in each of these factors across many orders of magnitude, with architectural details like depth and width mattering far less than scale.
These scaling laws turned language modeling into an engineering discipline: given a compute budget, practitioners can predict roughly how large a model and dataset should be. Later work on compute-optimal training (Hoffmann et al., 2022, the Chinchilla paper) refined the recommended balance between model size and data, but the central finding that scale drives capability has held.
As models scaled, they began to exhibit broad capabilities (translation, summarization, coding, reasoning) that were never explicitly trained, simply by predicting the next token well across diverse text. These emergent behaviors are why a single, simple objective produced such general-purpose systems.
- Scale = parameters + data + compute, governed by power-law scaling laws.
- Scaling laws let teams predict performance before training (Kaplan et al., 2020).
- Broad capabilities emerge from next-token prediction at sufficient scale.
Language Model vs. LLM vs. Generative AI
These terms are related but not interchangeable. A language model is any model of the probability of token sequences, including a small n-gram table. A large language model (LLM) is specifically a very large, Transformer-based language model trained on broad text, typically with billions of parameters. Every LLM is a language model, but not every language model is large.
Generative AI is the broadest term and is not limited to text. It covers any system that produces new content, including image, audio, video, and code generators built on diffusion models, GANs, or other architectures that are not language models at all.
An aligned chat assistant is also more than a raw language model: it is usually a pre-trained LLM that has been further tuned with instruction tuning and reinforcement learning from human feedback. The base language modeling objective is the foundation, not the finished product.
- Language model: any probability model over token sequences (including n-grams).
- LLM: a large Transformer language model, a strict subset.
- Generative AI: any content-generating system, including non-text and non-language-model methods.
Limitations: The Objective Does Not Guarantee Truth or Memory
The language modeling objective optimizes for plausible continuations, not for factual accuracy. A model can assign high probability to a fluent but false statement, which is the mechanism behind hallucination. Predicting likely text and stating true facts are different goals, and the standard training loss does not distinguish them.
Language models also have no persistent memory by default. Knowledge is frozen into the weights at training time, and anything the model 'remembers' within a conversation lives only inside its finite context window. Once that window is exceeded, earlier information is lost unless it is re-supplied. This is why retrieval-augmented generation and external memory systems are used to ground models in current, user-specific information. AI memory tools such as MemX store a user's documents, photos, and voice notes externally and retrieve them on demand, providing the durable recall a raw language model lacks.
Other well-documented limitations include sensitivity to prompt phrasing, reproduction of biases present in training data, and difficulty with precise arithmetic or strict logical constraints. Understanding these limits follows directly from understanding the objective: the model was trained to predict text, nothing more.
- Fluency is optimized, not truth, which is the root of hallucination.
- No built-in long-term memory; knowledge is frozen and context windows are finite.
- Biases and prompt sensitivity stem from the data and the predictive objective itself.
Key takeaways
- Language modeling is the task of assigning probabilities to token sequences, reduced via the chain rule to predicting the next token given prior context.
- Training maximizes the log-likelihood of the true next token (minimizing cross-entropy), a self-supervised objective that needs no human labels and scales to internet-sized data.
- Autoregressive models (GPT-style) generate left to right; masked models (BERT-style) use bidirectional context for understanding tasks.
- The field moved from count-based n-grams to neural models (Bengio 2003) to Transformers (2017), and scaling laws (Kaplan 2020) showed that scale drives capability.
- Perplexity measures predictive fit, but a low-perplexity model is not guaranteed to be truthful, and a raw language model has no persistent memory beyond its context window.
Frequently asked questions
Related terms
Sources
- Bengio et al. (2003), A Neural Probabilistic Language Model (JMLR)
- Vaswani et al. (2017), Attention Is All You Need (arXiv:1706.03762)
- Kaplan et al. (2020), Scaling Laws for Neural Language Models (arXiv:2001.08361)
- Perplexity of fixed-length models (Hugging Face Transformers docs)
- Language Modeling (Lena Voita, NLP Course for You)
Put the idea into practice
MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.
Try MemX Free