Large Language Model (LLM)

What is a large language model?

A large language model (LLM) is a neural network trained on very large quantities of text to predict and generate language. In practice, the term refers to transformer-based models with billions of parameters trained on hundreds of billions to trillions of tokens of text, a scale at which the model captures factual knowledge, grammar, discourse structure, reasoning patterns, and style well enough to produce fluent, context-appropriate output.

An LLM does not store sentences and look them up. It learns a probability distribution over sequences of tokens, then uses that distribution to continue, summarize, translate, classify, or answer prompts. The word 'large' is doing real work: the same architecture trained at small scale behaves very differently from one trained at the scale of modern systems, which is why capability and scale are tightly linked in this field.

• Architecture: almost always a transformer, usually decoder-only.
• Training signal: predict the next token in real text.
• Scale: billions of parameters, trillions of training tokens.
• Output: a probability distribution over the next token, sampled to generate text.

How LLMs work: next-token prediction

At inference time, an LLM is an autoregressive next-token predictor. The input prompt is split into tokens (subword units), each token is mapped to a vector embedding, and the model processes the sequence to produce a probability distribution over every possible next token in its vocabulary. The model then selects a token (by choosing the most likely one or sampling from the distribution), appends it to the sequence, and repeats. Generation is therefore one token at a time, with each new token conditioned on everything produced so far.

Training turns ordinary text into a self-supervised learning problem: for each position in a document, the correct label is simply the token that actually comes next. No human annotation is required for this stage, which is why an internet-scale text corpus can be used directly. Over trillions of such predictions, the model adjusts its parameters to assign higher probability to plausible continuations, and in doing so it internalizes a great deal about how language and the world described in that text are structured.

The transformer foundation and self-attention

The architecture underlying virtually all modern LLMs is the transformer, introduced in the 2017 paper 'Attention Is All You Need' by Vaswani and colleagues. Its central mechanism is self-attention, which lets each position in a sequence weigh and incorporate information from other positions, so a token's representation can depend on relevant context anywhere in the input. Self-attention replaced the sequential recurrence of earlier RNN-based models, making the computation highly parallelizable on modern hardware and enabling the much larger models that the field now relies on. Most generative LLMs use a decoder-only transformer with masked (causal) attention, meaning each position can attend only to earlier positions, which is exactly what next-token prediction requires.

Training pipeline: pretraining, fine-tuning, and alignment

Modern LLMs are typically built in stages. Pretraining is the largest and most expensive stage: the model learns next-token prediction over a broad text corpus and acquires general linguistic and world knowledge. The result is a capable but raw 'base model' that continues text rather than following instructions.

Subsequent stages adapt the base model to be useful and safe. Instruction tuning (a form of supervised fine-tuning) trains the model on examples of instructions paired with high-quality responses. Alignment via reinforcement learning from human feedback (RLHF), applied to this purpose in OpenAI's 2022 InstructGPT work, trains a reward model from human preference comparisons and then optimizes the model against it. Notably, in human evaluations on its prompt distribution, labelers preferred outputs from a 1.3B-parameter InstructGPT model over the 175B-parameter GPT-3 base model, evidence that alignment can matter more than raw size for usefulness.

• Pretraining: self-supervised next-token prediction on large corpora.
• Instruction / supervised fine-tuning: learn to follow instructions.
• RLHF or preference optimization: align outputs with human preferences for helpfulness and safety.

Parameters, scale, and emergent capabilities

An LLM's parameters are the learned weights of the network; parameter count is a rough proxy for capacity. As of 2026, models range from a few hundred million parameters up to hundreds of billions or more, trained on trillions of tokens, and many frontier systems use mixture-of-experts designs that activate only a fraction of their total parameters per token. Scaling up generally improves performance in a fairly predictable way, and laboratories use scaling laws to estimate the returns from more data and compute.

Some capabilities, however, appear only above a certain scale and are hard to predict by extrapolating from smaller models. The 2022 paper 'Emergent Abilities of Large Language Models' (Wei et al.) defined an ability as emergent if it is absent in smaller models but present in larger ones. The framing remains debated: a 2023 analysis by Schaeffer et al., 'Are Emergent Abilities of Large Language Models a Mirage?', argued that some apparent jumps are partly artifacts of nonlinear or discontinuous evaluation metrics rather than true phase changes. Either way, the practical link between scale and capability is one reason the field continues to invest heavily in larger training runs.

What LLMs are good at, and where they fail

LLMs excel at open-ended language tasks: drafting and editing text, summarization, translation, classification, code generation, question answering, and few-shot adaptation from examples given in the prompt. Their fluency and breadth come from the same statistical training that creates their limitations.

A central failure mode is hallucination: because the model optimizes for plausible continuations rather than verified truth, it can produce confident, fluent statements that are factually wrong or cite sources that do not exist. LLMs are also bounded by a fixed context window, and effective use of that window can degrade well before the stated limit, a phenomenon often called context rot, in which accuracy drops as input grows. Finally, a base LLM has no persistent memory between independent sessions: by default it does not retain anything that is not in the current context.

• Strengths: generation, summarization, translation, coding, in-context (few-shot) learning.
• Hallucination: fluent output is not the same as factually correct output.
• Context limits: a finite window, with quality often degrading before the limit.
• No persistent memory: nothing carries over between sessions unless re-supplied.

LLMs vs. NLP, generative AI, and traditional language models

These terms overlap but are not interchangeable. Natural language processing (NLP) is the broad field of using computers to work with human language; it long predates LLMs and includes rule-based and statistical methods such as parsing and named-entity recognition. A language model specifically assigns probabilities to sequences of words or tokens; classic n-gram models are language models too, but small and shallow. An LLM is a language model scaled up on the transformer architecture.

Generative AI is a wider umbrella covering any model that produces new content, including images, audio, and video, not only text. LLMs are the text-and-code branch of generative AI. In short: LLMs are a subset of language models, language modeling is one task within NLP, and text-generating LLMs are one part of generative AI.

• NLP: the broad field of computational language understanding and generation.
• Language model: any model that assigns probabilities to token sequences (including small n-gram models).
• LLM: a transformer-based language model at large scale.
• Generative AI: the broader category spanning text, image, audio, and video generation.

LLMs and memory: why a context window is not long-term memory

A context window is the span of tokens an LLM can attend to in a single request, covering the prompt plus the generated output. It is working memory, not long-term storage: once a conversation exceeds the window, or once a new independent session begins, earlier content is gone unless it is re-supplied. Even within the window, attention often concentrates on the beginning and end of the input, so simply enlarging the window does not guarantee reliable recall of everything inside it.

Durable memory across sessions is therefore an engineering layer built around the model, commonly retrieval-augmented generation (RAG), in which relevant stored information is fetched and inserted into the prompt at query time. AI memory and second-brain applications such as MemX apply this idea to personal knowledge, retrieving relevant stored items with semantic search so they can be supplied to a model on demand.