Reinforcement Learning

What Is Reinforcement Learning?

Reinforcement learning (RL) is a branch of machine learning in which an agent learns to behave by interacting with an environment through trial and error. Rather than being shown the correct answer for each input, the agent takes actions, observes the consequences, and receives a scalar feedback signal called a reward. Over many interactions, it learns a strategy that maximizes the cumulative reward it expects to collect.

The standard framework formalizes this as a Markov decision process: at each step the agent observes the current state, chooses an action, transitions to a new state, and receives a reward. The agent's goal is to discover a policy, a mapping from states to actions, that maximizes long-term return rather than just the immediate reward. The canonical reference for the field is Richard Sutton and Andrew Barto's textbook Reinforcement Learning: An Introduction (2nd edition, 2018).

RL differs from other paradigms in two important ways. First, feedback is evaluative (how good was this action) rather than instructive (what was the correct action). Second, actions affect future states and future rewards, so the agent must reason about delayed consequences, a challenge known as the credit assignment problem.

Core Components: Agent, Environment, State, Action, Reward, Policy

Every RL problem is built from a small set of interacting components. Understanding these terms is the foundation for everything else in the field.

• Agent: the learner and decision-maker that selects actions.
• Environment: everything outside the agent that it interacts with and that responds to its actions.
• State: a representation of the current situation the agent observes at a given time step.
• Action: a choice the agent makes from the set of options available in a state.
• Reward: a scalar feedback signal returned by the environment after an action, indicating how desirable the outcome was.
• Policy: the agent's behavior function, mapping states to actions (or to probabilities over actions); learning a good policy is the central objective.
• Value function: an estimate of the expected long-term return from a state (or state-action pair), used to guide the policy toward high-reward regions.

The Exploration vs. Exploitation Tradeoff

A defining challenge in RL is the tension between exploration and exploitation. Exploitation means choosing the action currently believed to be best, in order to collect known reward. Exploration means trying less certain actions to gather information that might reveal better strategies. An agent that only exploits may lock into a mediocre policy, while one that only explores never capitalizes on what it has learned.

Practical algorithms balance the two with mechanisms such as epsilon-greedy action selection (act greedily most of the time but occasionally act at random), softmax or entropy-based sampling, and optimism under uncertainty. The right balance typically shifts over training, with more exploration early and more exploitation as the agent's value estimates become reliable.

Value-Based, Policy-Based, and Actor-Critic Methods

RL algorithms are commonly grouped by what they learn directly. The three broad families are value-based, policy-based, and actor-critic methods.

Value-based methods learn a value function (such as the action-value function Q) and derive a policy by acting greedily with respect to it. Policy-based methods parameterize and optimize the policy directly, which suits continuous or high-dimensional action spaces and naturally represents stochastic policies. Actor-critic methods combine both: an actor proposes actions while a critic estimates values to reduce the variance of the policy updates.

• Value-based: learn values, derive policy (for example, Q-learning, Deep Q-Networks).
• Policy-based: optimize the policy directly via policy gradients (for example, REINFORCE).
• Actor-critic: an actor selects actions and a critic evaluates them (for example, A2C, PPO).
• Model-free vs. model-based: a separate axis describing whether the agent learns an explicit model of environment dynamics or learns purely from sampled experience.

Key Algorithms: Q-Learning, PPO, and GRPO (as of 2026)

A handful of algorithms anchor both classical RL and its modern application to language models. Q-learning, introduced by Christopher Watkins (1989) and analyzed with Peter Dayan (1992), is a model-free, value-based method that learns the optimal action-value function using the Bellman equation, updating estimates toward the immediate reward plus the discounted value of the best next action.

Proximal Policy Optimization (PPO), introduced by John Schulman and colleagues at OpenAI in 2017 (arXiv:1707.06347), is an actor-critic policy-gradient method that constrains each update so the new policy does not drift too far from the old one, using a clipped objective. PPO became a workhorse for both control tasks and language-model fine-tuning because it is relatively stable and simple to implement.

Group Relative Policy Optimization (GRPO) was introduced in the DeepSeekMath paper (Shao et al., 2024, arXiv:2402.03300) and later popularized by DeepSeek-R1. GRPO is a variant of PPO that removes the separate critic (value) network: instead, it samples a group of responses for each prompt and uses their mean reward as the baseline for computing relative advantages, reducing memory and compute. As of 2026, PPO and GRPO are both widely used for reinforcement learning on large language models, with GRPO especially associated with reasoning-focused training; the relative prevalence of these methods continues to evolve, so 'most common' claims should be treated as time-dependent.

Reinforcement Learning vs. Supervised and Unsupervised Learning

RL is often contrasted with the other two major machine learning paradigms. Supervised learning trains on labeled examples, learning a direct mapping from inputs to known correct outputs. Unsupervised learning finds structure (clusters, representations, density) in unlabeled data without any target signal. RL sits apart because it learns from a reward signal generated by interaction, not from a fixed labeled dataset.

Three distinctions matter most. The feedback in RL is evaluative and often sparse or delayed, whereas supervised labels are dense and immediate. RL data is collected by the agent's own actions, creating a feedback loop absent from the other paradigms. And RL explicitly optimizes long-term cumulative return, requiring the agent to reason about sequences of decisions rather than single predictions.

How RL Powers LLM Alignment (RLHF, RLAIF, RLVR)

Reinforcement learning underpins much of how modern large language models are aligned to human preferences and made more useful. Reinforcement Learning from Human Feedback (RLHF) was popularized by OpenAI's InstructGPT (Ouyang et al., 2022, arXiv:2203.02155). The recipe has three stages: supervised fine-tuning on demonstrations, training a reward model on human rankings of candidate outputs, and then optimizing the language model against that reward model with an RL algorithm such as PPO. InstructGPT showed that a 1.3B-parameter model tuned this way could be preferred by human raters over the much larger 175B GPT-3.

Two related variants address the cost and reliability of human labels. RLAIF (Reinforcement Learning from AI Feedback), associated with Anthropic's Constitutional AI work, replaces or supplements human preference labels with judgments produced by an AI model guided by a written set of principles. RLVR (Reinforcement Learning with Verifiable Rewards) skips a learned reward model entirely for domains where correctness can be checked automatically, such as math or code: the model samples a solution, a verifier checks it, and the reward is the verification outcome. RLVR has been central to training reasoning models such as DeepSeek-R1.

An AI memory or second-brain application can use a learned ranking or reward signal to improve how well retrieved documents, photos, or voice notes match a user's plain-English query, though such retrieval ranking is distinct from full policy-optimization RL.

Strengths, Limitations, and Reward Hacking

RL's central strength is that it can optimize for long-horizon objectives in interactive settings where labeled supervision is unavailable, learning behaviors no one explicitly demonstrated. It has driven landmark results in game playing, robotics, and the alignment of language models.

Its limitations are equally important. RL is often sample-inefficient, sensitive to hyperparameters, and unstable to train. Most consequentially, performance depends entirely on the reward function, which is hard to specify correctly. Reward hacking occurs when an agent maximizes the stated reward without achieving the intended goal, exploiting loopholes in the specification or in an imperfect verifier or reward model. It is a special case of specification gaming and a core concern in AI alignment, mitigated through better reward design, regularization toward a reference policy, adversarial testing, and human oversight.