Models & Evaluation

World Models

A world model is a learned internal model of an environment's dynamics that lets an agent predict how the world will change and what its actions will do, so it can plan or imagine rollouts instead of acting purely reactively.

What a World Model Is

A world model is a learned predictive model of how an environment behaves. Given the current state of the world and a candidate action, it predicts the next state, and often the resulting observation and reward. An agent equipped with such a model can reason about the consequences of actions before taking them, running predictions forward to evaluate plans rather than discovering outcomes only by trial in the real environment.

The term draws on a much older idea from cognitive science: that biological agents carry compressed internal representations of their surroundings and use them to anticipate events. In machine learning the phrase was popularized by David Ha and Jurgen Schmidhuber's 2018 paper World Models, which built a small generative model of a game environment and trained a controller to act using its predictions. The core claim is that a good model of dynamics is reusable knowledge: it supports planning, fast adaptation, and sample-efficient learning.

World models are usually contrasted with policies and value functions that map observations directly to actions or values without any explicit forecast. A policy answers what to do now; a world model answers what would happen if. The two are complementary, and most modern model-based agents learn both, using the world model to generate experience or gradients that train the policy.

A world model predicts environment dynamics: next state, observation, and reward given a state and action.
It lets an agent plan or imagine outcomes instead of relying only on reactive responses learned from past rewards.
The modern framing comes from Ha and Schmidhuber's 2018 paper World Models (arXiv:1803.10122).
A world model is reusable knowledge about dynamics, distinct from a policy that maps observations directly to actions.

The Formal Picture: Learned Latent Dynamics

Most learned world models operate in a compressed latent state rather than on raw pixels. An encoder maps high-dimensional observations to a compact latent representation, a transition model predicts how that latent state evolves under actions, and decoder heads reconstruct observations and predict rewards. Working in latent space keeps imagined rollouts cheap and focuses model capacity on the structure that matters for control.

The central object is a stochastic transition model that defines a probability distribution over the next latent state given the current state and action. Paired with a reward model and an observation model, this forms a learned partially observed Markov decision process that the agent can simulate. Training typically maximizes the likelihood of observed transitions while regularizing the latent space, for example through a variational objective.

The original World Models architecture made these pieces concrete: a variational autoencoder (the V model) compressed each frame into a latent vector, a recurrent mixture density network (the MDN-RNN, or M model) predicted a distribution over the next latent given the action, and a small linear controller (the C model) chose actions from the combined state. Later systems keep the same V, M, and controller decomposition while changing the specific architectures.

s_{t+1} ~ p_θ(s_{t+1} | s_t, a_t)   (transition)
r_t = R_θ(s_t)   (reward model)
o_t ~ q_θ(o_t | s_t)   (observation/decoder model)

A learned world model factors environment dynamics into a stochastic transition, a reward head, and an observation decoder, all parameterized by θ. Together they form a simulatable model the agent can roll forward.

An encoder compresses observations into a latent state; a transition model predicts the next latent under an action.
Reward and observation heads turn the latent dynamics into a simulatable Markov decision process.
Ha and Schmidhuber used a VAE plus an MDN-RNN plus a linear controller as the V, M, and C components.
Operating in latent space makes long imagined rollouts cheap enough to train a policy on.

Training Inside a Dream and Planning by Imagination

Once a world model can predict transitions, an agent can generate experience without touching the real environment. Ha and Schmidhuber showed an agent trained entirely inside rollouts produced by its own model, what the paper calls its dream, and then transferred to the actual task. This is appealing because simulated steps are far cheaper than real ones, which matters for robotics and any setting where real interaction is slow, expensive, or unsafe.

Planning by imagination means using the model to evaluate candidate action sequences. A planner can sample or optimize action sequences, roll each one forward through the transition model, score the predicted rewards, and pick the best, then re-plan at the next step. Alternatively, a policy and value function can be trained directly on imagined trajectories, propagating learning signals back through the model's predictions.

The main caution is compounding error. Because each predicted step feeds the next, small inaccuracies accumulate over long horizons, and a policy can learn to exploit flaws in the model rather than succeed in the real world. Effective systems limit imagination horizons, model uncertainty, and keep grounding the model with fresh real data.

python

# Latent-imagination planning loop (schematic, no specific library)
state = encoder(observation)          # compress real observation to latent

def imagine(state, policy, horizon):
    total_reward = 0.0
    for _ in range(horizon):
        action = policy(state)        # actor proposes an action
        state = transition(state, action)  # predicted next latent state
        total_reward += reward_model(state) # predicted reward
    return total_reward, state

# Train the policy on rollouts imagined inside the world model
for step in range(num_updates):
    ret, _ = imagine(state, policy, horizon=15)
    loss = -ret                       # maximize imagined return
    update(policy, loss)              # gradients flow back through the model

A schematic latent-imagination loop in the style of the Dreamer line: roll the learned transition model forward from a latent state and train the policy on the imagined return rather than on real environment steps.

Imagined rollouts let an agent generate cheap training experience without real-world interaction.
Planning can either optimize action sequences against the model or train a policy on imagined trajectories.
Prediction errors compound over long horizons, so agents can exploit model flaws if horizons are too long.
Practical systems keep refreshing the model with real data and bound how far ahead they imagine.

Model-Based vs Model-Free Reinforcement Learning

World models sit at the heart of model-based reinforcement learning. A model-free agent learns a policy or value function directly from rewarded experience and never builds an explicit model of dynamics. A model-based agent learns a model and uses it to plan, generate synthetic experience, or compute policy gradients. The trade is sample efficiency against modeling difficulty: model-based methods can extract more behavior from fewer real interactions, but they only help when the learned dynamics are accurate enough to trust.

The DeepMind Dreamer line is the most prominent recent demonstration that latent world models scale. Dream to Control (Hafner et al., 2019, arXiv:1912.01603) introduced learning behaviors by latent imagination. Mastering Atari with Discrete World Models (2021, arXiv:2010.02193) added discrete latents as DreamerV2. The third version, described in Mastering Diverse Domains through World Models (arXiv:2301.04104) and later published in Nature in 2025 as Mastering diverse control tasks through world models, used a single fixed configuration across more than 150 tasks and was the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula.

These results are established and reproducible in their stated benchmarks. They show that learned world models can match or beat specialized model-free agents on data efficiency and final performance in many control and game environments. They do not by themselves establish that the same approach yields broad real-world physical understanding, which remains a separate and open question.

Model-free RL learns policies directly from reward; model-based RL learns dynamics and plans or imagines with them.
Model-based methods can be far more sample-efficient when the learned model is accurate enough to trust.
The Dreamer line (DreamerV1 to V3) scaled latent imagination across Atari, control suites, and Minecraft.
DreamerV3 used one configuration across 150+ tasks and reached Minecraft diamonds without human data.

Generative Video Models as World Simulators

A more recent and more speculative framing treats large generative video models as a route to general world models. OpenAI presented its text-to-video system Sora in 2024 in a technical report titled Video generation models as world simulators, arguing that scaling such models is a promising path toward general purpose simulators of the physical world. Google DeepMind's Genie line generates action-controllable, playable environments from prompts; Genie: Generative Interactive Environments (2024, arXiv:2402.15391) learned interactive worlds from unlabeled video, and Genie 3, announced in August 2025, generates navigable environments in real time and was described by DeepMind as a step toward more general artificial intelligence.

The connection to the Dreamer-style definition is real but partial. Both kinds of system predict future observations conditioned on context, and Genie additionally conditions on user actions, which is the defining feature of a controllable world model. The difference is that pixel-accurate generation is not the same as accurate, controllable dynamics, and a model can produce convincing frames while violating physics or losing object permanence over time.

Whether next-frame or next-token prediction yields genuine physical understanding is an open debate, not a settled result. Yann LeCun's 2022 position paper A Path Towards Autonomous Machine Intelligence argues that predicting raw pixels wastes capacity on unpredictable detail and proposes joint embedding predictive architectures that model dynamics in a learned representation space instead. The honest reading in 2026 is that generative video models show impressive emergent consistency yet still exhibit physical errors, and the claim that they constitute general world models remains contested.

OpenAI framed Sora as a step toward general purpose simulators of the physical world in its 2024 report.
DeepMind's Genie generates action-controllable playable environments, with Genie 3 running in real time.
Convincing video does not guarantee correct dynamics; models can still break physics and object permanence.
Whether next-frame prediction produces true physical understanding is debated, not established.

Why World Models Matter and What Is Still Open

World models are attractive for robotics and planning because real interaction is expensive and risky. A robot that can imagine the result of a grasp or a step can evaluate options in simulation and act more deliberately, and a model that captures dynamics transfers more readily to new tasks than a policy fit to one reward. Sample efficiency, transfer, and the ability to reason about counterfactual actions are the recurring practical motivations.

Some researchers argue that a strong world model is a path toward more general intelligence, on the view that anticipating consequences and planning are prerequisites for common sense. This is a research direction and an argument, not a demonstrated outcome. The strongest current evidence is narrow: agents that plan inside learned models excel on specific benchmarks, and generative video models produce increasingly coherent but still imperfect simulations.

The open problems are concrete. Long-horizon prediction accumulates error, learned models can be exploited by the very policies they train, evaluating whether a model truly understands physics rather than memorizing surface statistics is hard, and grounding imagined plans in a changing real world requires continual correction. World models are a genuine and active line of work; specific claims about general physical understanding or a clear road to general AI should be treated as hypotheses to test, not as established facts.

World models support sample-efficient learning, transfer, and counterfactual planning, which is valuable in robotics.
Some researchers argue strong world models are a path to general AI, but this remains a hypothesis, not a result.
Demonstrated strengths are narrow: benchmark control tasks and increasingly coherent but imperfect video.
Open challenges include compounding error, model exploitation, and verifying genuine physical understanding.

Key takeaways

A world model is a learned model of environment dynamics that predicts next states, observations, and rewards from a state and an action.
It enables planning and imagined rollouts, so an agent can evaluate actions before taking them rather than acting purely reactively.
The modern framing comes from Ha and Schmidhuber's 2018 World Models, which combined a VAE, an MDN-RNN, and a small controller.
DeepMind's Dreamer line (V1 to V3) scaled latent imagination; DreamerV3 reached Minecraft diamonds from scratch and appeared in Nature in 2025.
Generative video models like Sora and Genie are framed as steps toward general world simulators, but this is contested, not established.
Core open problems include compounding prediction error, policy exploitation of model flaws, and verifying genuine physical understanding.

Frequently asked questions

A world model is a learned internal model of how an environment behaves. It predicts the next state, observation, and reward given the current state and a chosen action, which lets an agent imagine the consequences of actions and plan instead of relying only on reactive, previously rewarded behavior.

A policy maps observations directly to actions and answers what to do now. A world model predicts what would happen if an action were taken, answering a different question. Model-based agents usually learn both and use the world model to imagine experience or compute gradients that train the policy.

The 2018 paper (arXiv:1803.10122) compressed game frames with a variational autoencoder, predicted the next latent state with a recurrent mixture density network, and trained a small controller to act from those features. Notably it trained an agent entirely inside rollouts generated by its own model before transferring to the real task.

Dreamer is a family of model-based reinforcement learning agents from DeepMind that learn behaviors by latent imagination. It progressed from Dream to Control in 2019 (arXiv:1912.01603) to DreamerV2 and DreamerV3, the last of which used a single configuration across more than 150 tasks and collected diamonds in Minecraft from scratch without human data.

They are framed as steps toward world models because they predict future frames, and Genie also conditions on user actions, which makes it controllable. Whether next-frame prediction yields genuine physical understanding is an open debate, since these models can produce convincing video while still violating physics and losing consistency over time.

Real-world interaction is slow, costly, and sometimes unsafe, so an agent that can imagine outcomes can plan and learn from fewer real steps. Some researchers argue that anticipating consequences is a prerequisite for common sense and general intelligence, though this is a research hypothesis rather than a demonstrated result.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free