World Model Evaluation

Background and Evaluation Setting

A world model predicts how the state of an environment changes under a given action, instruction, or condition. There are many ways to represent the world state and the action. In our current evaluation setting, we focus on a general and fundamental setup:

• The initial world state is represented by an image.
• A text prompt specifies how the world should change.
• The model generates a video showing the resulting state transition.

The prompt can describe a semantic action, such as moving an object from one location to another. It can also contain numerical constraints, such as a distance, angle, or direction.

The goal of the evaluation is not only to measure whether the generated video looks realistic, but also whether the predicted future is consistent with the instruction, the initial state, and the physical rules of the environment.

What Makes World Model Evaluation Difficult

A visually convincing video is not necessarily a physically correct video.

Current world models can often generate smooth and realistic-looking motion while still producing impossible physical behavior. Objects may pass through each other, change shape unexpectedly, move without a plausible cause, or violate basic constraints such as gravity, contact, and object solidity.

Some properties are relatively easy to evaluate. For example, visual artifacts such as flickering, camera discontinuity, or sudden changes in object appearance can often be detected reliably.

Physical plausibility is much harder.

Judging whether an interaction is physically possible may require reasoning about contact, geometry, object permanence, gravity, motion, and causality. These are also areas where current VLM judges can be unreliable. A video may look visually strong while containing a physical failure that is obvious to a human observer.

Evaluation Principles

Our evaluation approach follows four main principles:

Together, these principles are designed to evaluate not just whether a generated video looks good, but whether it represents a reasonable prediction of how the world should evolve.

Six Evaluation Dimensions

We evaluate each generation along six dimensions. Together, these dimensions cover low-level visual consistency, task-level correctness, and higher-level physical reasoning.

These dimensions are intentionally separated because different failure modes can look similar at first glance.

For example, a model may successfully complete a task but do so using physically impossible motion. Another model may generate physically plausible motion but interact with the wrong object. A useful evaluation should distinguish these cases rather than collapse them into a single visual-quality score.

Human Pairwise Evaluation

Given these six dimensions, evaluation can be performed either pointwise or pairwise.

In pointwise evaluation, each generated video is scored independently, for example on a scale from 0 to 1. This is difficult to calibrate consistently. It is not easy for a judge to define the difference between, for example, a physical-plausibility score of 0.7 and 0.8.

Pairwise evaluation asks a simpler question: Given two videos generated from the same initial image and prompt, which one performs better? The judge compares the two generations along each evaluation dimension and can also provide an overall preference. This makes the evaluation more concrete and reduces the need for judges to maintain a global scoring scale across many different scenes and tasks.

Pairwise preferences collected across many model comparisons can then be aggregated into an Elo score for each model.

Real-World Scenario Coverage

A world model intended for physical AI should work across many different environments, objects, and interactions. For this reason, our evaluation is not limited to controlled tabletop scenes. We construct prompts and conditioning images across a broad range of practical deployment settings, including:

• Home assistance
• Manufacturing
• Warehouse operations
• Food service
• Laboratory operations
• Wholesale and retail
• Healthcare services

Scenario diversity is not only about changing the background or industry. The evaluation also varies the types of objects, actions, spatial relationships, and physical interactions involved in each task. Examples include picking up objects, pushing, opening, placing, pouring, articulated motion, contact between objects, occlusion, navigation around obstacles, and interactions involving multiple objects.

A diverse evaluation set helps expose different weaknesses in world models and reduces the risk that model performance is dominated by a narrow set of familiar environments.

What the Evaluation Reveals

Running this evaluation also highlights several general properties of current world models.

Visual coherence can hide physical failure. A video can look smooth, realistic, and visually consistent while still showing physically impossible behavior. This means visual quality alone is not a sufficient proxy for world-model capability.

Physical failures often appear during interaction. Many failures become visible when objects make contact, move relative to each other, or undergo changes in configuration. These interaction-heavy moments often require more physical reasoning than passive scene generation.

Task completion and physical plausibility are different capabilities. A model can reach the requested final state while using an implausible transition. Conversely, a physically plausible video can still fail because the wrong object was manipulated or the requested goal was not reached. Both therefore need to be evaluated independently.

Different environments expose different weaknesses. Performance in one environment does not necessarily transfer to another. Home environments, warehouses, laboratories, manufacturing settings, and other physical AI domains contain different objects, interaction patterns, constraints, and failure modes. Broad scenario coverage is therefore necessary to understand the general capability of a world model.

The leaderboard below summarizes current World Model results using average Elo scores from pairwise comparisons. Read these rankings alongside the evaluation dimensions discussed above.

Takeaway

Evaluating a world model requires more than asking whether the generated video looks realistic. For physical AI applications, the more important question is whether the predicted future is consistent with the instruction, the initial world state, and the physical rules of the environment.

Our evaluation therefore combines human-grounded physical judgment, six complementary evaluation dimensions, pairwise model comparison, Elo-based aggregation, and broad real-world scenario coverage. The goal is to measure not just how well a model can generate video, but how well it can predict how the world should actually change.