Action Model Evaluation

Background

An action model takes the world state and task instruction as input, and outputs actions that drive the robot toward task goal. They can be VLAs, WAMs, and general-purpose VLMs used for robotic control. Their harnesses can also differ: a VLM may act as a high-level planner paired with a low-level policy, or directly serve as the policy itself.

Current action model eval face several challenges:

• Self-claimed real-robot results are hard to audit. Evaluations are often self-reported under different hardware, tasks, and protocols, making fair comparison difficult.
• Results depend heavily on the harness. Different simulators, control interfaces, and execution settings can lead to very different outcomes.
• Existing evaluations are fragmented. Results are scattered across different benchmarks, tasks, and platforms, without a unified evaluation protocol.

We evaluate action models in both real-robot and simulation settings under a consistent evaluation framework.

Simulation vs Real Robots

Simulation and real-robot eval serve complementary purposes. Simulation scales efficiently and is best suited for measuring cognitive capabilities, such as perception, reasoning, planning, and task execution across a wide range of scenarios. Real robots are essential for measuring physical execution capability, including grasp precision, contact handling, robustness, and performance under real-world dynamics. The conceptual relationship is illustrated below:

Therefore, we use both simulation and real robots for evaluation. Simulation is used for broad cognitive capability eval, and real robots for reliable physical behavior end-to-end. The conceptual relationship between cognition, execution and task success is shown as below:

Adaptation across Models and Embodiment

Different action models can produce different types of outputs. To evaluate them fairly, we convert these outputs into a common robot control format.

For example, a VLM may output a target end-effector pose, which is then converted into joint movements by an inverse kinematics controller. Other models may directly output end-effector movements or joint commands.

We use a similar approach across different robot embodiments. We currently focus mainly on bimanual robots with two-finger grippers. The same target actions are adapted to each robot based on its kinematics and control interface, while keeping the task and evaluation criteria unchanged.

Human Pairwise Evaluation

Task success is not always easy to check with code. In real-robot evaluation, object states and relations are often not directly available. Even in simulation, complex tasks may have several valid ways to succeed, so a fixed success rule can be hard to define.

We therefore use human pairwise comparison. For the same task, two models are run independently, and their head-camera videos are shown side by side to a human judge. The judge picks the better execution using the following priorities:

Pairwise results across many tasks are then combined into Elo scores for each model. The overall process follows the illustration below:

Evaluation Setup and Insights

Our evaluation covers both VLMs and VLAs. For VLMs, we report the average performance across two execution modes: direct action execution and using a VLA as the low-level controller. All models are evaluated under the same unified harness and settings. The current task set is sampled from the well-acknowledged simulation benchmark. Real-robot evaluation will be added soon. Results are shown below:

Our evaluation highlights several common patterns in current action models.

Good spatial cognitive capability does not always lead to successful execution. A model may understand the task and choose the right action, but still fail during grasping, contact, or precise motion. Task success depends on both cognitive capability and physical execution.

The evaluation harness matters. The same model can perform differently depending on its observation setup, action space, controller, and execution loop. A unified harness is therefore important for fair comparison.

Behavior and partial progress is informative. A failed task does not always mean the model has no useful capability. Progress toward the goal, reasonable recovery attempts, and efficient actions can reveal meaningful differences between models even when neither fully succeeds.

Takeaway

Action model evaluation should measure more than whether a task is eventually completed. It should capture both cognitive capability and physical execution, while keeping the evaluation harness consistent across models. Our framework combines diverse settings with unified model adaptation, and human pairwise judging to compare different action models under a common protocol. The goal is to provide a fairer and more complete view of how well a model can understand a task, act toward it, and execute it in the physical world.