Hydra-0: Action Flow for Generalist World Modeling and Control

Overview video

Figure 1: Action flow as a shared control interface. Top: Hydra-0 learns from diverse interaction videos featuring egocentric human demonstrations, handheld UMI grippers, bimanual robot arms, and single-arm robots. Middle: Visible embodiment motion is represented as image-plane flow trajectories, placing heterogeneous interactions in a common, pixel-aligned action flow space independent of embodiment-specific joint or end-effector coordinates. This unified interface enables a single generalist world model to learn from multi-embodiment data and transfer across interaction settings. Bottom: In forward mode, provided gripper flow conditions future scene prediction, whose rollouts support open-loop policy evaluation; in inverse mode, desired object flow elicits compatible robot motion that a supervised readout converts into executable actions.

Action flow as a shared visual interface

We introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r = 0.96 between replayed and reference success rates.

Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible robot motion from desired object flow transferred from a human demonstration. A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations. Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control.

Method

One conditioning interface, built two ways: from video during training, from kinematics at deployment.

Action flow is a set of camera-plane trajectories describing the commanded motion of the visible embodiment: image positions with visibility flags, tracked over the prediction horizon. The same representation can describe a robot arm, a handheld gripper, or a human hand, without exposing any of their native action spaces to the video model.

Online deployment a hybrid loop: physics moves the robot, the world model moves the world

Deployment is not one simulator but two, and each does only what it is good at. Isaac Lab owns the robot: a rigid, calibrated, fully known body whose response to a command is a solved problem, so it is executed exactly rather than predicted. Hydra-0 owns everything the physics engine would need an asset and a material model for — the cloth, the deformable, the contact, the lighting, the scene it has never been given a mesh of. Action flow is the seam between them: it is what the physics engine hands over, and the only thing about the robot the video model is ever told.

1

2

3

4

Visible robot-surface points are propagated by the commanded link transforms and projected through the calibrated camera; the model then predicts the command's consequence. Step 3 opens on the Isaac Lab render carrying its own projected flow and dissolves to the real observation carrying the same flow — both are the same calibrated top camera, which is why the tracks sit on the arms in either, and is the whole reason a command executed in simulation can condition a prediction about the real scene. The three clips are one seventeen-second closed-loop rollout on a single clock, so the arm pose in the simulator, the flow drawn on the observation and the generated frame all belong to the same instant.

Grounding and sampling

Figure 2 · dense tracks are split by mask, then one conditioning mode is drawn per training step.

All seven panels are the same 81-frame window, played together. One conditioning mode is drawn per training step, from the pools the masks define: 1,819 embodiment tracks and 3,250 object tracks out of 14,997 the tracker returns.

Qualitative rollouts

Evaluation on multi-embodiment datasets

All methods see the same 100 validation clips per dataset across five held-out sets.

Within the same Cosmos 2.5 backbone, swapping the native relative 6D action for action flow improves PSNR, SSIM, gripper EPE, FID and FVD on all five datasets, and object EPE on the four where it is measured — a controlled comparison that isolates the conditioning representation. VLM scores are the exception, and are mixed: the judge rates physical plausibility and temporal consistency rather than whether the generated motion follows the commanded trajectories, which flow EPE measures directly. The best configuration is the distilled four-step Wan2.2 A14B model, best in nearly every cell and also the fastest.

Gray rows are zero-shot baselines using released checkpoints; the best value per column is bold green. Standard errors are reported in the paper. Object EPE is omitted for DROID, where object grounding is unreliable in cluttered scenes.

Policy evaluation by open-loop replay

The simulator is initialized from an episode's first observation and replays that episode's achieved trajectory as action flow.

Because the policy is never queried on generated observations, this measures whether the simulator preserves an outcome it is shown — not prospective evaluation of an unexecuted command. Generated rollouts are scored by a human rater applying the same task predicate, with policy identity withheld.

RoboLab

5 policies × 6 tasks · 300 episodes

π₀, π₀.₅, GR00T N1.7, Cosmos-3 Nano, and Cosmos-3 Edge, rolled out 10 times per task.

0.96

Pearson r

0.93

Spearman ρ

5.7 pp

Mean abs. error

• Aggregate success 26.3% simulated vs. 26.7% in RoboLab — a bias of −0.3 points.
• The two agree on the ordering of all five policies.
• Per-episode verdicts match 93.0% of the time (Cohen's κ = 0.82): 10 false positives, 11 false negatives.

Running the interface backwards

Figure 4 · flexible-pipe bending, from the human demonstration through to the robot's execution.

Supply desired object flow instead of embodiment flow and the world model generates a compatible robot motion, from which a learned action readout produces executable 14-DoF commands. The object flow here is transferred from a held-out human demonstration, and the readout is trained on paired real-world rollouts of both successes and failures — no task-specific expert robot demonstration is required.

BibTeX

If you find this work useful, please cite the report.

@misc{li_hydra-0_2026,
    title = {Hydra-0: {Action} {Flow} for {Generalist} {World} {Modeling} and {Control}},
    shorttitle = {Hydra-0},
    url = {http://arxiv.org/abs/2608.18077},
    doi = {10.48550/arXiv.2608.18077},
    publisher = {arXiv},
    author = {Li, Hongyu and Wen, Bowen and Zhu, Xinghao and Wang, Yixuan and Du, Yilun and Li, Yunzhu and Konidaris, George and Birchfield, Stan and Pouya, Soha and Li, Chenran and Chang, Yan},
    month = aug,
    year = {2026},
    note = {arXiv:2608.18077 [cs.RO]},
}