Video to Data (V2D) Challenge

One RGB camera, third person, no rigs, no depth — the setting that scales to internet video and low-cost capture. From a single clip, teams recover the full 4D scene: human body, object pose, and geometry, in metric scale and a consistent world frame. This track identifies the frontier of monocular HOI reconstruction, allowing practitioners to calibrate future stages to the errors expected when applied at scale.

Data. Our third-person RGB video dataset with held-out ground truth. This dataset includes challenging scenarios such as heavy occlusion, bimanual coordination, and long horizons.

Submissions. Teams will submit reconstruction results for the test split using the eval_reconstruction.py script to produce the artifact for submission.

Evaluation. Results will be evaluated on 2 axes against a multi-view reconstruction (MV) baseline, representing an upper bound.

• Axis 1, Accuracy: chamfer distance to MV human mesh, chamfer distance to MV object mesh
• Axis 2, Physical Plausibility: acceleration error of joints compared to MV, acceleration error of object compared to MV, contact penetration compared to MV

Each axis is weighted equally when awarding points.

We have split Track 2 into 3 tiers based on the quality of the reference trajectory.

Teams will be evaluated against the clean multi-view captioned object trajectory in all three tiers. A valid Track 2 submission consists of a separate submission to each tier. Teams may modify their approach between tiers. For Tiers 2 and 3, teams are encouraged to propose methods for recovering better estimates of the object trajectory using anything other than the clean trajectory captions provided in Tier 1. For Tier 3, video to data provides CARI4D as an off the shelf solution.

Submission. Teams will submit recorded evaluations of their trained policies using the eval_robotic_grounding.py script provided to produce the artifact for submission.

Evaluation. At each tier, submissions will be evaluated for object-tracking accuracy using AUC, SP-SR, MP-SR, and MPPE as defined in the CHORD manuscript. Each tier is weighted equally when calculating final scores for this track.

Data. The provided testing dataset is curated from our large-scale in-house dataset with held-out ground truth. We provide an end-to-end baseline that participants may build upon, modify, or choose not to use.

Method-agnostic. This track holistically evaluates the end-to-end results without constraining the overall approach, whether through explicit reconstruct-then-retargeting or implicit end-to-end learning with pre-trained VLA or WAM.

Submission. Teams will submit reconstructions and recorded evaluations of their end-to-end pipeline using the eval_e2e.py script to produce the artifact for submission.