Video to Data (V2D) Challenge

One RGB camera, third person, no rigs, no depth — the setting that scales to internet video and low-cost capture. From a single clip, teams recover the full 4D scene: human body, object pose, and geometry, in metric scale and a consistent world frame. This track identifies the frontier of monocular HOI reconstruction, allowing practitioners to calibrate future stages to the errors expected when applied at scale.

Data. Our third-person RGB video dataset with held-out ground truth. This dataset includes challenging scenarios such as heavy occlusion, bimanual coordination, and long horizons.

Submissions. Teams will submit reconstruction results for the test split using the eval_reconstruction.py script to produce the artifact for submission.

Evaluation. Results will be evaluated on 2 axes against a multi-view reconstruction (MV) baseline, representing an upper bound.

• Axis 1, Accuracy: chamfer distance to MV human mesh, chamfer distance to MV object mesh
• Axis 2, Physical Plausibility: acceleration error of joints compared to MV, acceleration error of object compared to MV, contact penetration compared to MV

Each axis is weighted equally when awarding points.

Submission. Teams will submit recorded evaluations of their trained policies using the eval_robotic_grounding.py script provided to produce the artifact for submission.

Evaluation. At each tier, submissions will be evaluated for object-tracking accuracy using AUC, SP-SR, MP-SR, and MPPE as defined in the CHORD manuscript. Strong performance on more challenging tiers will be weighted appropriately when awarding points.

Data. The provided testing dataset is curated from our large-scale in-house dataset with held-out ground truth. We provide an end-to-end baseline that participants may build upon, modify, or choose not to use.

Method-agnostic. This track holistically evaluates the end-to-end results without constraining the overall approach, whether through explicit reconstruct-then-retargeting or implicit end-to-end learning with pre-trained VLA or WAM.

Submission. Teams will submit reconstructions and recorded evaluations of their end-to-end pipeline using the eval_e2e.py script to produce the artifact for submission.