V Video to Data / Reconstruction Docs home GitHub
← Video to Data docs

Package Docs

Reconstruction

Turn human demonstration video into geometric reconstructions with the reconstruction components available in Video to Data.

Egocentric reconstruction output showing a source camera view and reconstructed hand-object scene from top, side, and front world views.
Ego-centric hand-object reconstruction output for the sample airplane clip.

Overview

Video to Data includes a variety of reconstruction packages for recovering human, hand, and object trajectories from demonstration video. These outputs can be used by downstream tasks such as motion retargeting, robot simulation, policy training, and dataset analysis.

All reconstruction components are containerized with scripts to easily build and execute them. This makes reproducing results straightforward.

Ego-Centric Hand-Object Reconstruction

Video to Data includes a pipeline for ego-centric hand-object reconstruction, detailed in ego_e2e_setup.md. Given an MP4 and a text prompt for the held object, it estimates the object, hands, depth, pose trajectory, and final alignment.

Object Path

Grounding DINO locates the prompted object, SAM2 tracks its mask, SAM3D builds a textured mesh, and FoundationPose estimates a smoothed 6D pose trajectory.

Hand Path

Ego hand reconstruction recovers hand motion, converts hand records and intrinsics into pipeline formats, and re-expresses tracks for alignment.

Depth Alignment

MoGe depth provides the metric cues for mesh scale estimation, object tracking, EKF smoothing, and hand-object alignment.

Inspection Renders

The pipeline writes overlay and multi-view videos so object poses, hand tracks, and alignment quality can be checked without opening raw artifacts manually.

New in v0.2.0

  • Gravity Alignment: Align reconstructed hand and object trajectories to gravity for a stable, physically meaningful world orientation.
  • SAM2 + HaMeR Hand Tracking: Choose an alternative hand-tracking pipeline that uses SAM2 hand masks with HaMeR hand reconstruction.
  • Gaussian-Splat Optimization: Refine global trajectories using Gaussian-splat-based optimization for improved consistency across the sequence.

Object Mesh Reconstruction

As of v0.2.0, Video to Data now supports object mesh reconstruction using a variety of methods. BundleSDF reconstructs a textured mesh from a multi-view scan, while SAM3D generates representative single-image meshes with silhouette-based scale estimation. A predefined object mesh can also be used when a suitable asset is available. See v2d_hoi_object_reconstruction/README.md for capture requirements and usage details.

Toy airplane used as the input reference for object mesh reconstruction.
Input reference for the toy-airplane object mesh reconstruction example.

What It Produces

A run produces a structured output directory with extracted frames, depth maps, camera intrinsics, object masks, a textured object mesh, raw and smoothed object poses, hand reconstruction records, aligned hand tracks, and rendered diagnostic videos. The key reusable outputs are the scale-corrected object mesh, the smoothed per-frame object poses, and the depth-aligned hand records.

The pipeline is resumable: each step checks for its existing artifacts and skips completed work, which is important for long GPU reconstruction jobs and model-comparison runs.

Next Steps

The full setup guide still lives in the repository while this web page is expanded with installation steps, diagrams, examples, and generated references for module entry points.