Object Path
Grounding DINO locates the prompted object, SAM2 tracks its mask, SAM3D builds a textured mesh, and FoundationPose estimates a smoothed 6D pose trajectory.
Overview
Video to Data includes a variety of reconstruction packages for recovering human, hand, and object trajectories from demonstration video. These outputs can be used by downstream tasks such as motion retargeting, robot simulation, policy training, and dataset analysis.
The reconstruction stack is modular: depth estimation, segmentation, mesh generation, object pose tracking, hand/body reconstruction, calibration, and visualization are separate tools with file-based inputs and outputs. That makes pipelines easier to resume, debug, compare, and extend as better models become available.
Ego-Centric Pipeline
The current first-class workflow is
modules/v2d_pipelines/run_v2d_ego_e2e.py. Given an MP4
and a text prompt for the held object, it runs an end-to-end
egocentric hand-object reconstruction flow. The pipeline first
undistorts footage into a pinhole camera model, then estimates the
object, hands, depth, pose trajectory, and final alignment.
Hand Path
Ego hand reconstruction recovers hand motion, converts hand records and intrinsics into pipeline formats, and re-expresses tracks for alignment.
Depth Alignment
MoGe depth provides the metric cues for mesh scale estimation, object tracking, EKF smoothing, and hand-object alignment.
Inspection Renders
The pipeline writes overlay and multi-view videos so object poses, hand tracks, and alignment quality can be checked without opening raw artifacts manually.
What It Produces
A run produces a structured output directory with extracted frames, depth maps, camera intrinsics, object masks, a textured object mesh, raw and smoothed object poses, hand reconstruction records, aligned hand tracks, and rendered diagnostic videos. The key reusable outputs are the scale-corrected object mesh, the smoothed per-frame object poses, and the depth-aligned hand records.
The pipeline is resumable: each step checks for its existing artifacts and skips completed work, which is important for long GPU reconstruction jobs and model-comparison runs.
Next Steps
The full setup guide still lives in the repository while this web page is expanded with installation steps, diagrams, examples, and generated references for module entry points.