Overview
Robot learning needs more scalable ways to turn real-world experience into usable training data. Video is an abundant record of human-object interaction — but it is unstructured and hard to connect to simulation or policy learning.
This tutorial motivates Video to Data as a practical bridge from raw demonstrations to actionable robot-learning assets, showing how modern vision-language models, reconstruction tools, and robotic grounding convert everyday video into structured data for learning generalizable robot skills. We teach the pipeline as a sequence of validation gates: queryable video segments, plausible reconstructions, simulation-ready task assets, retargeted motions, and policy-training artifacts.
How can raw human video become robot-usable data?
Participants leave with runnable scripts or notebooks, precomputed artifacts, and an adaptation checklist for deciding whether their own video data is ready for robot learning.
We are hosting the Video to Data Challenge! Registrations are now open. Find out more at this link ↗.
Learning Objectives
By the end of the tutorial, participants will be able to:
Hands-on Format
A guided coding session: short conceptual introductions followed by runnable exercises.
Because full reconstruction and policy training can require GPUs, every exercise has three paths: laptop-only inspection, optional live GPU execution, and precomputed artifacts — so every attendee can complete the objectives.
Each module begins with a brief explanation, then moves quickly to an exercise.
Participants compare retrieved clips, overlays, pose tracks, motion-schema fields, and simulation rollouts.
Groups classify outputs as ready, questionable, or unsuitable for robot learning.
Instructors run selected reconstruction overlays and a short Isaac Lab dexterity run.
Each exercise produces a visible artifact: a JSONL clip list, retrieved clip, mesh/pose overlay, retargeted motion file, scene viewer, or policy rollout.
Tentative Schedule
Instructors
Each instructor owns a distinct pipeline stage and supports the hands-on exercises, artifact clinics, and debugging sessions.
Participant Materials
Materials will be public and useful even for participants who do not use NVIDIA infrastructure after the event.
We encourage participants to bring their own data questions and use the final guided extension to decide whether their videos are ready, need preprocessing, or need additional sensing or annotation.
Organizers
Contact
For questions about the tutorial, please contact the organizing team.