Architecture and Data Requirements#

The repository is a reference implementation built on NVIDIA Isaac Lab for composing a robot-learning workflow from thin stages. The current workflow paths use Unitree G1 and Unitree H2 Plus robots to move an apple to a plate. They provide a reproducible path from simulation and XR demonstration collection through post-training and closed-loop simulator evaluation. Physical Unitree G1 deployment guidance is also in scope for this documentation. The executable workflow described here ends at simulator evaluation. Simulator validation does not validate behavior on the physical robot.

See Conceptual Overview for the role of each stage and the distinction between the simulation and physical data paths.

Pipeline#

        flowchart TB
    recording["Record in simulation"] --> source_hdf5["Manifested HDF5"]
    source_hdf5 -. "native G1" .-> annotation["Annotate"]
    source_hdf5 -. "Isaac Lab-Arena with Pink IK" .-> generation["AutoData"]
    annotation --> generation
    generation --> generated_hdf5["Generated HDF5"]
    source_hdf5 --> conversion
    generated_hdf5 --> conversion
    conversion["Convert to LeRobot"] --> training["Train GR00T N1.7"]
    training --> evaluation["Arena evaluation"]

    physical_recording["Record on the physical G1"] --> mcap["MCAP recording"]
    mcap --> physical_conversion["Convert to LeRobot"]
    physical_conversion --> physical_training["Train GR00T N1.7"]
    physical_training --> leapp["LEAPP export"]
    leapp --> deployment["Isaac ROS deployment"]
    deployment --> physical_evaluation["Physical task evaluation"]

    class recording,physical_recording,evaluation,physical_evaluation accent
    class source_hdf5,generated_hdf5,mcap data
    class annotation,generation optional
    

Isaac AutoData is the optional branch: it amplifies a few recorded episodes into many. Demonstrations recorded with Unitree G1 in native Isaac Lab require an annotation stage; AutoData-compatible Isaac Lab-Arena recordings made with Pink IK contain datagen_info. Joint-control recordings are not AutoData inputs.

Conversion uses the robot-specific configuration and articulation targets rather than treating raw inputs to the robot-control software as interchangeable. Unitree G1 conversion writes a 43-D joint state and target; Unitree H2 writes a 75-D joint state and target. Their GR00T modalities expose 31-D state/35-D action for Unitree G1 and 75-D state/61-D action for Unitree H2. Demonstrations recorded with G1 using Pink IK and AGILE in Isaac Lab-Arena, or with AGILE in native Isaac Lab, including generated data, provide the navigation and base-height fields required by the supplied G1 configuration. Demonstrations recorded with a fixed-base G1 omit them and need custom training and inference integration. Demonstrations recorded with H2 in Isaac Lab-Arena provide navigation only, as required by its separate modality. Policy post-training for Unitree H2 has been validated. Learned-policy evaluation is implemented but not yet validated.

Both acquisition paths produce LeRobot datasets, but the resulting datasets and checkpoints are not interchangeable by format alone. Each retains the camera, robot-state and action formats, timing, robot configuration, and task settings of its source.

Robot-Control Software#

        flowchart TB
    command["XR, AutoData, or GR00T action"] --> manager["Selected environment's<br/>Isaac Lab ActionManager"]
    manager --> terms["Configured action terms<br/>Built-in G1, Arena G1 WBC, or Arena H2"]
    terms --> targets["Articulation joint targets"]
    targets --> robot["Robot in the workflow task scene"]

    class command,robot accent
    class targets data
    

Actions enter the selected environment’s ActionManager, which dispatches its configured action terms to run the robot-control software; that software does not feed the manager. The three robot-control setups are mutually exclusive environment configurations. In the built-in Unitree G1 task, fixed mode omits the lower-body action term and pins the base. Agile mode adds the AGILE lower-body term alongside Pink and records its locomotion commands.

Isaac Lab-Arena Unitree G1 instead combines upper- and lower-body control inside one WBC action term. It runs a separately pinned AGILE artifact and takes a different locomotion command shape. sim-agile-locomotion and sim-arena-locomotion therefore qualify both, against one shared set of behavior limits.

Unitree H2 is Isaac Lab-Arena-only and uses separate upper-body and lower-body terms. Its recording embodiment combines pelvis-relative Pink flange targets, 44 Sharpa joint targets, and a three-value AGILE velocity command. Trigger or squeeze input from the handheld XR controllers is converted into a binary open/close command with hysteresis, which expands into those hand targets. The evaluation embodiment replaces Pink with named upper-body joint targets while retaining the same AGILE velocity tail.

The built-in Unitree G1 Pink-plus-AGILE runtime is active. The separately named g1_pink_ik_agile_stabilized controller-profile metadata remains planned and cannot be selected as a recording-ready profile.

Where the Scene Comes From#

Isaac Lab-Arena composes the scene, embodiment, and task for Arena-based Unitree G1 runs and all Unitree H2 simulation runs. The native Isaac Lab scripts for recording demonstrations and driving Unitree G1 build the scene through G1StaticAppleEnvCfg. This remains a supported path and manual behavior cross-check. The two Unitree G1 runtimes use the same task definition, which sim-common-environment compares, but their robot-control software and raw action formats are not interchangeable. Environment ownership describes the supported paths and their validation boundaries.