G1 Policy Evaluation#
Evaluate a trained Unitree G1 policy on the apple-to-plate task in simulation. A GR00T server loads the checkpoint while Isaac Lab-Arena runs episodes and writes metrics. The two processes use separate locked environments. For Unitree H2 Plus, use H2 policy evaluation.
Use the published checkpoint below or the complete
outputs/g1/training/checkpoint-20000 directory from
G1 policy training. Run commands from this repository’s
root on the evaluation workstation.
Use a Published Checkpoint#
We provide a trained policy for G1 so you can skip recording, data generation, conversion, and training, or check your work against our reference. Run the command below from the workflow repository root to download it, then follow G1 policy evaluation to run it in simulation.
If you want to train your own model, return to G1 policy training.
Before downloading, complete Hugging Face access setup.
uv run --frozen --no-sync hf download nvidia/GR00T-E2E-Reference-Workflow-SIM-G1-N1.7-AppletoPlate-Model \
--revision d3075997eb71095238324746f0507f5d707cd895 \
--exclude optimizer.pt rng_state.pth scheduler.pt trainer_state.json training_args.bin \
--local-dir outputs/g1/published-checkpoint
The model is saved in outputs/g1/published-checkpoint. Keep the
.cache/huggingface/ directory created inside the download directory; the
launcher uses it to check that the model matches the pinned download.
Requirements#
This repository is synced with
uv sync --frozen.Isaac Lab-Arena is checked out cleanly at the revision in
runtime.toml.Isaac-GR00T is checked out separately and cleanly at the revision in
runtime.toml, withuv sync --frozenalready completed.The Hugging Face login used by the GR00T environment can access
nvidia/Cosmos-Reason2-2B.The checkpoint was trained for
NEW_EMBODIMENTusing GR00T N1.7, a 40-step action horizon,ego_view, five Unitree G1 state groups (31 values), and seven action groups (35 values). Isaac Lab-Arena maps those groups to its 43-joint state and 50-dimensional simulator action and obtainsego_viewfromrobot_head_cam_rgb.
The supplied training recipe accepts data recorded with Pink IK and AGILE in either Isaac Lab-Arena or native Isaac Lab. Both provide the navigation and base-height action groups this evaluator requires. Checkpoints trained with either group removed need custom evaluation integration.
Important
For a locally trained checkpoint, the path must name its checkpoint-<step> directory, not its parent
training-output directory. The launcher checks the required model, processor,
statistics, trainer-state, and experiment-configuration files for this default route.
--model-path is treated as a locally trained checkpoint by default. The
released Hugging Face checkpoint requires an explicit
--checkpoint-source released. For the workflow model downloaded above, use
--checkpoint-source published.
Prepare the Isaac Lab-Arena Checkout#
In the first terminal, set the paths and fetch the managed sparse checkout at the revision in
runtime.toml:export E2E_ROOT="$PWD" export ISAAC_GR00T_ROOT=/absolute/path/to/Isaac-GR00T export ARENA_ROOT="$(./tools/fetch_arena_checkout.sh)"
Verify the checkout in the same terminal:
uv run --frozen --no-sync ./tools/check_arena_pin.py \ --arena-root "$ARENA_ROOT"
The sparse checkout includes Isaac Lab-Arena’s source-only GR00T client. Passing the standalone checkout through
--gr00t-rootsupplies the model implementation, so no submodule is required. A clean full Isaac Lab-Arena clone at the same revision also remains supported asE2E_ARENA_ROOT, with the same explicit standalone GR00T root.Choose one model using the tabs below. Run only the matching block in the same terminal:
Use the model downloaded above:
export G1_CHECKPOINT="$E2E_ROOT/outputs/g1/published-checkpoint" export G1_CHECKPOINT_SOURCE=published
Use the model produced by policy training for G1:
export G1_CHECKPOINT="$E2E_ROOT/outputs/g1/training/checkpoint-20000" export G1_CHECKPOINT_SOURCE=trained
If you used a different training output directory, adjust the checkpoint path.
Both commands below use your selected model and source.
Run the Preflight#
The preflight checks both source revisions, clean checkouts, gated-model access, the trained-checkpoint layout, GR00T N1.7, and the checkpoint’s complete observation and action formats and normalization statistics against the Isaac Lab-Arena client’s requirements.
In that first terminal, run the preflight:
./tools/launch_gr00t_server.sh \ --arena-root "${ARENA_ROOT}" \ --gr00t-root "$ISAAC_GR00T_ROOT" \ --model-path "$G1_CHECKPOINT" \ --checkpoint-source "$G1_CHECKPOINT_SOURCE" \ --check
This command does not load the checkpoint or start Isaac Sim. A passing run prints:
GR00T evaluation contract passed: policy=Gr00tRemoteClosedloopPolicy, validation_scope=checkpoint, checkpoint_metadata=validated, expected_checkpoint_state=31, expected_checkpoint_action=35, sim_state=43, sim_action=50, horizon=40, camera=robot_head_cam_rgb
Fix any preflight failure before starting evaluation. Do not bypass a revision, model-type, joint, camera, or action-horizon mismatch.
Start the Policy Server#
In the same first terminal, start the server after preflight passes:
./tools/launch_gr00t_server.sh \ --arena-root "${ARENA_ROOT}" \ --gr00t-root "$ISAAC_GR00T_ROOT" \ --model-path "$G1_CHECKPOINT" \ --checkpoint-source "$G1_CHECKPOINT_SOURCE" \ --device cuda \ --host 127.0.0.1 \ --port 5555
Wait for:
Server is ready and listening on tcp://127.0.0.1:5555
Keep this process running while Isaac Lab-Arena evaluates the policy.
Run One Visual Episode#
In a second terminal at this repository’s root, set the runtime paths and run:
export ARENA_ROOT="$(./tools/fetch_arena_checkout.sh)" export ISAAC_GR00T_ROOT=/absolute/path/to/Isaac-GR00T
OMNI_KIT_ACCEPT_EULA=YES ACCEPT_EULA=Y \ ./tools/launch_gr00t_eval.sh \ --arena-root "${ARENA_ROOT}" \ --gr00t-root "${ISAAC_GR00T_ROOT}" \ --viz kit \ --remote_host 127.0.0.1 \ --remote_port 5555 \ --num_episodes 1 \ --output_base_dir outputs/g1/evaluation
This is a smoke evaluation. Inspect the complete workflow scene: the Unitree G1 should be upright, with the apple, plate, table, and head-camera view configured correctly. The run should complete one episode with a 300-step time limit, print metrics, write results under
outputs/g1/evaluation/, and close Isaac Sim automatically. Task terminations may end it earlier.Use
--viz noneinstead of--viz kitfor unattended evaluation.
Measure the Policy#
After the quick visual test passes, run enough complete episodes for the desired statistical confidence. For example:
OMNI_KIT_ACCEPT_EULA=YES ACCEPT_EULA=Y \ ./tools/launch_gr00t_eval.sh \ --arena-root "${ARENA_ROOT}" \ --gr00t-root "${ISAAC_GR00T_ROOT}" \ --viz none \ --remote_host 127.0.0.1 \ --remote_port 5555 \ --num_episodes 100 \ --output_base_dir outputs/g1/evaluation
Isaac Lab-Arena reports:
success_rate: episodes with settled, supported placement and both hands clear;object_moved_rate: episodes in which the apple moved; andprogress details for intermediate task objectives.
Each run creates a timestamped directory below
--output_base_dircontainingepisode_results_rank0.jsonlandindex.html. The final terminal metrics line gives the completed episode count and rates. For example, a one-episode smoke evaluation with the published checkpoint and seed 42 printed:[Rank 0/1] Metrics: {'num_episodes': 1, 'success_rate': 0.0, 'object_moved_rate': 1.0}The apple moved, but placement success was not reached within the 300-step limit. Treat this as an integration check, not a performance estimate. Compare
num_episodeswith your requested count and the records inepisode_results_rank0.jsonl. An incomplete run must not be reported as a completed 100-episode evaluation.The success rate is comparable to the training workflow only when the checkpoint used data from the same workflow environment and the same robot, camera, joint ordering, modality configuration, and action horizon.
Shut Down#
Isaac Lab-Arena exits when the requested episode count is reached.
In the server terminal, press Ctrl+C and wait for:
Shutting down server...
Confirm that port 5555 is free before starting another server:
ss -ltn 'sport = :5555'
The output should have no listening socket row.
Next Stage#
Review the timestamped index.html and episode_results_rank0.jsonl under
outputs/g1/evaluation/. Use failed episodes to decide whether to collect
more demonstrations, generate more data, or revise training. Keep the dataset,
checkpoint, and evaluation report together as the evidence for this run.
Optional: Use the Pinned Released Checkpoint#
To exercise the integration without training, download the released static-apple checkpoint pinned in
runtime.toml:checkpoint_pin="$( uv run --no-sync python -c \ 'import tomllib; p=tomllib.load(open("runtime.toml", "rb"))["tested"]; print(p["static_apple_checkpoint_repository"] + "@" + p["static_apple_checkpoint_revision"])' )"
"$ISAAC_GR00T_ROOT/.venv/bin/hf" download \ "${checkpoint_pin%@*}" \ --revision "${checkpoint_pin#*@}" \ --exclude optimizer.pt rng_state.pth scheduler.pt trainer_state.json training_args.bin \ --local-dir outputs/g1/released-checkpoint
Select this checkpoint in the server terminal, then rerun the preflight and server commands above:
export G1_CHECKPOINT="$E2E_ROOT/outputs/g1/released-checkpoint" export G1_CHECKPOINT_SOURCE=released
The released checkpoint was trained on a different scene, so its metrics prove integration only and are not a performance baseline for this workflow’s scene.
Troubleshooting#
Expected Gr00tN1d7,Expected action_horizon 40, orCheckpoint modality mismatch: select the N1.7 checkpoint produced by the matching training and modality configuration.Trained checkpoint must contain ...: point--model-pathatcheckpoint-<step>, not the parent output directory.A Cosmos access error: log in with
/absolute/path/to/Isaac-GR00T/.venv/bin/hf auth loginafter obtaining access.Cannot reach GR00T policy server: wait for the server-ready message and verify the host and port passed to both processes.A zero success rate with valid JSONL/report output means integration worked but the policy did not complete the task. Inspect the scene and dataset, then evaluate more episodes before drawing a performance conclusion.
The corresponding NVIDIA reference flow is documented in Sim Evaluation.