File size: 6,642 Bytes
3aeb818 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 | # SimplerEnv
Framework for evaluating real-world robot manipulation policies (RT-1, RT-1-X, Octo) in simulation. Replicates common setups like Google Robot and WidowX+Bridge, with GPU-accelerated simulations (10-15x speedup). Offers visual matching and variant aggregation evaluation methods for robust policy assessment.
For more information, see the [official repository](https://github.com/simpler-env/SimplerEnv).
---
# Benchmark results
These values come from the default SimplerEnv evaluation runs (the per-task success rates are listed below).
## Bridge (WidowX robot)
Provided checkpoints:
- [nvidia/GR00T-N1.6-bridge](https://huggingface.co/nvidia/GR00T-N1.6-bridge)
- [nvidia/GR00T-N1.7-SimplerEnv-Bridge](https://huggingface.co/nvidia/GR00T-N1.7-SimplerEnv-Bridge)
| Task | N1.6 success rate | N1.7 success rate |
| --- | ---: | ---: |
| `widowx_spoon_on_towel` | 56/101 (55.4%) | 78/100 (78.0%) |
| `widowx_carrot_on_plate` | 46/100 (46.0%) | 58/100 (58.0%) |
| `widowx_put_eggplant_in_basket` | 89/100 (89.0%) | 53/100 (53.0%) |
| `widowx_stack_cube` | 5/100 (5.0%) | 48/100 (48.0%) |
| `widowx_put_eggplant_in_sink` | 33/100 (33.0%) | 2/100 (2.0%) |
| `widowx_close_drawer` | 73/100 (73.0%) | 97/100 (97.0%) |
| `widowx_open_drawer` | 95/100 (95.0%) | 100/100 (100.0%) |
| **Average** | **56.6%** | **62.3%** |
## Fractal (Google Robot)
Provided checkpoints:
- [nvidia/GR00T-N1.6-fractal](https://huggingface.co/nvidia/GR00T-N1.6-fractal)
- [nvidia/GR00T-N1.7-SimplerEnv-Fractal](https://huggingface.co/nvidia/GR00T-N1.7-SimplerEnv-Fractal)
| Task | N1.6 success rate | N1.7 success rate |
| --- | ---: | ---: |
| `google_robot_pick_coke_can` | 95/100 (95.0%) | 100/100 (100.0%) |
| `google_robot_pick_object` | 87/100 (87.0%) | 94/100 (94.0%) |
| `google_robot_move_near` | 81/100 (81.0%) | 100/100 (100.0%) |
| `google_robot_open_drawer` | 0/100 (0.0%) | 65/100 (65.0%) |
| `google_robot_close_drawer` | 44/100 (44.0%) | 69/100 (69.0%) |
| `google_robot_place_in_closed_drawer` | 5/100 (5.0%) | 7/100 (7.0%) |
| **Average** | **52.0%** | **72.5%** |
# Fine-tune Simpler Env bridge dataset (WidowX robot)
To reproduce our finetune results, use the following commands to setup dataset and launch finetune experiments. Please remember to set `WANDB_API_KEY` since W&B logging is on by default (`USE_WANDB=1` in `examples/finetune.sh`). If you don't have a WANDB account, prepend `USE_WANDB=0` to the launch command to disable it:
```bash
uv run hf download \
--repo-type dataset IPEC-COMMUNITY/bridge_orig_lerobot \
--local-dir examples/SimplerEnv/bridge_orig_lerobot/
# Copy the patches and run the finetune script
cp examples/SimplerEnv/bridge_modality.json examples/SimplerEnv/bridge_orig_lerobot/meta/modality.json
```
```bash
NUM_GPUS=8 MAX_STEPS=20000 GLOBAL_BATCH_SIZE=1024 SAVE_STEPS=1000 uv run bash examples/finetune.sh \
--base-model-path nvidia/GR00T-N1.7-3B \
--dataset-path examples/SimplerEnv/bridge_orig_lerobot/ \
--embodiment-tag SIMPLER_ENV_WIDOWX \
--output-dir /tmp/bridge_finetune \
--state-dropout-prob 0.8
```
# Fine-tune Simpler Env fractal dataset (Google robot)
```bash
uv run hf download \
--repo-type dataset IPEC-COMMUNITY/fractal20220817_data_lerobot \
--local-dir examples/SimplerEnv/fractal20220817_data_lerobot/
# Copy the patches and run the finetune script
cp -r examples/SimplerEnv/fractal_modality.json examples/SimplerEnv/fractal20220817_data_lerobot/meta/modality.json
uv run python examples/SimplerEnv/convert_av1_to_h264.py --root examples/SimplerEnv/fractal20220817_data_lerobot --jobs 16
```
```bash
NUM_GPUS=8 MAX_STEPS=20000 GLOBAL_BATCH_SIZE=1024 SAVE_STEPS=1000 uv run bash examples/finetune.sh \
--base-model-path nvidia/GR00T-N1.7-3B \
--dataset-path examples/SimplerEnv/fractal20220817_data_lerobot/ \
--embodiment-tag SIMPLER_ENV_GOOGLE \
--output-dir /tmp/fractal_finetune \
--state-dropout-prob 0.5
```
# Evaluate checkpoint
First, complete the [one-time simulation environment setup](../../README.md#one-time-simulation-environment-setup), then run this benchmark's setup script (only needed once per benchmark):
```bash
bash gr00t/eval/sim/SimplerEnv/setup_SimplerEnv.sh
```
Then, run client server evaluation under the project root directory in separate terminals:
## Fractal (Google Robot) Evaluation
**Terminal 1 - Server:**
You can use either a local finetuned checkpoint path or the remote finetuned checkpoint (provided by us):
**Option 1: Local finetuned checkpoint**
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path /tmp/fractal_finetune/checkpoint-30000 \
--embodiment-tag SIMPLER_ENV_GOOGLE \
--use-sim-policy-wrapper
```
**Option 2: Remote finetuned checkpoint (directly runnable)**
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path nvidia/GR00T-N1.7-SimplerEnv-Fractal \
--embodiment-tag SIMPLER_ENV_GOOGLE \
--use-sim-policy-wrapper
```
**Terminal 2 - Client:**
```bash
gr00t/eval/sim/SimplerEnv/simpler_uv/.venv/bin/python gr00t/eval/rollout_policy.py \
--n-episodes 10 \
--policy-client-host 127.0.0.1 \
--policy-client-port 5555 \
--max-episode-steps 300 \
--env-name simpler_env_google/google_robot_pick_coke_can \
--n-action-steps 1 \
--n-envs 5
```
## Bridge (WidowX) Evaluation
**Terminal 1 - Server:**
**Option 1: Local finetuned checkpoint**
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path /tmp/bridge_finetune/checkpoint-30000 \
--embodiment-tag SIMPLER_ENV_WIDOWX \
--use-sim-policy-wrapper
```
**Option 2: Remote finetuned checkpoint (directly runnable)**
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path nvidia/GR00T-N1.7-SimplerEnv-Bridge \
--embodiment-tag SIMPLER_ENV_WIDOWX \
--use-sim-policy-wrapper
```
**Terminal 2 - Client:**
```bash
gr00t/eval/sim/SimplerEnv/simpler_uv/.venv/bin/python gr00t/eval/rollout_policy.py \
--n-episodes 10 \
--policy-client-host 127.0.0.1 \
--policy-client-port 5555 \
--max-episode-steps 300 \
--env-name simpler_env_widowx/widowx_spoon_on_towel \
--n-action-steps 4 \
--n-envs 5
```
Other supported tasks are:
```
simpler_env_google/google_robot_pick_object
simpler_env_google/google_robot_move_near
simpler_env_google/google_robot_open_drawer
...
simpler_env_widowx/widowx_spoon_on_towel
simpler_env_widowx/widowx_carrot_on_plate
simpler_env_widowx/widowx_stack_cube
```
you can replace the env_name with the corresponding tasks listed in the SimplerEnv fork this repo pins at `external_dependencies/SimplerEnv` (see `.gitmodules`).
|