Update X-NavDP README
Browse files
README.md
CHANGED
|
@@ -1,3 +1,241 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<p align="center">
|
| 2 |
+
<h1 align="center"><strong>X-NavDP</strong></h1>
|
| 3 |
+
<h2 align="center">Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching</h2>
|
| 4 |
+
<p align="center">
|
| 5 |
+
<a href="https://yty-sky.github.io/" target="_blank">Tianyu Yang</a><sup>1,2*</sup> 
|
| 6 |
+
<a href="https://jzengym.github.io/JZENGYM/" target="_blank">Yiming Zeng</a><sup>3,2*</sup> 
|
| 7 |
+
<a href="https://wzcai99.github.io/" target="_blank">Wenzhe Cai</a><sup>2*</sup> 
|
| 8 |
+
<a href="https://yuqiang-yang.github.io/" target="_blank">Yuqiang Yang</a><sup>2</sup> 
|
| 9 |
+
<a href="https://steinate.github.io/" target="_blank">Jiaqi Peng</a><sup>4,2</sup> 
|
| 10 |
+
<a href="https://scholar.google.com/citations?view_op=search_authors&mauthors=Hui+Cheng+Sun+Yat-sen+University&hl=en" target="_blank">Hui Cheng</a><sup>3</sup> 
|
| 11 |
+
<a href="https://oceanpang.github.io/" target="_blank">Jiangmiao Pang</a><sup>2</sup> 
|
| 12 |
+
<a href="https://tai-wang.github.io/" target="_blank">Tai Wang</a><sup>2†</sup>
|
| 13 |
+
</p>
|
| 14 |
+
<p align="center">
|
| 15 |
+
<sup>1</sup>Fudan University 
|
| 16 |
+
<sup>2</sup>Shanghai AI Laboratory 
|
| 17 |
+
<sup>3</sup>Sun Yat-sen University 
|
| 18 |
+
<sup>4</sup>Tsinghua University
|
| 19 |
+
</p>
|
| 20 |
+
</p>
|
| 21 |
+
|
| 22 |
+
<div id="top" align="center">
|
| 23 |
+
|
| 24 |
+
[](https://yty-sky.github.io/x-navdp-project-page/)
|
| 25 |
+
[](https://yty-sky.github.io/x-navdp-project-page/#bibtex)
|
| 26 |
+
[](https://yty-sky.github.io/x-navdp-project-page/#video)
|
| 27 |
+
[](../../README.md#-internvla-n1-system-1-benchmark)
|
| 28 |
+
[](https://huggingface.co/datasets/InternRobotics/Scene-N1/tree/main/n1_eval_scenes)
|
| 29 |
+
[](https://github.com/InternRobotics/NavDP)
|
| 30 |
+
|
| 31 |
+
</div>
|
| 32 |
+
|
| 33 |
+
<p align="center">
|
| 34 |
+
<a href="#installation">Installation</a> |
|
| 35 |
+
<a href="#assets">Assets</a> |
|
| 36 |
+
<a href="#training">Training</a> |
|
| 37 |
+
<a href="#evaluation">Evaluation</a> |
|
| 38 |
+
<a href="#results">Results</a> |
|
| 39 |
+
<a href="#citation">Citation</a>
|
| 40 |
+
</p>
|
| 41 |
+
|
| 42 |
+
## Introduction
|
| 43 |
+
|
| 44 |
+
X-NavDP post-trains a pretrained RGBD camera-based navigation diffusion policy via online reinforcement learning across heterogeneous embodiments. Beyond improving general navigation and obstacle avoidance performance, it gains new capabilities for backing out of traps, long-obstacle detours, and embodiment-aware behavior adaptation. Our contributions to navigation diffusion policy are threefold:
|
| 45 |
+
|
| 46 |
+
- ⚡ **Data-efficient RL post-training.** X-NavDP enhances pretrained diffusion policies through efficient large-scale, multi-scene online post-training for stronger general navigation ability.
|
| 47 |
+
- ⚒ **Structured exploration and stable training.** Goal-agnostic diffusion trajectories and **Group Q-score Reweighted Matching** enable structured exploration, improve training stability, and handle hard cases.
|
| 48 |
+
- 🤝 **Cross-robot generalization and temporal consistency.** Lightweight embodiment modulation and RTC guidance improve cross-robot generalization and temporal consistency, leading to superior post-trained navigation performance.
|
| 49 |
+
|
| 50 |
+
<p align="center">
|
| 51 |
+
<img src="fig/teaser.jpg" width="80%">
|
| 52 |
+
</p>
|
| 53 |
+
|
| 54 |
+
## Installation
|
| 55 |
+
|
| 56 |
+
Clone NavDP and enter the self-contained X-NavDP baseline:
|
| 57 |
+
|
| 58 |
+
```bash
|
| 59 |
+
git clone https://github.com/InternRobotics/NavDP.git
|
| 60 |
+
cd NavDP/baselines/x-navdp
|
| 61 |
+
```
|
| 62 |
+
|
| 63 |
+
Create a Python environment and install the dependencies:
|
| 64 |
+
|
| 65 |
+
```bash
|
| 66 |
+
conda create -n navrl python=3.11
|
| 67 |
+
conda activate navrl
|
| 68 |
+
pip install -r requirements.txt
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
Install [Isaac Sim 5.0.0](https://docs.isaacsim.omniverse.nvidia.com/5.0.0/installation/index.html), [Isaac Lab 0.46.2](https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/index.html), and `isaaclab-rl==0.4.0` following their official instructions. Then install acados following the [acados installation guide](https://docs.acados.org/installation/index.html) and [Python interface guide](https://docs.acados.org/python_interface/index.html). The acados setup is lightweight and mainly requires a CMake build plus the Python interface. Before running training or evaluation, expose the acados libraries and accept the Omniverse EULA:
|
| 72 |
+
|
| 73 |
+
```bash
|
| 74 |
+
export ACADOS_SOURCE_DIR=/path/to/acados
|
| 75 |
+
export LD_LIBRARY_PATH="${ACADOS_SOURCE_DIR}/lib:${LD_LIBRARY_PATH:-}"
|
| 76 |
+
export OMNI_KIT_ACCEPT_EULA=YES
|
| 77 |
+
export OMNI_KIT_ALLOW_ROOT=1
|
| 78 |
+
```
|
| 79 |
+
|
| 80 |
+
## Assets
|
| 81 |
+
|
| 82 |
+
Large assets, datasets, robot USDs, low-level controller checkpoints, and pretrained policy checkpoints are not included in this repository. You need to download these assets manually before running training or evaluation.
|
| 83 |
+
|
| 84 |
+
The **GRScenes100 / N1 evaluation scenes** are available from [InternRobotics/Scene-N1](https://huggingface.co/datasets/InternRobotics/Scene-N1/tree/main/n1_eval_scenes). This dataset contains sky textures, materials, cluttered easy/hard scenes, InternScenes home scenes, and InternScenes commercial scenes.
|
| 85 |
+
|
| 86 |
+
The **X-NavDP assets** are available from [InternRobotics/X-NavDP](https://huggingface.co/InternRobotics/X-NavDP/tree/main). This repository provides `navigation_metadata`, robot assets, low-level controller checkpoints, `scene_split.json`, the NavDP pretrained checkpoint, and the X-NavDP post-trained checkpoint.
|
| 87 |
+
|
| 88 |
+
After downloading, place or symlink the scene data, metadata, robot assets, and checkpoints in the following layout. The scene root can also be kept elsewhere and passed with `SCENE_DIR`.
|
| 89 |
+
|
| 90 |
+
<table>
|
| 91 |
+
<tr>
|
| 92 |
+
<td width="52%" valign="top">
|
| 93 |
+
|
| 94 |
+
**Scene data and navigation metadata**
|
| 95 |
+
|
| 96 |
+
```text
|
| 97 |
+
x-navdp/
|
| 98 |
+
+-- data/scenes/
|
| 99 |
+
+-- scene_split.json
|
| 100 |
+
+-- SkyTexture/
|
| 101 |
+
+-- Materials/
|
| 102 |
+
+-- cluttered_easy/
|
| 103 |
+
+-- cluttered_hard/
|
| 104 |
+
+-- internscenes_commercial/
|
| 105 |
+
| +-- models/
|
| 106 |
+
| +-- Materials/
|
| 107 |
+
| +-- scenes_commercial/
|
| 108 |
+
+-- internscenes_home/
|
| 109 |
+
| +-- models/
|
| 110 |
+
| +-- Materials/
|
| 111 |
+
| +-- scenes_home/
|
| 112 |
+
+-- navigation_metadata/
|
| 113 |
+
+-- internscenes_commercial/
|
| 114 |
+
+-- internscenes_home/
|
| 115 |
+
+-- cluttered_easy/
|
| 116 |
+
+-- cluttered_hard/
|
| 117 |
+
```
|
| 118 |
+
|
| 119 |
+
</td>
|
| 120 |
+
<td width="48%" valign="top">
|
| 121 |
+
|
| 122 |
+
**Code, robot assets, and checkpoints**
|
| 123 |
+
|
| 124 |
+
```text
|
| 125 |
+
x-navdp/
|
| 126 |
+
+-- train.py
|
| 127 |
+
+-- config/
|
| 128 |
+
+-- scripts/
|
| 129 |
+
+-- src/
|
| 130 |
+
| +-- environment/
|
| 131 |
+
| | +-- controllers/checkpoints/
|
| 132 |
+
| | +-- humanoid_g1/policy.pt
|
| 133 |
+
| | +-- quadruped_go2/policy.pt
|
| 134 |
+
| +-- x_navdp/
|
| 135 |
+
| +-- training/
|
| 136 |
+
| +-- utils/
|
| 137 |
+
+-- eval/
|
| 138 |
+
+-- data/robots/
|
| 139 |
+
| +-- dingo.usd
|
| 140 |
+
| +-- unitreeg1.usd
|
| 141 |
+
+-- pretrain_model/
|
| 142 |
+
| +-- navdp_pretrained.ckpt
|
| 143 |
+
+-- checkpoints/
|
| 144 |
+
+-- x-navdp_posttrain.ckpt
|
| 145 |
+
```
|
| 146 |
+
|
| 147 |
+
</td>
|
| 148 |
+
</tr>
|
| 149 |
+
</table>
|
| 150 |
+
|
| 151 |
+
Training and evaluation expect `SCENE_DIR` to point to the scene root. The training config initializes from `pretrain_model/navdp_pretrained.ckpt`; evaluation examples use the released post-trained checkpoint under `checkpoints/`. Dingo and G1 load robot USDs from `data/robots/`, while Unitree Go2 uses the Isaac Lab asset path by default.
|
| 152 |
+
|
| 153 |
+
## Training
|
| 154 |
+
|
| 155 |
+
The released [post-trained model](https://huggingface.co/InternRobotics/X-NavDP/blob/main/x-navdp_posttrain.ckpt) was trained for 24,000 steps on 72 scenes, including 16 clutter scenes and 56 home/commercial scenes.
|
| 156 |
+
|
| 157 |
+
Single-node 8-GPU distributed training can be launched with:
|
| 158 |
+
|
| 159 |
+
```bash
|
| 160 |
+
export SCENE_DIR=/path/to/NavDP/baselines/x-navdp/data/scenes
|
| 161 |
+
export NPROC_PER_NODE=8
|
| 162 |
+
|
| 163 |
+
bash scripts/run_ddp_train.sh \
|
| 164 |
+
--config_file config/x-navdp_config.yaml
|
| 165 |
+
```
|
| 166 |
+
|
| 167 |
+
For a smaller debug run, reduce the number of processes and environments:
|
| 168 |
+
|
| 169 |
+
```bash
|
| 170 |
+
export NPROC_PER_NODE=1
|
| 171 |
+
|
| 172 |
+
bash scripts/run_ddp_train.sh \
|
| 173 |
+
--num_envs 1 \
|
| 174 |
+
--embodiments dingo \
|
| 175 |
+
--max_steps 100
|
| 176 |
+
```
|
| 177 |
+
|
| 178 |
+
Training writes periodic evaluation records to the directory passed through
|
| 179 |
+
`--txt_dir`. Aggregate these logs into global, per-embodiment, and per-scene
|
| 180 |
+
success-rate CSV files and plots with:
|
| 181 |
+
|
| 182 |
+
```bash
|
| 183 |
+
python scripts/aggregate_success.py ./txt/x-navdp
|
| 184 |
+
```
|
| 185 |
+
|
| 186 |
+
By default, outputs are written to `./result/<txt_subdir_name>/`; for the
|
| 187 |
+
command above, the output directory is `./result/x-navdp/`.
|
| 188 |
+
Each log row has the form
|
| 189 |
+
`rank,step,episode,success,trainer_success_rate`. Home/commercial and clutter
|
| 190 |
+
results are reported separately. Use `--output-dir` to override the result
|
| 191 |
+
directory, or `--no-plots` to generate CSV files only; run
|
| 192 |
+
`python scripts/aggregate_success.py --help` for EMA and milestone options.
|
| 193 |
+
|
| 194 |
+
## Evaluation
|
| 195 |
+
|
| 196 |
+
Evaluation uses a policy server and an Isaac Lab client.
|
| 197 |
+
|
| 198 |
+
Start the policy server:
|
| 199 |
+
|
| 200 |
+
```bash
|
| 201 |
+
bash eval/scripts/start_policy_server.sh \
|
| 202 |
+
--checkpoint checkpoints/x-navdp_posttrain.ckpt \
|
| 203 |
+
--embodiment quadruped
|
| 204 |
+
```
|
| 205 |
+
|
| 206 |
+
Run point-goal evaluation:
|
| 207 |
+
|
| 208 |
+
```bash
|
| 209 |
+
bash eval/scripts/run_evaluation.sh \
|
| 210 |
+
--config_file eval/config/eval_pointgoal/quadruped_internscene_commercial.yaml
|
| 211 |
+
```
|
| 212 |
+
|
| 213 |
+
By default, evaluation runs all start-goal samples in the selected scene. Use `--num_episodes` to evaluate a subset and `--max_steps` to cap the total simulation steps for smoke tests or debugging.
|
| 214 |
+
|
| 215 |
+
Evaluation configs are provided under `eval/config/eval_pointgoal/`, and outputs are written under `outputs/evaluation/<embodiment>_<scene_type>/`. Each scene directory contains a `metric.csv` file whose first two columns are success and SPL. To print per-USD scene SR and SPL, run:
|
| 216 |
+
|
| 217 |
+
```bash
|
| 218 |
+
python eval/scripts/stat_eval_metrics.py outputs/evaluation/quadruped_commercial
|
| 219 |
+
```
|
| 220 |
+
|
| 221 |
+
## Results
|
| 222 |
+
|
| 223 |
+
On the project benchmark, X-NavDP improves the overall simulation success rate from 61.20% to 84.28% and improves real-world hard-case success rate from 10% to 65%. See the [project results](https://yty-sky.github.io/x-navdp-project-page/#results) for simulation and real-world comparisons across wheeled Dingo, quadruped Unitree Go2, and humanoid Unitree G1 embodiments.
|
| 224 |
+
|
| 225 |
+
## Citation
|
| 226 |
+
|
| 227 |
+
```bibtex
|
| 228 |
+
@misc{yang2026xnavdp,
|
| 229 |
+
title = {X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching},
|
| 230 |
+
author = {Tianyu Yang and Yiming Zeng and Wenzhe Cai and Yuqiang Yang and Jiaqi Peng and Hui Cheng and Jiangmiao Pang and Tai Wang},
|
| 231 |
+
year = {2026}
|
| 232 |
+
}
|
| 233 |
+
```
|
| 234 |
+
|
| 235 |
+
## License
|
| 236 |
+
|
| 237 |
+
Original X-NavDP code is released under the [MIT License](LICENSE). Vendored and external dependencies retain their upstream terms; see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
|
| 238 |
+
|
| 239 |
+
## Acknowledgement
|
| 240 |
+
|
| 241 |
+
This project builds on Isaac Sim, Isaac Lab, acados, and NavDP. X-NavDP is distributed as a self-contained baseline in the [InternRobotics/NavDP](https://github.com/InternRobotics/NavDP) repository.
|