--- license: apache-2.0 library_name: transformers pipeline_tag: image-text-to-text language: - en tags: - navigation - vln - vln-ce - instruction-generation - visual-trajectory-prompting - qwen3-vl - acmmm-2026 base_model: - Qwen/Qwen3-VL-8B-Instruct paper: https://arxiv.org/abs/2608.15284 --- # VTInstructor-8B **VTInstructor-8B** is the final SoTA checkpoint of **[VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments](https://arxiv.org/abs/2608.15284)** (ACM MM 2026). This release is the **VP-GRPO** model: Qwen3-VL-8B-Instruct + VP-Adapter, after supervised fine-tuning and then VP-GRPO refinement, which updates the VP-Adapter gates together with the language decoder while the vision tower and the VP encoder stay frozen. It generates natural-language navigation instructions from egocentric RGB trajectories in continuous environments (R2R-CE / RxR-CE). - **Paper:** [arXiv:2608.15284](https://arxiv.org/abs/2608.15284) - **Code:** [github.com/TidalHarley/VTInstructor](https://github.com/TidalHarley/VTInstructor) - **Backbone:** [`Qwen/Qwen3-VL-8B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)

VTInstructor overview

## What this model does Given a first-person trajectory (RGB keyframes + discrete actions), VTInstructor writes a concise indoor navigation instruction that matches the full path. Visual Trajectory Prompts (path ribbon, turn arrows, goal marker) and a 3-channel semantic mask are injected through a lightweight VP-Adapter; the same checkpoint also works on plain RGB renders without VTP, because the trajectory context is already absorbed into the weights. Typical uses: 1. **Instruction generation** on R2R-CE / RxR-CE trajectories 2. **Data augmentation** for downstream VLN followers (see §3 below) 3. **Drop-in instruction replacement** on VLN-CE `.json.gz` splits via the open-source generation script This model is intended for **non-commercial research** on vision-and-language navigation. See [License](#license). ## Results All numbers below are copied from the ACM MM 2026 paper. Figures are the original tables. ### 1. Navigation instruction generation (NLG) VTInstructor is the best model on both **R2R-CE Val Unseen** and **RxR-CE Val Unseen**, including against GPT-5.4, Gemini-3.1-Pro-Preview, and much larger open-source VLMs. Relative to the strongest baseline it gains **+0.357 CIDEr on R2R-CE** and **+0.109 CIDEr on RxR-CE**.

Paper Table 1: NLG results on R2R-CE and RxR-CE Val Unseen

**R2R-CE Val Unseen** | Method | BLEU-1 | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | Qwen3.5-Plus | 0.557 | 0.131 | 0.223 | 0.387 | 0.137 | 0.180 | | GPT-5.4 | 0.482 | 0.075 | 0.197 | 0.348 | 0.078 | 0.144 | | Gemini-3.1-Pro-Preview | 0.631 | 0.166 | 0.209 | 0.392 | 0.203 | 0.167 | | xAI Grok 4 | 0.486 | 0.084 | 0.207 | 0.354 | 0.073 | 0.160 | | Claude Opus 4.6 | 0.484 | 0.084 | 0.190 | 0.337 | 0.068 | 0.119 | | Qwen3.5-397B-A17B | 0.525 | 0.113 | 0.216 | 0.382 | 0.120 | 0.174 | | Kimi-K2.5 | 0.478 | 0.088 | 0.195 | 0.357 | 0.098 | 0.146 | | LLaVA-Video-7B-Qwen2 | 0.506 | 0.100 | 0.146 | 0.317 | 0.108 | 0.109 | | Qwen3-VL-8B-Instruct | 0.524 | 0.085 | 0.175 | 0.334 | 0.147 | 0.137 | | Qwen3.5-9B | 0.579 | 0.121 | 0.194 | 0.373 | 0.181 | 0.151 | | GLM-4.1V-9B-Thinking | 0.490 | 0.080 | 0.156 | 0.321 | 0.130 | 0.115 | | GLM-4.6V | 0.556 | 0.120 | 0.171 | 0.348 | 0.168 | 0.122 | | Qwen3-VL-8B (SFT-only) | 0.720 | 0.282 | 0.232 | 0.485 | 0.484 | 0.201 | | **VTInstructor (this repo)** | **0.765** | **0.320** | **0.263** | **0.511** | **0.560** | **0.245** | **RxR-CE Val Unseen** | Method | BLEU-1 | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | Qwen3.5-Plus | 0.527 | 0.080 | 0.157 | 0.254 | 0.033 | 0.156 | | GPT-5.4 | 0.493 | 0.056 | 0.146 | 0.241 | 0.029 | 0.143 | | Gemini-3.1-Pro-Preview | 0.393 | 0.073 | 0.128 | 0.242 | 0.029 | 0.135 | | xAI Grok 4 | 0.430 | 0.049 | 0.181 | 0.249 | 0.014 | 0.144 | | Claude Opus 4.6 | 0.367 | 0.043 | 0.176 | 0.223 | 0.005 | 0.123 | | Qwen3.5-397B-A17B | 0.422 | 0.059 | 0.128 | 0.249 | 0.027 | 0.142 | | Kimi-K2.5 | 0.432 | 0.071 | 0.195 | 0.259 | 0.011 | 0.145 | | LLaVA-Video-7B-Qwen2 | 0.064 | 0.012 | 0.063 | 0.169 | 0.002 | 0.074 | | Qwen3-VL-8B-Instruct | 0.366 | 0.053 | 0.119 | 0.230 | 0.022 | 0.113 | | Qwen3.5-9B | 0.515 | 0.077 | 0.171 | 0.259 | 0.027 | 0.164 | | GLM-4.1V-9B-Thinking | 0.074 | 0.014 | 0.054 | 0.144 | 0.003 | 0.064 | | GLM-4.6V | 0.489 | 0.065 | 0.143 | 0.242 | 0.027 | 0.145 | | Qwen3-VL-8B (SFT-only) | 0.630 | 0.201 | 0.210 | 0.357 | 0.060 | 0.162 | | **VTInstructor (this repo)** | **0.774** | **0.308** | **0.265** | **0.431** | **0.142** | **0.206** | Comparison with prior *speaker* methods trained on discretized R2R / RxR viewpoint graphs (those numbers are **not** on the same continuous-env protocol):

Paper Table 2: comparison with prior speaker methods

### 2. Frozen VLN follower Instructions from VTInstructor are executed by a **frozen CorrectNav** follower on R2R-CE Val Unseen. VTInstructor improves frozen-follower success by **14.7 percentage points** over the strongest baseline, and matches or exceeds official human instructions on SR / OSR / NE.

Paper Table 5: frozen CorrectNav follower on R2R-CE Val Unseen

| Instruction source | SR↑ | OSR↑ | SPL↑ | NE↓ | | :--- | :---: | :---: | :---: | :---: | | Official human instruction | 61.6 | 67.2 | **53.3** | 4.53 | | Qwen3.5-Plus | 48.6 | 61.8 | 39.0 | 5.70 | | GPT-5.4 | 35.9 | 51.4 | 28.6 | 6.14 | | Gemini-3.1-Pro-Preview | 48.5 | 56.4 | 39.9 | 5.59 | | xAI Grok 4 | 45.5 | 64.8 | 33.4 | 6.18 | | Claude Opus 4.6 | 21.7 | 48.8 | 15.2 | 9.56 | | Qwen3.5-397B-A17B | 45.2 | 63.9 | 33.3 | 6.15 | | Kimi-K2.5 | 43.9 | 58.6 | 33.6 | 6.49 | | LLaVA-Video-7B-Qwen2 | 30.2 | 44.8 | 24.3 | 7.91 | | Qwen3-VL-8B-Instruct | 33.4 | 46.8 | 25.9 | 7.09 | | Qwen3.5-9B | 36.2 | 45.2 | 29.1 | 6.82 | | GLM-4.1V-9B-Thinking | 31.1 | 42.9 | 24.4 | 7.37 | | GLM-4.6V | 29.9 | 40.6 | 24.0 | 7.11 | | **VTInstructor (this repo)** | **63.3** | **70.0** | 52.7 | **4.47** | ### 3. Data augmentation for a VLN follower Training CorrectNav (LLaVA-Video-7B backbone) with human R2R-CE+RxR-CE data **plus** VTInstructor-generated instructions yields about **+3 SR** on both Val Unseen splits.

Paper Tables 6-7: data-augmentation results

**R2R-CE Val Unseen** | Set | Training data | SR↑ | OSR↑ | SPL↑ | NE↓ | | :--- | :--- | :---: | :---: | :---: | :---: | | A | R2R-CE + RxR-CE (human) | 45.1 | 52.3 | 44.6 | 6.20 | | B | Setting A + VTInstructor-generated | **48.4** | **54.2** | **46.8** | **5.85** | **RxR-CE Val Unseen** | Set | Training data | SR↑ | OSR↑ | SPL↑ | NE↓ | | :--- | :--- | :---: | :---: | :---: | :---: | | A | R2R-CE + RxR-CE (human) | 41.2 | 51.1 | 39.6 | 8.34 | | B | Setting A + VTInstructor-generated | **44.4** | **53.2** | **41.3** | **7.74** | ## Files in this repo | File | Role | | :--- | :--- | | `model-0000{1–4}-of-00004.safetensors` | Qwen3-VL-8B weights (~8.8B params, bf16). The VP tensors also appear here because the modules are registered on the model, but the code loads the two `.pt` files below. | | `vp_encoder.pt` | VP-Encoder CNN (3-channel ribbon/arrow/endpoint mask → features) | | `vp_adapters.pt` | Gated VP-Adapter at ViT layer 7 | | `config.json`, tokenizer, preprocessor | Standard Qwen3-VL processor files | | `figures/` | Paper overview + result tables | Both `vp_encoder.pt` and `vp_adapters.pt` ship in this repo, so pointing `CKPT` / `EVAL_CKPT` at the downloaded directory is enough — no extra files are needed. `transformers==4.57.6` is the validated version. Newer `transformers` may break the VP-Adapter injection into Qwen3-VL internals. Loading **only** with `Qwen3VLForConditionalGeneration.from_pretrained` gives the language/vision backbone; to reproduce paper numbers you must attach the VP modules from the [code repo](https://github.com/TidalHarley/VTInstructor). ## Quick start ```bash # 1. weights hf download TidalYang/VTInstructor-8b --local-dir ./VTInstructor-8b # 2. code git clone https://github.com/TidalHarley/VTInstructor.git && cd VTInstructor pip install -r requirements.txt ``` Generate instructions (RGB-only, or with VTP masks if the split has them): ```bash CKPT=../VTInstructor-8b \ DATA_DIR=/path/to/rendered_split \ DATASET_TYPE=r2rce \ OUT_JSON=outputs/augment/r2r_style.json \ bash generate/run_generate.sh ``` `DATASET_TYPE=rxrce` switches to the longer RxR-style narration. `VP_MODE=auto` (default) uses `vp_masks` when present and falls back to RGB otherwise. `MODEL_DIR` is optional — the processor files in this repo are used by default. `GEN_CUDA` selects GPUs (default `0`); shards are merged automatically. To reproduce the paper's benchmark numbers instead (needs rendered val_unseen splits): ```bash EVAL_CKPT=../VTInstructor-8b \ MODEL_DIR=/path/to/Qwen3-VL-8B-Instruct \ EVAL_CUDA=0 \ bash eval/run_eval.sh ``` Set `EVAL_CUDA` to the GPU ordinals you actually have — it defaults to all eight. Scoring additionally needs `pycocoevalcap` and a Java runtime on `PATH` (for METEOR / SPICE). ## Training recipe (how this checkpoint was produced) 1. **SFT** of Qwen3-VL-8B-Instruct + VP-Adapter on filtered R2R-CE / RxR-CE visual trajectories 2. **VP-GRPO** RL with a weighted NLG reward (BLEU-1/4, METEOR, ROUGE-L, CIDEr), updating the VP-Adapter gates and the language decoder while the vision tower and the VP encoder stay frozen → this checkpoint See the GitHub README for rendering, VTP mask construction, SFT, GRPO, and evaluation. ## Citation If you use this model, please cite: ```bibtex @misc{yang2026vtinstructorvisualtrajectoryprompting, title={VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments}, author={Haolin Yang and Yuxing Long and Zihan Yang and Hao Dong}, year={2026}, eprint={2608.15284}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2608.15284}, } ``` ## License The code release is [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0), and these weights are distributed under the same terms. This model was trained on data derived from **R2R / R2R-CE**, **RxR / RxR-CE** and **Matterport3D**, which are distributed under CC BY-NC-SA 3.0 US and the [Matterport3D Terms of Use](http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf). Please respect those terms — the checkpoint is intended for **non-commercial research use**. The [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) backbone license also applies.