README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
language:
  - en
tags:
  - navigation
  - vln
  - vln-ce
  - instruction-generation
  - visual-trajectory-prompting
  - qwen3-vl
  - acmmm-2026
base_model:
  - Qwen/Qwen3-VL-8B-Instruct
paper: https://arxiv.org/abs/2608.15284
---

# VTInstructor-8B

**VTInstructor-8B** is the final SoTA checkpoint of **[VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments](https://arxiv.org/abs/2608.15284)** (ACM MM 2026).

This release is the **VP-GRPO** model: Qwen3-VL-8B-Instruct + VP-Adapter, after supervised fine-tuning and then VP-GRPO refinement, which updates the VP-Adapter gates together with the language decoder while the vision tower and the VP encoder stay frozen. It generates natural-language navigation instructions from egocentric RGB trajectories in continuous environments (R2R-CE / RxR-CE).

- **Paper:** [arXiv:2608.15284](https://arxiv.org/abs/2608.15284)
- **Code:** [github.com/TidalHarley/VTInstructor](https://github.com/TidalHarley/VTInstructor)
- **Backbone:** [`Qwen/Qwen3-VL-8B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)

<p align="center">
  <img src="figures/overview.png" alt="VTInstructor overview" width="100%"/>
</p>

## What this model does

Given a first-person trajectory (RGB keyframes + discrete actions), VTInstructor writes a concise indoor navigation instruction that matches the full path. Visual Trajectory Prompts (path ribbon, turn arrows, goal marker) and a 3-channel semantic mask are injected through a lightweight VP-Adapter; the same checkpoint also works on plain RGB renders without VTP, because the trajectory context is already absorbed into the weights.

Typical uses:

1. **Instruction generation** on R2R-CE / RxR-CE trajectories
2. **Data augmentation** for downstream VLN followers (see §3 below)
3. **Drop-in instruction replacement** on VLN-CE `.json.gz` splits via the open-source generation script

This model is intended for **non-commercial research** on vision-and-language navigation. See [License](#license).

## Results

All numbers below are copied from the ACM MM 2026 paper. Figures are the original tables.

### 1. Navigation instruction generation (NLG)

VTInstructor is the best model on both **R2R-CE Val Unseen** and **RxR-CE Val Unseen**, including against GPT-5.4, Gemini-3.1-Pro-Preview, and much larger open-source VLMs. Relative to the strongest baseline it gains **+0.357 CIDEr on R2R-CE** and **+0.109 CIDEr on RxR-CE**.

<p align="center">
  <img src="figures/table1.png" alt="Paper Table 1: NLG results on R2R-CE and RxR-CE Val Unseen" width="100%"/>
</p>

**R2R-CE Val Unseen**

| Method | BLEU-1 | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| Qwen3.5-Plus | 0.557 | 0.131 | 0.223 | 0.387 | 0.137 | 0.180 |
| GPT-5.4 | 0.482 | 0.075 | 0.197 | 0.348 | 0.078 | 0.144 |
| Gemini-3.1-Pro-Preview | 0.631 | 0.166 | 0.209 | 0.392 | 0.203 | 0.167 |
| xAI Grok 4 | 0.486 | 0.084 | 0.207 | 0.354 | 0.073 | 0.160 |
| Claude Opus 4.6 | 0.484 | 0.084 | 0.190 | 0.337 | 0.068 | 0.119 |
| Qwen3.5-397B-A17B | 0.525 | 0.113 | 0.216 | 0.382 | 0.120 | 0.174 |
| Kimi-K2.5 | 0.478 | 0.088 | 0.195 | 0.357 | 0.098 | 0.146 |
| LLaVA-Video-7B-Qwen2 | 0.506 | 0.100 | 0.146 | 0.317 | 0.108 | 0.109 |
| Qwen3-VL-8B-Instruct | 0.524 | 0.085 | 0.175 | 0.334 | 0.147 | 0.137 |
| Qwen3.5-9B | 0.579 | 0.121 | 0.194 | 0.373 | 0.181 | 0.151 |
| GLM-4.1V-9B-Thinking | 0.490 | 0.080 | 0.156 | 0.321 | 0.130 | 0.115 |
| GLM-4.6V | 0.556 | 0.120 | 0.171 | 0.348 | 0.168 | 0.122 |
| Qwen3-VL-8B (SFT-only) | 0.720 | 0.282 | 0.232 | 0.485 | 0.484 | 0.201 |
| **VTInstructor (this repo)** | **0.765** | **0.320** | **0.263** | **0.511** | **0.560** | **0.245** |

**RxR-CE Val Unseen**

| Method | BLEU-1 | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| Qwen3.5-Plus | 0.527 | 0.080 | 0.157 | 0.254 | 0.033 | 0.156 |
| GPT-5.4 | 0.493 | 0.056 | 0.146 | 0.241 | 0.029 | 0.143 |
| Gemini-3.1-Pro-Preview | 0.393 | 0.073 | 0.128 | 0.242 | 0.029 | 0.135 |
| xAI Grok 4 | 0.430 | 0.049 | 0.181 | 0.249 | 0.014 | 0.144 |
| Claude Opus 4.6 | 0.367 | 0.043 | 0.176 | 0.223 | 0.005 | 0.123 |
| Qwen3.5-397B-A17B | 0.422 | 0.059 | 0.128 | 0.249 | 0.027 | 0.142 |
| Kimi-K2.5 | 0.432 | 0.071 | 0.195 | 0.259 | 0.011 | 0.145 |
| LLaVA-Video-7B-Qwen2 | 0.064 | 0.012 | 0.063 | 0.169 | 0.002 | 0.074 |
| Qwen3-VL-8B-Instruct | 0.366 | 0.053 | 0.119 | 0.230 | 0.022 | 0.113 |
| Qwen3.5-9B | 0.515 | 0.077 | 0.171 | 0.259 | 0.027 | 0.164 |
| GLM-4.1V-9B-Thinking | 0.074 | 0.014 | 0.054 | 0.144 | 0.003 | 0.064 |
| GLM-4.6V | 0.489 | 0.065 | 0.143 | 0.242 | 0.027 | 0.145 |
| Qwen3-VL-8B (SFT-only) | 0.630 | 0.201 | 0.210 | 0.357 | 0.060 | 0.162 |
| **VTInstructor (this repo)** | **0.774** | **0.308** | **0.265** | **0.431** | **0.142** | **0.206** |

Comparison with prior *speaker* methods trained on discretized R2R / RxR viewpoint graphs (those numbers are **not** on the same continuous-env protocol):

<p align="center">
  <img src="figures/table2.png" alt="Paper Table 2: comparison with prior speaker methods" width="100%"/>
</p>

### 2. Frozen VLN follower

Instructions from VTInstructor are executed by a **frozen CorrectNav** follower on R2R-CE Val Unseen. VTInstructor improves frozen-follower success by **14.7 percentage points** over the strongest baseline, and matches or exceeds official human instructions on SR / OSR / NE.

<p align="center">
  <img src="figures/table3.png" alt="Paper Table 5: frozen CorrectNav follower on R2R-CE Val Unseen" width="100%"/>
</p>

| Instruction source | SR↑ | OSR↑ | SPL↑ | NE↓ |
| :--- | :---: | :---: | :---: | :---: |
| Official human instruction | 61.6 | 67.2 | **53.3** | 4.53 |
| Qwen3.5-Plus | 48.6 | 61.8 | 39.0 | 5.70 |
| GPT-5.4 | 35.9 | 51.4 | 28.6 | 6.14 |
| Gemini-3.1-Pro-Preview | 48.5 | 56.4 | 39.9 | 5.59 |
| xAI Grok 4 | 45.5 | 64.8 | 33.4 | 6.18 |
| Claude Opus 4.6 | 21.7 | 48.8 | 15.2 | 9.56 |
| Qwen3.5-397B-A17B | 45.2 | 63.9 | 33.3 | 6.15 |
| Kimi-K2.5 | 43.9 | 58.6 | 33.6 | 6.49 |
| LLaVA-Video-7B-Qwen2 | 30.2 | 44.8 | 24.3 | 7.91 |
| Qwen3-VL-8B-Instruct | 33.4 | 46.8 | 25.9 | 7.09 |
| Qwen3.5-9B | 36.2 | 45.2 | 29.1 | 6.82 |
| GLM-4.1V-9B-Thinking | 31.1 | 42.9 | 24.4 | 7.37 |
| GLM-4.6V | 29.9 | 40.6 | 24.0 | 7.11 |
| **VTInstructor (this repo)** | **63.3** | **70.0** | 52.7 | **4.47** |

### 3. Data augmentation for a VLN follower

Training CorrectNav (LLaVA-Video-7B backbone) with human R2R-CE+RxR-CE data **plus** VTInstructor-generated instructions yields about **+3 SR** on both Val Unseen splits.

<p align="center">
  <img src="figures/table4.png" alt="Paper Tables 6-7: data-augmentation results" width="100%"/>
</p>

**R2R-CE Val Unseen**

| Set | Training data | SR↑ | OSR↑ | SPL↑ | NE↓ |
| :--- | :--- | :---: | :---: | :---: | :---: |
| A | R2R-CE + RxR-CE (human) | 45.1 | 52.3 | 44.6 | 6.20 |
| B | Setting A + VTInstructor-generated | **48.4** | **54.2** | **46.8** | **5.85** |

**RxR-CE Val Unseen**

| Set | Training data | SR↑ | OSR↑ | SPL↑ | NE↓ |
| :--- | :--- | :---: | :---: | :---: | :---: |
| A | R2R-CE + RxR-CE (human) | 41.2 | 51.1 | 39.6 | 8.34 |
| B | Setting A + VTInstructor-generated | **44.4** | **53.2** | **41.3** | **7.74** |

## Files in this repo

| File | Role |
| :--- | :--- |
| `model-0000{1–4}-of-00004.safetensors` | Qwen3-VL-8B weights (~8.8B params, bf16). The VP tensors also appear here because the modules are registered on the model, but the code loads the two `.pt` files below. |
| `vp_encoder.pt` | VP-Encoder CNN (3-channel ribbon/arrow/endpoint mask → features) |
| `vp_adapters.pt` | Gated VP-Adapter at ViT layer 7 |
| `config.json`, tokenizer, preprocessor | Standard Qwen3-VL processor files |
| `figures/` | Paper overview + result tables |

Both `vp_encoder.pt` and `vp_adapters.pt` ship in this repo, so pointing `CKPT` / `EVAL_CKPT` at the downloaded directory is enough — no extra files are needed.

`transformers==4.57.6` is the validated version. Newer `transformers` may break the VP-Adapter injection into Qwen3-VL internals.

Loading **only** with `Qwen3VLForConditionalGeneration.from_pretrained` gives the language/vision backbone; to reproduce paper numbers you must attach the VP modules from the [code repo](https://github.com/TidalHarley/VTInstructor).

## Quick start

```bash
# 1. weights
hf download TidalYang/VTInstructor-8b --local-dir ./VTInstructor-8b

# 2. code
git clone https://github.com/TidalHarley/VTInstructor.git && cd VTInstructor
pip install -r requirements.txt
```

Generate instructions (RGB-only, or with VTP masks if the split has them):

```bash
CKPT=../VTInstructor-8b \
DATA_DIR=/path/to/rendered_split \
DATASET_TYPE=r2rce \
OUT_JSON=outputs/augment/r2r_style.json \
  bash generate/run_generate.sh
```

`DATASET_TYPE=rxrce` switches to the longer RxR-style narration.
`VP_MODE=auto` (default) uses `vp_masks` when present and falls back to RGB otherwise.
`MODEL_DIR` is optional — the processor files in this repo are used by default.
`GEN_CUDA` selects GPUs (default `0`); shards are merged automatically.

To reproduce the paper's benchmark numbers instead (needs rendered val_unseen splits):

```bash
EVAL_CKPT=../VTInstructor-8b \
MODEL_DIR=/path/to/Qwen3-VL-8B-Instruct \
EVAL_CUDA=0 \
  bash eval/run_eval.sh
```

Set `EVAL_CUDA` to the GPU ordinals you actually have — it defaults to all eight.
Scoring additionally needs `pycocoevalcap` and a Java runtime on `PATH` (for METEOR / SPICE).

## Training recipe (how this checkpoint was produced)

1. **SFT** of Qwen3-VL-8B-Instruct + VP-Adapter on filtered R2R-CE / RxR-CE visual trajectories
2. **VP-GRPO** RL with a weighted NLG reward (BLEU-1/4, METEOR, ROUGE-L, CIDEr), updating the VP-Adapter gates and the language decoder while the vision tower and the VP encoder stay frozen → this checkpoint

See the GitHub README for rendering, VTP mask construction, SFT, GRPO, and evaluation.


## Citation

If you use this model, please cite:

```bibtex
@misc{yang2026vtinstructorvisualtrajectoryprompting,
      title={VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments},
      author={Haolin Yang and Yuxing Long and Zihan Yang and Hao Dong},
      year={2026},
      eprint={2608.15284},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2608.15284},
}
```

## License

The code release is [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0), and these weights are distributed under the same terms.

This model was trained on data derived from **R2R / R2R-CE**, **RxR / RxR-CE** and **Matterport3D**, which are distributed under CC BY-NC-SA 3.0 US and the [Matterport3D Terms of Use](http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf). Please respect those terms — the checkpoint is intended for **non-commercial research use**. The [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) backbone license also applies.