README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen3-VL-4B-Instruct
language:
- en
tags:
- qwen3-vl
- video
- video-grounding
- spatio-temporal-video-grounding
- temporal-localization
- parallel-tube-decoding
- ptd
---

# ParallelTubeDecoding-Qwen3-VL-4B

This repository contains the merged Qwen3-VL-4B checkpoint for **Parallel
Tube Decoding (PTD)** from *Locate Anything in Videos: Rethinking Efficient
Generative Spatio-Temporal Video Grounding*.

PTD first predicts the temporal interval of a queried event and then generates
all time-conditioned spatial blocks in parallel. Decoupled Block Attention
allows every spatial block to access the shared video-query context and the
predicted temporal block while preventing dependencies between spatial
blocks. The complete spatial tube is therefore produced in one parallel
decoding round after temporal localization.

- **Code:** [mbzuai-oryx/ParallelTubeDecoding](https://github.com/mbzuai-oryx/ParallelTubeDecoding)
- **Project page:** [Parallel Tube Decoding](https://mbzuai-oryx.github.io/ParallelTubeDecoding/)
- **Base model:** [Qwen/Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct)

## Model details

- Backbone: Qwen3-VL-4B-Instruct
- Method: Parallel Tube Decoding (PTD)
- Tasks: spatio-temporal video grounding and temporal localization
- Spatial representation: 1,001 discrete coordinate tokens over `[0, 1000]`
- Temporal representation: 100 discrete time tokens
- PTD spatial block size: 6
- Checkpoint format: merged model weights in FP16
- Transformers version: `5.12.1`

The checkpoint supports the released PTD inference path and Quantized (NTP)
inference. It does not include data or a separate LoRA adapter.

## Important loading note

Use this checkpoint with the released
[ParallelTubeDecoding](https://github.com/mbzuai-oryx/ParallelTubeDecoding)
code. Standard Transformers loading provides the Qwen3-VL architecture, but
PTD generation and Decoupled Block Attention are implemented by the released
codebase.

```bash
git clone https://github.com/mbzuai-oryx/ParallelTubeDecoding.git
cd ParallelTubeDecoding

conda create -n ptd python=3.11 -y
conda activate ptd
pip install --upgrade pip
pip install -r requirements.txt
```

The release uses:

```text
transformers==5.12.1
```

For evaluation, set the model path to this Hugging Face repository:

```bash
export MODEL="MBZUAI/ParallelTubeDecoding-Qwen3-VL-4B"
```

Then follow the repository's
[evaluation instructions](https://github.com/mbzuai-oryx/ParallelTubeDecoding/blob/main/evaluation/README.md).
The released adapter is based on `lmms-eval` v0.7.1 at commit
`88b23e2bfa16a1edbc16e9e238ed82130b3a4f56`.

## Evaluation configuration

The paper evaluation uses:

```text
fps=2
max_num_frames=64
min_pixels=131072
max_pixels=786432
temporal_patch_size=1
```

For PTD inference, use:

```text
attn_implementation=sdpa
vision_attn_implementation=flash_attention_2
ptd_attn_implementation=flash_attention_2
```

Quantized (NTP) inference uses:

```text
attn_implementation=sdpa
```

FlashAttention-2 must match the installed PyTorch and CUDA/ROCm environment.

## Training

The released model was trained in two stages:

1. Supervised fine-tuning with the joint NTP/MTP formulation on the training
   splits of VidSTG and HC-STVG v1/v2.
2. Localization-aware GRPO with temporal IoU and spatial GIoU/L1 rewards.

Both stages used LoRA with rank 32 and alpha 64 while optimizing the newly
introduced localization-token embeddings. The released checkpoint contains
the merged weights. Charades-STA and ActivityNet Captions were not used for
SFT; results on both are zero-shot.

The training video configuration was:

```text
video_max_pixels=151200
fps=2
max_frames=64
temporal_patch_size=1
```

No training data, GRPO data, or GRPO data-selection code is included with the
model release.

## Results

PTD is evaluated on VidSTG, HC-STVG v1/v2, Charades-STA, and ActivityNet
Captions. On the decoding-efficiency protocol, PTD achieves a Tube Completion
Latency (TCL) of 0.4 seconds and 45.9 Boxes Per Second (BPS). Relative to
unquantized autoregressive decoding, this corresponds to 79x lower TCL and 92x
higher BPS while improving grounding accuracy.

The efficiency measurements use BF16, batch size 1, and a single 64-GB AMD
Instinct MI210 GPU with synchronized decode-only timing. Refer to the paper and
project page for complete benchmark tables and comparisons.

## Intended use

This model is intended for research on:

- spatio-temporal grounding of a referred entity in video;
- temporal localization of described events;
- efficient structured localization with PTD; and
- evaluation on VidSTG, HC-STVG, Charades-STA, and ActivityNet Captions.

Users are responsible for complying with the licenses and terms of the source
videos and evaluation datasets.

## Limitations

The released formulation predicts one continuous temporal interval and one
spatial tube per query. It is not designed for multiple disjoint event
occurrences or multiple simultaneously valid instances. Temporally subtle
state changes can produce ambiguous boundaries, while small, fast-moving, or
occluded targets remain challenging.

## Citation

```bibtex
@misc{rasheed2026locateanythingvideos,
  title  = {Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding},
  author = {Hanoona Rasheed and Haania Siddiqui and Ming-Hsuan Yang and Fahad Shahbaz Khan and Salman Khan},
  year   = {2026}
}
```

## Acknowledgements

This work builds on Qwen3-VL and the Qwen-VL-Series-Finetune training
framework. See the code repository for full acknowledgements.