---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: video-text-to-text
base_model: OpenMOSS-Team/MOSS-VL-Base-0708
tags:
- moss_vl
- feature-extraction
- SFT
- Video-Understanding
- Image-Understanding
- MOSS-VL
- OpenMOSS
- multimodal
- video
- vision-language
- custom_code
---
# MOSS-VL-Instruct-0708
## Introduction
MOSS-VL-Instruct-0708 is the instruction-tuned checkpoint of the MOSS-VL 0708 release, part of the OpenMOSS ecosystem for open visual understanding.
Built on top of MOSS-VL-Base-0708 through supervised fine-tuning (SFT), this checkpoint is designed as a high-performance offline multimodal model. It supports image understanding, OCR, document parsing, visual reasoning, instruction following, and video understanding, with particular strength in long-form video comprehension, temporal reasoning, action recognition, and fine-grained event localization.
The 0708 release keeps the MOSS-VL cross-attention design and a 256K text context window while refreshing the data and instruction-tuning recipe for stronger offline multimodal usage.
## Highlights
- Strong video understanding: designed for long videos, temporal reasoning, action recognition, and second-level event localization.
- General multimodal perception: supports image understanding, fine-grained visual recognition, OCR, and document analysis.
- Reliable instruction following: SFT aligns the base checkpoint with user instructions across image, video, and text tasks.
- Open model family: released together with MOSS-VL-Base-0708 for continued pretraining, fine-tuning, and applied research.
## Model Architecture
MOSS-VL-Instruct-0708 adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. The model processes images, videos, and text in a unified pipeline and uses cross-attention layers to connect language tokens with visual representations.
Key configuration details:
| Item | Value |
| --- | --- |
| Parameters | 11B |
| Tensor type | BF16 |
| Context length | 256K |
| Vision patch size | 16 |
| Temporal patch size | 1 |
| Default video FPS | 1.0 |
| Default max video frames | 256 |
## Absolute Timestamps
For video inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the model reason about event order, duration, pacing, and temporal localization instead of relying only on frame order.
## Cross-attention RoPE (XRoPE)
MOSS-VL uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image and video reasoning.
## Model Performance
MOSS-VL-Instruct-0708 is intended for offline multimodal evaluation across visual perception, multimodal reasoning, OCR/document understanding, and video understanding. Detailed benchmark tables for the 0708 release will be maintained in the MOSS-VL project resources.
For the previous public checkpoint, see [MOSS-VL-Instruct-0408](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408).
## Quickstart
### Installation
Clone the MOSS-VL repository and install the project requirements:
```bash
git clone https://github.com/OpenMOSS/MOSS-VL.git
cd MOSS-VL
conda create -n moss_vl python=3.12 pip -y
conda activate moss_vl
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt
```
### Load Model
```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
checkpoint = "OpenMOSS-Team/MOSS-VL-Instruct-0708"
processor = AutoProcessor.from_pretrained(
checkpoint,
trust_remote_code=True,
frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
checkpoint,
trust_remote_code=True,
device_map="auto",
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
)
```
### Run Inference
Single-image Inference
```python
image_path = "data/example_image.jpg"
prompt = "Describe this image."
text = model.offline_image_generate(
processor,
prompt=prompt,
image=image_path,
shortest_edge=4096,
longest_edge=16777216,
multi_image_max_pixels=201326592,
patch_size=16,
temporal_patch_size=1,
merge_size=2,
image_mean=[0.5, 0.5, 0.5],
image_std=[0.5, 0.5, 0.5],
max_new_tokens=256,
temperature=1.0,
top_k=50,
top_p=1.0,
repetition_penalty=1.0,
do_sample=False,
vision_chunked_length=64,
)
print(text)
```
Single-video Inference
```python
video_path = "data/example_video.mp4"
prompt = "Describe this video."
text = model.offline_video_generate(
processor,
prompt=prompt,
video=video_path,
shortest_edge=4096,
longest_edge=16777216,
video_max_pixels=201326592,
patch_size=16,
temporal_patch_size=1,
merge_size=2,
video_fps=1.0,
min_frames=1,
max_frames=256,
num_extract_threads=4,
image_mean=[0.5, 0.5, 0.5],
image_std=[0.5, 0.5, 0.5],
max_new_tokens=256,
temperature=1.0,
top_k=50,
top_p=1.0,
repetition_penalty=1.0,
do_sample=False,
vision_chunked_length=64,
)
print(text)
```
Batched Offline Inference
`offline_batch_generate` accepts independent image/video/text queries. Queries in the same batch should share the same `media_kwargs` and `generate_kwargs`.
```python
queries = [
{
"prompt": "Describe sample A.",
"images": [],
"videos": ["data/sample_a.mp4"],
"media_kwargs": {
"video_fps": 1.0,
"min_frames": 8,
"max_frames": 256,
},
"generate_kwargs": {
"temperature": 1.0,
"top_k": 50,
"top_p": 1.0,
"max_new_tokens": 256,
"repetition_penalty": 1.0,
"do_sample": False,
},
},
{
"prompt": "Describe sample B.",
"images": [],
"videos": ["data/sample_b.mp4"],
"media_kwargs": {
"video_fps": 1.0,
"min_frames": 8,
"max_frames": 256,
},
"generate_kwargs": {
"temperature": 1.0,
"top_k": 50,
"top_p": 1.0,
"max_new_tokens": 256,
"repetition_penalty": 1.0,
"do_sample": False,
},
},
]
with torch.no_grad():
result = model.offline_batch_generate(
processor,
queries,
vision_chunked_length=64,
)
texts = [item["text"] for item in result["results"]]
print(texts)
```
## Related Checkpoints
| Model | Parameters | Context | Usage | Hugging Face |
| --- | ---: | ---: | --- | --- |
| MOSS-VL-Realtime | 11B | 256K | Realtime streaming video interaction | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime |
| MOSS-VL-Instruct | 11B | 256K | Offline multimodal instruction following | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0708 |
| MOSS-VL-Base | 11B | 256K | Continued pretraining and fine-tuning | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0708 |
| MOSS-VL-Instruct-0408 | 11B | 256K | Previous instruction-tuned checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408 |
| MOSS-VL-Base-0408 | 11B | 256K | Previous base checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408 |
## Limitations and Future Work
MOSS-VL-Instruct-0708 is optimized for general offline multimodal understanding. Very dense videos, highly specialized domains, precise small-text OCR, and tasks requiring strict numerical reasoning may still require task-specific prompting, sampling choices, or fine-tuning.
We are continuing to improve mathematical reasoning, code reasoning, RL post-training, and broader task-specific evaluations for future MOSS-VL releases.
## Citation
```bibtex
@misc{moss_vl_2026,
title = {{MOSS-VL Technical Report}},
author = {OpenMOSS Team},
year = {2026},
howpublished = {\url{https://github.com/OpenMOSS/MOSS-VL}},
note = {GitHub repository}
}
```