Upload MOSS-VL-Realtime-0708 release

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/architecture.png filter=lfs diff=lfs merge=lfs -text
+assets/benchmark-streaming.png filter=lfs diff=lfs merge=lfs -text
+assets/logo.png filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,371 @@
 ---
 license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: video-text-to-text
+base_model: OpenMOSS-Team/MOSS-VL-Base-0708
+tags:
+- moss_vl
+- feature-extraction
+- Realtime
+- Streaming
+- Video-Understanding
+- Image-Understanding
+- MOSS-VL
+- OpenMOSS
+- multimodal
+- video
+- vision-language
+- custom_code
 ---
+
+<p align="center">
+  <img src="assets/logo.png" width="300" alt="MOSS-VL"/>
+</p>
+
+# MOSS-VL-Realtime-0708
+
+## Introduction
+
+MOSS-VL-Realtime-0708 is the realtime streaming checkpoint of the MOSS-VL 0708 release, part of the OpenMOSS ecosystem for open visual understanding.
+
+Unlike offline video-language models that first read a complete video and then answer, MOSS-VL-Realtime-0708 is designed for continuous video streams. It can perceive incoming frames and generate text in parallel, support multi-turn realtime interaction, answer questions at arbitrary moments in the stream, and decide whether to speak or keep observing when the visual evidence is not yet sufficient.
+
+The 0708 release keeps the MOSS-VL cross-attention design and a 256K text context window while adding realtime streaming data and an inference interface for timestamped frame-by-frame input.
+
+## Highlights
+
+- Interruptible realtime interaction: users can ask questions at any timestamp in a running video stream, and the model answers based on the frames observed so far.
+- Proactive silence: the model can emit `<|silence|>` and continue observing when no important visual event has happened or the context is insufficient.
+- Dynamic correction: as new frames arrive, the model can update or correct earlier responses instead of being locked to an initial interpretation.
+- Native timestamped frames: each streamed frame is associated with an absolute timestamp, helping the model reason about event order, duration, pacing, and fine-grained temporal localization.
+- Open MOSS-VL family: released together with MOSS-VL-Instruct-0708 and MOSS-VL-Base-0708 for offline use, continued pretraining, fine-tuning, and applied research.
+
+## Realtime Interaction
+
+MOSS-VL-Realtime-0708 exposes a model-owned streaming loop through `create_realtime_session(...)`, `online_generate(...)`, and `real_time_generate(...)`.
+
+The recommended deployment API is `create_realtime_session(...)`. A service or application owns the video capture pipeline, converts camera, screen, or video-file input into PIL-compatible frames, and pushes each frame with a non-decreasing timestamp:
+
+- `session.push_frame(image, timestamp=...)` appends one visual frame.
+- `session.push_prompt("...")` appends a user question while the stream is running.
+- `session.push_prompt_frame(prompt, image, timestamp=...)` aligns a prompt with a specific frame.
+- `session.poll_output(...)` or `session.stream_outputs(...)` returns incremental text chunks.
+
+For backend systems that already use queues, `online_generate(...)` accepts dictionaries containing frames, prompts, events, reset controls, and stop controls.
+
+## Model Architecture
+
+MOSS-VL-Realtime-0708 adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. This design is important for realtime usage because incoming visual content can be integrated into the running generation context without forcing the model into a strictly offline "load all frames, then answer" workflow.
+
+<p align="center">
+  <img src="assets/architecture.png" alt="MOSS-VL Architecture" width="100%"/>
+</p>
+
+Key configuration details:
+
+| Item | Value |
+| --- | --- |
+| Parameters | 11B |
+| Tensor type | BF16 |
+| Context length | 256K |
+| Vision patch size | 16 |
+| Temporal patch size | 1 |
+| Default video FPS | 1.0 |
+| Default max video frames | 256 |
+| Realtime frame format | PIL-compatible image plus timestamp |
+| Realtime session scope | One active realtime session per model instance |
+
+## Absolute Timestamps
+
+For video and realtime frame inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the model reason about when an event happens, how long it lasts, and how the scene changes over time instead of relying only on frame order.
+
+## Cross-attention RoPE (XRoPE)
+
+MOSS-VL uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image, offline video, and realtime streaming video reasoning.
+
+## Model Performance
+
+MOSS-VL-Realtime-0708 is designed for streaming video understanding benchmarks where questions can arrive before a full video has been observed and correct answers may change as the scene evolves. It targets realtime interaction quality, proactive silence, and dynamic response updates in addition to standard video understanding accuracy.
+
+<p align="center">
+  <img src="assets/benchmark-streaming.png" alt="MOSS-VL Streaming Benchmark" width="100%"/>
+</p>
+
+Detailed benchmark tables and comparisons for the 0708 release will be maintained in the MOSS-VL project resources.
+
+## Quickstart
+
+### Installation
+
+Clone the MOSS-VL repository and install the project requirements:
+
+```bash
+git clone https://github.com/OpenMOSS/MOSS-VL.git
+cd MOSS-VL
+conda create -n moss_vl python=3.12 pip -y
+conda activate moss_vl
+pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt
+```
+
+### Load Model
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoProcessor
+
+checkpoint = "OpenMOSS-Team/MOSS-VL-Realtime-0708"
+
+processor = AutoProcessor.from_pretrained(
+    checkpoint,
+    trust_remote_code=True,
+    frame_extract_num_threads=1,
+)
+model = AutoModelForCausalLM.from_pretrained(
+    checkpoint,
+    trust_remote_code=True,
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+    attn_implementation="flash_attention_2",
+)
+model.eval()
+```
+
+If FlashAttention is unavailable in your environment, pass `attn_implementation="eager"` when loading the model.
+
+### Run Realtime Streaming
+
+The session API keeps generation active while new frames and prompts are appended.
+
+```python
+import time
+from PIL import Image
+
+session = model.create_realtime_session(
+    processor,
+    initial_prompt=(
+        "As the video streams frame by frame, describe important changes as they happen. "
+        "Stay silent when there is no relevant update."
+    ),
+    frame_queue_size=256,
+    max_tokens_per_turn=12,
+    max_new_tokens=4096,
+    do_sample=False,
+)
+
+frame_paths = [
+    "data/frame_0001.jpg",
+    "data/frame_0002.jpg",
+    "data/frame_0003.jpg",
+]
+
+try:
+    session.start()
+
+    for index, frame_path in enumerate(frame_paths):
+        image = Image.open(frame_path).convert("RGB")
+        session.push_frame(image, timestamp=index / 1.0)
+
+        while True:
+            chunk = session.poll_output(timeout=0.0)
+            if chunk is None:
+                break
+            print(chunk, end="", flush=True)
+
+        time.sleep(1.0)
+
+    session.push_prompt("What changed in the latest frames?")
+
+    for chunk in session.stream_outputs(poll_interval=0.1):
+        print(chunk, end="", flush=True)
+finally:
+    session.close()
+```
+
+Frame timestamps are measured in seconds and must be non-decreasing within a session. The input producer can be a camera, screen capture, decoded video file, browser frame sampler, or any other source that yields images with timestamps.
+
+### Queue-style Online Inference
+
+`online_generate(...)` is useful for services that separate frame production and model inference through queues.
+
+```python
+import queue
+import threading
+from PIL import Image
+
+input_queue = queue.Queue()
+output_queue = queue.Queue()
+
+worker = threading.Thread(
+    target=model.online_generate,
+    args=(processor, input_queue, output_queue),
+    kwargs={
+        "frame_queue_size": 256,
+        "max_tokens_per_turn": 12,
+        "max_new_tokens": 4096,
+        "do_sample": False,
+    },
+    daemon=True,
+)
+worker.start()
+
+input_queue.put({
+    "initial_prompt": "Answer only when the streamed video provides enough evidence.",
+})
+
+input_queue.put({"frame": Image.open("data/frame_0001.jpg").convert("RGB"), "timestamp": 0.0})
+input_queue.put({"frame": Image.open("data/frame_0002.jpg").convert("RGB"), "timestamp": 1.0})
+input_queue.put({"prompt": "What is happening now?"})
+
+try:
+    while True:
+        chunk = output_queue.get(timeout=0.5)
+        print(chunk, end="", flush=True)
+except queue.Empty:
+    pass
+
+input_queue.put({"stop_online_generate": True})
+worker.join()
+```
+
+Each queue item can contain `frame` or `image`, `timestamp`, `prompt`, `frames`, `event`, `events`, `initial_prompt`, `system_prompt`, `generate_kwargs`, `reset_session`, or stop controls such as `stop_online_generate`.
+
+### Offline Inference Compatibility
+
+MOSS-VL-Realtime-0708 also keeps the offline helper APIs for image and video prompts. For purely offline use, MOSS-VL-Instruct-0708 is usually the preferred checkpoint, but the realtime checkpoint can still process complete image and video inputs.
+
+<details>
+<summary><b>Single-video Offline Inference</b></summary>
+
+```python
+video_path = "data/example_video.mp4"
+prompt = "Describe this video."
+
+text = model.offline_video_generate(
+    processor,
+    prompt=prompt,
+    video=video_path,
+    shortest_edge=4096,
+    longest_edge=16777216,
+    video_max_pixels=201326592,
+    patch_size=16,
+    temporal_patch_size=1,
+    merge_size=2,
+    video_fps=1.0,
+    min_frames=1,
+    max_frames=256,
+    num_extract_threads=4,
+    image_mean=[0.5, 0.5, 0.5],
+    image_std=[0.5, 0.5, 0.5],
+    max_new_tokens=256,
+    temperature=1.0,
+    top_k=50,
+    top_p=1.0,
+    repetition_penalty=1.0,
+    do_sample=False,
+    vision_chunked_length=64,
+)
+
+print(text)
+```
+
+</details>
+
+<details>
+<summary><b>Batched Offline Inference</b></summary>
+
+`offline_batch_generate` accepts independent image/video/text queries. Queries in the same batch should share the same `media_kwargs` and `generate_kwargs`.
+
+```python
+queries = [
+    {
+        "prompt": "Describe sample A.",
+        "images": [],
+        "videos": ["data/sample_a.mp4"],
+        "media_kwargs": {
+            "video_fps": 1.0,
+            "min_frames": 8,
+            "max_frames": 256,
+        },
+        "generate_kwargs": {
+            "temperature": 1.0,
+            "top_k": 50,
+            "top_p": 1.0,
+            "max_new_tokens": 256,
+            "repetition_penalty": 1.0,
+            "do_sample": False,
+        },
+    },
+    {
+        "prompt": "Describe sample B.",
+        "images": [],
+        "videos": ["data/sample_b.mp4"],
+        "media_kwargs": {
+            "video_fps": 1.0,
+            "min_frames": 8,
+            "max_frames": 256,
+        },
+        "generate_kwargs": {
+            "temperature": 1.0,
+            "top_k": 50,
+            "top_p": 1.0,
+            "max_new_tokens": 256,
+            "repetition_penalty": 1.0,
+            "do_sample": False,
+        },
+    },
+]
+
+with torch.no_grad():
+    result = model.offline_batch_generate(
+        processor,
+        queries,
+        vision_chunked_length=64,
+    )
+
+texts = [item["text"] for item in result["results"]]
+print(texts)
+```
+
+</details>
+
+## Related Checkpoints
+
+| Model | Parameters | Context | Usage | Hugging Face |
+| --- | ---: | ---: | --- | --- |
+| MOSS-VL-Realtime-0708 | 11B | 256K | Realtime streaming video interaction | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime-0708 |
+| MOSS-VL-Instruct-0708 | 11B | 256K | Offline multimodal instruction following | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0708 |
+| MOSS-VL-Base-0708 | 11B | 256K | Continued pretraining and fine-tuning | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0708 |
+| MOSS-VL-Instruct-0408 | 11B | 256K | Previous instruction-tuned checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408 |
+| MOSS-VL-Base-0408 | 11B | 256K | Previous base checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408 |
+
+## Limitations and Future Work
+
+MOSS-VL-Realtime-0708 is optimized for timestamped frame-by-frame streaming, but production latency depends on GPU hardware, frame sampling rate, transport overhead, and decoding speed. One model instance supports one active realtime session. The default frame queue bounds latency by dropping older pending frames when needed.
+
+The model may emit realtime control tokens such as `<|silence|>`, `<|round_start|>`, and `<|round_end|>` depending on the application protocol. Downstream services should filter or render these tokens according to their UI needs.
+
+We are continuing to improve realtime response timing, dynamic correction, broader streaming evaluations, RL post-training, and task-specific deployment recipes for future MOSS-VL releases.
+
+## Citation
+
+```bibtex
+@misc{moss_vl_2026,
+  title         = {{MOSS-VL Technical Report}},
+  author        = {OpenMOSS Team},
+  year          = {2026},
+  howpublished  = {\url{https://github.com/OpenMOSS/MOSS-VL}},
+  note          = {GitHub repository}
+}
+
+@misc{mossvideopreview2026,
+  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},
+  author        = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},
+  year          = {2026},
+  eprint        = {2606.07639},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.CV},
+  url           = {https://arxiv.org/abs/2606.07639}
+}
+```

chat_template.json

@@ -0,0 +1,3 @@
+{
+    "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' }}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- for message in messages %}\n    {%- if message.role == \"user\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|image|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|video|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content_item in message.content %}\n                {%- if 'text' in content_item %}\n                    {{- content_item.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and message.content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|image|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|video|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n"
+}

config.json

@@ -0,0 +1,83 @@
+{
+  "architectures": [
+    "MossVLForConditionalGeneration"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_moss_vl.MossVLConfig",
+    "AutoModel": "modeling_moss_vl.MossVLForConditionalGeneration",
+    "AutoModelForCausalLM": "modeling_moss_vl.MossVLForConditionalGeneration"
+  },
+  "dtype": "bfloat16",
+  "image_token_id": 151655,
+  "model_type": "moss_vl",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "cross_attention_layers": [
+      2,
+      6,
+      10,
+      14,
+      18,
+      22,
+      26,
+      30,
+      34,
+      38,
+      42,
+      46
+    ],
+    "dtype": "bfloat16",
+    "eos_token_id": 151645,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "initializer_range": 0.02,
+    "intermediate_size": 12288,
+    "max_position_embeddings": 262144,
+    "model_type": "moss_vl_text",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        24,
+        20,
+        20
+      ],
+      "rope_type": "default"
+    },
+    "rope_theta": 5000000,
+    "use_cache": true,
+    "vocab_size": 151936
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.3",
+  "video_token_id": 151656,
+  "vision_config": {
+    "deepstack_visual_indexes": [
+      8,
+      16,
+      24
+    ],
+    "depth": 27,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "model_type": "moss_vl_vision",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 4096,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 1
+  },
+  "vision_end_token_id": 151653,
+  "vision_seq_pad_multiple": 1,
+  "vision_start_token_id": 151652
+}

configuration_moss_vl.py

@@ -0,0 +1,164 @@
+# coding=utf-8
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""MossVL model configuration"""
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.modeling_rope_utils import rope_config_validation
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class MossVLVisionConfig(PretrainedConfig):
+    """
+    Configuration for MossVL Vision Model
+    """
+
+    model_type = "moss_vl_vision"
+    base_config_key = "vision_config"
+
+    def __init__(
+        self,
+        depth=27,
+        hidden_size=1152,
+        hidden_act="gelu_pytorch_tanh",
+        intermediate_size=4304,
+        num_heads=16,
+        in_channels=3,
+        patch_size=16,
+        spatial_merge_size=2,
+        temporal_patch_size=1,
+        out_hidden_size=3584,
+        num_position_embeddings=2304,
+        deepstack_visual_indexes=[8, 16, 24],
+        initializer_range=0.02,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.depth = depth
+        self.hidden_size = hidden_size
+        self.hidden_act = hidden_act
+        self.intermediate_size = intermediate_size
+        self.num_heads = num_heads
+        self.in_channels = in_channels
+        self.patch_size = patch_size
+        self.spatial_merge_size = spatial_merge_size
+        self.temporal_patch_size = temporal_patch_size
+        self.out_hidden_size = out_hidden_size
+        self.num_position_embeddings = num_position_embeddings
+        self.initializer_range = initializer_range
+        self.deepstack_visual_indexes = deepstack_visual_indexes
+
+
+class MossVLTextConfig(PretrainedConfig):
+    """
+    Configuration for MossVL Text Model
+    """
+
+    model_type = "moss_vl_text"
+    base_config_key = "text_config"
+
+    def __init__(
+        self,
+        vocab_size=151936,
+        hidden_size=4096,
+        intermediate_size=22016,
+        num_hidden_layers=32,
+        num_attention_heads=32,
+        num_key_value_heads=32,
+        head_dim=128,
+        hidden_act="silu",
+        max_position_embeddings=128000,
+        initializer_range=0.02,
+        rms_norm_eps=1e-6,
+        use_cache=True,
+        tie_word_embeddings=False,
+        rope_theta=5000000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        # Cross attention specific
+        cross_attention_layers=None,  # List of layer indices to insert cross attention
+        **kwargs,
+    ):
+
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+
+        # for backward compatibility
+        if num_key_value_heads is None:
+            num_key_value_heads = num_attention_heads
+
+        self.num_key_value_heads = num_key_value_heads
+        self.head_dim = head_dim
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+
+        rope_config_validation(self, ignore_keys={"mrope_section", "mrope_interleaved"})
+        self.cross_attention_layers = cross_attention_layers or [2, 6, 10, 14, 18, 22, 26, 30, 34, 38, 42, 46]
+        super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)
+
+class MossVLConfig(PretrainedConfig):
+    """
+    Configuration for MossVL Model
+    """
+
+    model_type = "moss_vl"
+    sub_configs = {"vision_config": MossVLVisionConfig, "text_config": MossVLTextConfig}
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        text_config=None,
+        vision_config=None,
+        image_token_id=151655,
+        video_token_id=151656,
+        vision_start_token_id=151652,
+        vision_end_token_id=151653,
+        vision_seq_pad_multiple=8,
+        tie_word_embeddings=False,
+        **kwargs,
+    ):
+        if isinstance(vision_config, dict):
+            self.vision_config = self.sub_configs["vision_config"](**vision_config)
+        elif vision_config is None:
+            self.vision_config = self.sub_configs["vision_config"]()
+
+        if isinstance(text_config, dict):
+            self.text_config = self.sub_configs["text_config"](**text_config)
+        elif text_config is None:
+            self.text_config = self.sub_configs["text_config"]()
+
+        self.image_token_id = image_token_id
+        self.video_token_id = video_token_id
+        self.vision_start_token_id = vision_start_token_id
+
+
+        self.vision_end_token_id = vision_end_token_id
+        self.vision_seq_pad_multiple = vision_seq_pad_multiple
+        super().__init__(**kwargs, tie_word_embeddings=tie_word_embeddings)
+        
+
+__all__ = ["MossVLConfig", "MossVLTextConfig"]

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "transformers_version": "4.57.3"
+}

model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fe2d46a92e3c036e8dfbb2519f65415f8f1ab4f731ffdcb9c8bbff4e66fc3d6
+size 5274500800

model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18f3579907933bad721053b3ac405c51045db4539fedcc7ee3c8db79e79f70e5
+size 5360568508

model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cecf1e02afe5c7250bb43d6b6e99c7cb430a02afc07a2d697ab1dc4b5f784567
+size 5360577920

model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57d794a266a1283d209d054114d9888a6a6b36a0af87fbf314c92b76c89dc294
+size 5366957460

model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34009172ba732eb447caf1293126bba35eb934a622677ac7c24bb2d6af462b0d
+size 1310247928

model.safetensors.index.json

@@ -0,0 +1,902 @@
+{
+  "metadata": {
+    "total_size": 22672742416
+  },
+  "weight_map": {
+    "model.language_model.embed_tokens.weight": "model-00001-of-00005.safetensors",
+    "model.separator_token": "model-00001-of-00005.safetensors",
+    "model.language_model.norm.weight": "model-00001-of-00005.safetensors",
+    "lm_head.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.0.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.1.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.cross_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.cross_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.cross_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.cross_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.cross_attn.q_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.cross_attn.k_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.cross_attn_attn_gate": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.cross_attn_mlp_gate": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.2.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.3.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.4.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.5.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.cross_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.cross_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.cross_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.cross_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.cross_attn.q_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.cross_attn.k_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.cross_attn_attn_gate": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.cross_attn_mlp_gate": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.mlp.gate_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.mlp.up_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.6.mlp.down_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.7.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.7.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.7.self_attn.k_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.7.self_attn.v_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.7.self_attn.o_proj.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.7.self_attn.q_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.7.self_attn.k_norm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.7.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.language_model.layers.7.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.7.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.7.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.8.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.9.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.cross_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.cross_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.cross_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.cross_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.cross_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.cross_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.cross_attn_attn_gate": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.cross_attn_mlp_gate": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.10.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.11.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.12.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.13.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.cross_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.cross_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.cross_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.cross_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.cross_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.cross_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.cross_attn_attn_gate": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.cross_attn_mlp_gate": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.14.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.15.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.16.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.17.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.cross_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.cross_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.cross_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.cross_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.cross_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.cross_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.cross_attn_attn_gate": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.cross_attn_mlp_gate": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.18.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.19.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.self_attn.v_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.self_attn.o_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.self_attn.q_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.self_attn.k_norm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.mlp.gate_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.mlp.up_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.20.mlp.down_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.21.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.21.self_attn.q_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.21.self_attn.k_proj.weight": "model-00002-of-00005.safetensors",
+    "model.language_model.layers.21.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.21.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.21.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.21.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.21.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.21.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.21.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.21.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.cross_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.cross_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.cross_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.cross_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.cross_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.cross_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.cross_attn_attn_gate": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.cross_attn_mlp_gate": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.22.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.23.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.24.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.25.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.cross_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.cross_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.cross_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.cross_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.cross_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.cross_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.cross_attn_attn_gate": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.cross_attn_mlp_gate": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.26.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.27.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.28.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.29.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.cross_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.cross_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.cross_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.cross_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.cross_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.cross_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.cross_attn_attn_gate": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.cross_attn_mlp_gate": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.30.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.31.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.32.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.self_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.self_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.self_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.self_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.self_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.self_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.33.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.cross_attn.q_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.cross_attn.k_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.cross_attn.v_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.cross_attn.o_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.cross_attn.q_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.cross_attn.k_norm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.post_attention_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.cross_attn_attn_gate": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.cross_attn_mlp_gate": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.mlp.gate_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.mlp.up_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.34.mlp.down_proj.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.35.input_layernorm.weight": "model-00003-of-00005.safetensors",
+    "model.language_model.layers.35.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.35.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.35.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.35.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.35.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.35.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.35.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.35.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.35.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.35.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.36.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.37.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.cross_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.cross_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.cross_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.cross_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.cross_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.cross_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.cross_attn_attn_gate": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.cross_attn_mlp_gate": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.38.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.39.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.40.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.41.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.cross_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.cross_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.cross_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.cross_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.cross_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.cross_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.cross_attn_attn_gate": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.cross_attn_mlp_gate": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.42.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.43.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.44.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.45.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.cross_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.cross_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.cross_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.cross_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.cross_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.cross_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.cross_attn_attn_gate": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.cross_attn_mlp_gate": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.46.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.input_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.self_attn.q_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.self_attn.k_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.self_attn.v_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.self_attn.o_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.self_attn.q_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.self_attn.k_norm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.post_attention_layernorm.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.mlp.gate_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.mlp.up_proj.weight": "model-00004-of-00005.safetensors",
+    "model.language_model.layers.47.mlp.down_proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.patch_embed.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.patch_embed.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.pos_embed.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.0.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.1.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.2.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.3.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.4.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.5.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.6.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.7.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.8.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.9.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.attn.proj.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.attn.proj.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.norm2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.norm2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.mlp.linear_fc1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.mlp.linear_fc1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.mlp.linear_fc2.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.10.mlp.linear_fc2.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.11.norm1.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.11.norm1.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.11.attn.qkv.weight": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.11.attn.qkv.bias": "model-00004-of-00005.safetensors",
+    "model.visual.blocks.11.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.11.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.11.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.11.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.11.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.11.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.11.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.11.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.12.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.13.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.14.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.15.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.16.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.17.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.18.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.19.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.20.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.21.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.22.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.23.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.24.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.25.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.norm1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.norm1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.attn.qkv.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.attn.qkv.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.attn.proj.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.attn.proj.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.norm2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.norm2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.mlp.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.mlp.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.mlp.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.blocks.26.mlp.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.merger.linear_fc1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.merger.linear_fc1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.merger.linear_fc2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.merger.linear_fc2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.merger.norms.0.weight": "model-00005-of-00005.safetensors",
+    "model.visual.merger.norms.0.bias": "model-00005-of-00005.safetensors",
+    "model.visual.merger.norms.1.weight": "model-00005-of-00005.safetensors",
+    "model.visual.merger.norms.1.bias": "model-00005-of-00005.safetensors",
+    "model.visual.merger.norms.2.weight": "model-00005-of-00005.safetensors",
+    "model.visual.merger.norms.2.bias": "model-00005-of-00005.safetensors",
+    "model.visual.merger.norms.3.weight": "model-00005-of-00005.safetensors",
+    "model.visual.merger.norms.3.bias": "model-00005-of-00005.safetensors"
+  }
+}

preprocessor_config.json

@@ -0,0 +1,26 @@
+{
+    "auto_map": {
+        "AutoProcessor": "processing_moss_vl.MossVLProcessor",
+        "AutoImageProcessor": "processing_moss_vl.MossVLImageProcessorFast"
+    },
+    "size": {
+        "longest_edge": 16777216,
+        "shortest_edge": 4096
+    },
+    "multi_image_max_pixels": 65536000,
+    "patch_size": 16,
+    "temporal_patch_size": 1,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "MossVLProcessor",
+    "image_processor_type": "MossVLImageProcessorFast"
+}

processing_moss_vl.py

@@ -0,0 +1,1143 @@
+# coding=utf-8
+# Copyright 2025 The FNLP Vision Team and The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Processor class for Moss-VL.
+"""
+
+from typing import Any, Dict, List, Optional, Union
+
+import numpy as np
+import torch
+from torchvision.transforms.v2 import functional as F
+from PIL import Image
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.image_utils import ImageInput, SizeDict
+from transformers.image_processing_utils_fast import group_images_by_shape, reorder_images
+from transformers.utils import TensorType
+from transformers.processing_utils import (
+    ImagesKwargs,
+    ProcessingKwargs,
+    ProcessorMixin,
+    Unpack,
+    VideosKwargs,
+)
+from transformers.tokenization_utils_base import PreTokenizedInput, TextInput
+from transformers.utils import logging
+from transformers.models.qwen2_vl.image_processing_qwen2_vl_fast import Qwen2VLImageProcessorFast
+from transformers.models.qwen2_vl.image_processing_qwen2_vl import smart_resize
+
+
+logger = logging.get_logger(__name__)
+
+
+class MossVLImageProcessorFast(Qwen2VLImageProcessorFast):
+    """
+    Custom image processor that overrides _preprocess to support multi_image_max_pixels.
+    Inherits from Qwen2VLImageProcessorFast.
+    """
+    # Multi-image batch total pixels limit (read from config)
+    multi_image_max_pixels = None
+    
+
+    def _preprocess(
+        self,
+        images: list["torch.Tensor"],
+        do_resize: bool,
+        size: SizeDict,
+        interpolation: Optional["F.InterpolationMode"],
+        do_rescale: bool,
+        rescale_factor: float,
+        do_normalize: bool,
+        image_mean: Optional[Union[float, list[float]]],
+        image_std: Optional[Union[float, list[float]]],
+        patch_size: int,
+        temporal_patch_size: int,
+        merge_size: int,
+        disable_grouping: Optional[bool],
+        return_tensors: Optional[Union[str, TensorType]],
+        **kwargs,
+    ):
+        """Override _preprocess to use custom smart_resize with batch-level max_pixels.
+        
+        multi_image_max_pixels is treated as a batch-level total budget, proportionally allocated
+        to each image based on its original pixel count. min_pixels remains a per-image
+        constraint. multi_image_max_pixels can be configured separately from longest_edge.
+        """
+        min_pixels = size["shortest_edge"]
+        max_pixels = size["longest_edge"]  # Per-image upper limit
+        # Use multi_image_max_pixels if configured, otherwise fall back to longest_edge
+        multi_image_max_pixels = getattr(self, "multi_image_max_pixels", None) or max_pixels
+        
+        # Calculate total original pixels across all images in the batch
+        # This is used to proportionally allocate max_pixels to each image
+        total_original_pixels = sum(img.shape[-2] * img.shape[-1] for img in images)
+
+        # Group images by size for batched resizing
+        grouped_images, grouped_images_index = group_images_by_shape(images, disable_grouping=disable_grouping)
+        resized_images_grouped = {}
+        for shape, stacked_images in grouped_images.items():
+            height, width = stacked_images.shape[-2:]
+            if do_resize:
+                # Calculate proportional max_pixels for images with this shape
+                # Each image's max_pixels is allocated based on its proportion of total pixels
+                original_pixels = height * width
+                if total_original_pixels > 0:
+                    proportion = original_pixels / total_original_pixels
+                    proportional_max_pixels = int(multi_image_max_pixels * proportion)
+                else:
+                    proportional_max_pixels = multi_image_max_pixels
+                
+                # Ensure proportional max_pixels is within [min_pixels, max_pixels] range
+                # min_pixels: per-image lower limit (shortest_edge)
+                # max_pixels: per-image upper limit (longest_edge)
+                proportional_max_pixels = max(proportional_max_pixels, min_pixels)
+                proportional_max_pixels = min(proportional_max_pixels, max_pixels)
+                
+                resized_height, resized_width = smart_resize(
+                    height,
+                    width,
+                    factor=patch_size * merge_size,
+                    min_pixels=min_pixels,
+                    max_pixels=proportional_max_pixels,
+                )
+                stacked_images = self.resize(
+                    image=stacked_images,
+                    size=SizeDict(height=resized_height, width=resized_width),
+                    interpolation=interpolation,
+                )
+            resized_images_grouped[shape] = stacked_images
+        resized_images = reorder_images(resized_images_grouped, grouped_images_index)
+        
+        # Warn if multi-image batch exceeds multi_image_max_pixels due to min_pixels constraint
+        if len(images) > 1:
+            total_resized_pixels = sum(img.shape[-2] * img.shape[-1] for img in resized_images)
+            if total_resized_pixels > multi_image_max_pixels:
+                logger.warning_once(
+                    f"Multi-image batch total pixels ({total_resized_pixels}) exceeds multi_image_max_pixels ({multi_image_max_pixels}). "
+                    f"This may happen when image_count * min_pixels > multi_image_max_pixels."
+                )
+
+        # Group images by size for further processing
+        # Needed in case do_resize is False, or resize returns images with different sizes
+        grouped_images, grouped_images_index = group_images_by_shape(resized_images, disable_grouping=disable_grouping)
+        processed_images_grouped = {}
+        processed_grids = {}
+        for shape, stacked_images in grouped_images.items():
+            resized_height, resized_width = stacked_images.shape[-2:]
+            # Fused rescale and normalize
+            patches = self.rescale_and_normalize(
+                stacked_images, do_rescale, rescale_factor, do_normalize, image_mean, image_std
+            )
+            if patches.ndim == 4:
+                # add a temporal dimension if we have images
+                patches = patches.unsqueeze(1)
+            if patches.shape[1] % temporal_patch_size != 0:
+                repeats = patches[:, -1:].repeat(1, temporal_patch_size - 1, 1, 1, 1)
+                patches = torch.cat([patches, repeats], dim=1)
+            batch_size, grid_t, channel = patches.shape[:3]
+            grid_t = grid_t // temporal_patch_size
+            grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+
+            patches = patches.view(
+                batch_size,
+                grid_t,
+                temporal_patch_size,
+                channel,
+                grid_h // merge_size,
+                merge_size,
+                patch_size,
+                grid_w // merge_size,
+                merge_size,
+                patch_size,
+            )
+            # Reorder dimensions to group grid and patch information for subsequent flattening.
+            # (batch, grid_t, grid_h, grid_w, merge_h, merge_w, channel, temp_patch_size, patch_h, patch_w)
+            patches = patches.permute(0, 1, 4, 7, 5, 8, 3, 2, 6, 9)
+            flatten_patches = patches.reshape(
+                batch_size,
+                grid_t * grid_h * grid_w,
+                channel * temporal_patch_size * patch_size * patch_size,
+            )
+
+            processed_images_grouped[shape] = flatten_patches
+            processed_grids[shape] = [[grid_t, grid_h, grid_w]] * batch_size
+
+        processed_images = reorder_images(processed_images_grouped, grouped_images_index)
+        processed_grids = reorder_images(processed_grids, grouped_images_index)
+        pixel_values = torch.cat(processed_images, dim=0)
+        image_grid_thw = torch.tensor(processed_grids)
+
+        return BatchFeature(
+            data={"pixel_values": pixel_values, "image_grid_thw": image_grid_thw}, tensor_type=return_tensors
+        )
+
+def _to_numpy(x):
+    """
+    Convert various tensor types to numpy array.
+    Supports torch.Tensor, tf.Tensor, jax.Array, np.ndarray, lists, and primitives.
+    
+    Args:
+        x: Input value that can be a tensor from various frameworks or a Python primitive
+        
+    Returns:
+        np.ndarray: NumPy array representation of the input
+    """
+    # Already numpy
+    if isinstance(x, np.ndarray):
+        return x
+    
+    # Torch tensor or TensorFlow tensor (both have .numpy() method)
+    if hasattr(x, 'numpy'):
+        # For torch tensors on CUDA, need to move to CPU first
+        if hasattr(x, 'cpu'):
+            return x.cpu().numpy()
+        # For TensorFlow or already on CPU
+        return x.numpy()
+    
+    # JAX arrays and other array-like objects that support __array__ protocol
+    if hasattr(x, '__array__'):
+        return np.asarray(x)
+    
+    # Python primitives (list, tuple, int, float)
+    return np.array(x)
+
+
+def _split_array_or_tensor(x, split_indices):
+    """Split along the first dimension while preserving tensor/array type."""
+    split_indices = [int(idx) for idx in split_indices]
+    if isinstance(x, torch.Tensor):
+        if not split_indices:
+            return [x]
+        chunks = []
+        start = 0
+        for end in split_indices:
+            chunks.append(x[start:end])
+            start = end
+        chunks.append(x[start:])
+        return chunks
+    return np.split(x, split_indices)
+
+
+def _concat_array_or_tensor(items, axis=0):
+    """Concatenate while preserving tensor/array type and device."""
+    if not items:
+        return None
+
+    if any(isinstance(item, torch.Tensor) for item in items):
+        ref = next(item for item in items if isinstance(item, torch.Tensor))
+        tensor_items = [
+            item
+            if isinstance(item, torch.Tensor)
+            else torch.as_tensor(item, device=ref.device, dtype=ref.dtype)
+            for item in items
+        ]
+        return torch.cat(tensor_items, dim=axis)
+
+    return np.concatenate(items, axis=axis)
+
+
+def _stack_array_or_tensor(items, axis=0):
+    """Stack while preserving tensor/array type and device."""
+    if not items:
+        return None
+
+    if any(isinstance(item, torch.Tensor) for item in items):
+        ref = next(item for item in items if isinstance(item, torch.Tensor))
+        tensor_items = [
+            item
+            if isinstance(item, torch.Tensor)
+            else torch.as_tensor(item, device=ref.device, dtype=ref.dtype)
+            for item in items
+        ]
+        return torch.stack(tensor_items, dim=axis)
+
+    return np.stack(items, axis=axis)
+
+
+class MossVLImagesKwargs(ImagesKwargs):
+    min_pixels: Optional[int]
+    max_pixels: Optional[int]
+    patch_size: Optional[int]
+    temporal_patch_size: Optional[int]
+    merge_size: Optional[int]
+
+
+
+class MossVLVideosKwargs(VideosKwargs, total=False):
+    video_fps: Optional[Union[int, float]]
+    min_frames: Optional[int]
+    max_frames: Optional[int]
+    num_extract_threads: Optional[int]
+
+
+class MossVLProcessorKwargs(ProcessingKwargs, total=False):
+    images_kwargs: MossVLImagesKwargs
+    videos_kwargs: MossVLVideosKwargs
+    # _defaults = {
+    #     "text_kwargs": {
+    #         "padding": True,                    # 👈 启用 padding
+    #         "padding_side": "left",            # 👈 左 padding
+    #         "pad_to_multiple_of": 8,           # 👈 pad 到 8 的倍数
+    #         "return_token_type_ids": False,
+    #         "return_mm_token_type_ids": False,
+    #     },
+    #     "videos_kwargs": {"return_metadata": True},
+    # }
+    _defaults = {
+        "text_kwargs": {
+            "padding": False,
+            "return_token_type_ids": False,
+            "return_mm_token_type_ids": False,
+        },
+        "videos_kwargs": {"return_metadata": True},
+    }
+
+class MossVLProcessor(ProcessorMixin):
+    r"""
+    Constructs a Moss-VL processor which wraps a Qwen2VL image processor, Moss-VL video processor and a Qwen2 tokenizer
+    into a single processor.
+
+    [`MossVLProcessor`] offers all the functionalities of [`Qwen2VLImageProcessor`], [`MossVLVideoProcessor`] and [`Qwen2TokenizerFast`].
+    See the [`~MossVLProcessor.__call__`] and [`~MossVLProcessor.decode`] for more information.
+
+    Args:
+        image_processor ([`Qwen2VLImageProcessor`], *optional*):
+            The image processor is a required input.
+        tokenizer ([`Qwen2TokenizerFast`], *optional*):
+            The tokenizer is a required input.
+        video_processor ([`MossVLVideoProcessor`], *optional*):
+            The video processor is a required input.
+        chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages
+            in a chat into a tokenizable string.
+    """
+
+    attributes = ["image_processor", "tokenizer", "video_processor"]
+    image_processor_class = "AutoImageProcessor"
+    video_processor_class = "AutoVideoProcessor"
+    tokenizer_class = ("Qwen2Tokenizer", "Qwen2TokenizerFast")
+
+    def __init__(
+        self, 
+        image_processor=None, 
+        tokenizer=None, 
+        video_processor=None, 
+        chat_template=None,
+        **kwargs
+    ):
+        super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
+        
+
+        self.image_token = "<|image_pad|>" if not hasattr(tokenizer, "image_token") else tokenizer.image_token
+        self.video_token = "<|video_pad|>" if not hasattr(tokenizer, "video_token") else tokenizer.video_token
+
+        
+        self.image_token_id = (
+            tokenizer.image_token_id
+            if getattr(tokenizer, "image_token_id", None)
+            else tokenizer.convert_tokens_to_ids(self.image_token)
+        )
+        self.video_token_id = (
+            tokenizer.video_token_id
+            if getattr(tokenizer, "video_token_id", None)
+            else tokenizer.convert_tokens_to_ids(self.video_token)
+        )
+        
+        self.vision_start_token = (
+            "<|vision_start|>" if not hasattr(tokenizer, "vision_start_token") else tokenizer.vision_start_token
+        )
+        self.vision_end_token = (
+            "<|vision_end|>" if not hasattr(tokenizer, "vision_end_token") else tokenizer.vision_end_token
+        )
+
+        # Placeholders used in input text
+        self.image_placeholder = "<|image|>"
+        self.video_placeholder = "<|video|>"
+
+        self.time_start_token = "<|time_start|>"
+        self.time_end_token = "<|time_end|>"
+        
+        # EOS token for labels generation (assistant's response should end with this)
+        self.im_end_token = "<|im_end|>"
+        self.im_end_token_id = tokenizer.convert_tokens_to_ids(self.im_end_token)
+        
+        # Vision-related token ids (all should be masked in labels)
+        self.vision_start_token_id = tokenizer.convert_tokens_to_ids(self.vision_start_token)
+        self.vision_end_token_id = tokenizer.convert_tokens_to_ids(self.vision_end_token)
+        
+        # Token ids that should always be masked in labels (e.g. <|image_pad|>)
+        self.mask_token_ids = {self.image_token_id}
+
+    def __call__(
+        self,
+        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,
+        images: ImageInput = None,
+        videos: Union[str, Dict[str, Any], List[Union[str, Dict[str, Any]]]] = None,
+        labels_spans: Optional[Union[List[tuple], List[List[tuple]]]] = None,
+        ignore_index: int = -100,
+        **kwargs: Unpack[MossVLProcessorKwargs],
+    ) -> BatchFeature:
+        """
+        Main method to prepare for the model one or several sequences(s) and image(s)/video(s).
+
+        Args:
+            text (`str`, `list[str]`, `list[list[str]]`):
+                The sequence or batch of sequences to be encoded.
+            images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`):
+                The image or batch of images to be prepared.
+            videos (`str`, `Dict`, `list[str]`, `list[Dict]`):
+                The video or batch of videos to be prepared. Each video can be:
+                - A string path to a video file
+                - A dict with keys:
+                    - "video_path": str, path to the video file
+                    - "segments": list of segments, where each segment is:
+                        - [start, end]: a time segment (left-closed, right-open interval in seconds)
+                        - [time]: a single frame at the specified time (in seconds)
+                  The number of segments should match the number of video placeholders in the text.
+            labels_spans (`list[list[int]]`, `list[list[list[int]]]`, *optional*):
+                Character-level spans indicating assistant regions in original text.
+                Each span is a [start, end] list with inclusive start and exclusive end.
+                Example: [[10, 50], [100, 150]] means characters [10:50) and [100:150) are assistant.
+                Note: Use list (not tuple) for spans as they will be modified in place during processing.
+                When provided, the processor will generate `labels` in the output, where:
+                - Non-assistant tokens have value `ignore_index` (-100 by default)
+                - Image tokens always have value `ignore_index` even in assistant part
+                - Other assistant tokens have their token id as label
+            ignore_index (`int`, *optional*, defaults to -100):
+                Value for masked positions in labels.
+            return_tensors (`str` or [`~utils.TensorType`], *optional*):
+                If set, will return tensors of a particular framework. Acceptable values are:
+                - `'tf'`: Return TensorFlow `tf.constant` objects.
+                - `'pt'`: Return PyTorch `torch.Tensor` objects.
+                - `'np'`: Return NumPy `np.ndarray` objects.
+                - `'jax'`: Return JAX `jnp.ndarray` objects.
+
+
+        Returns:
+            [`BatchFeature`]: A [`BatchFeature`] with the following fields:
+            - **input_ids** -- List of token ids to be fed to a model.
+            - **attention_mask** -- List of indices specifying which tokens should be attended to by the model.
+            - **pixel_values** -- Pixel values to be fed to a model (concatenation of images and videos).
+            - **grid_thw** -- List of grid sizes (t, h, w) for each media item.
+            - **media_nums_per_sample** -- List of number of media items per sample.
+            - **labels** -- (Optional) Labels for training, only present when `labels_spans` is provided.
+        """
+        # Merge kwargs with defaults
+        output_kwargs = self._merge_kwargs(
+            MossVLProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+        
+        # Step 1: Process images if provided
+        if images is not None:
+            images_kwargs = output_kwargs["images_kwargs"].copy()
+            images_kwargs["return_tensors"] = None
+            image_inputs = self.image_processor(images=images, **images_kwargs)
+            image_grid_thw = image_inputs["image_grid_thw"]
+        else:
+            image_inputs = {}
+            image_grid_thw = None
+
+        # Step 2: Process videos if provided
+        if videos is not None:
+            videos_kwargs = output_kwargs["videos_kwargs"].copy()
+            videos_kwargs["return_tensors"] = None
+            videos_inputs = self.video_processor(videos=videos, **videos_kwargs)
+            video_grid_thw = videos_inputs["video_grid_thw"]
+            # If user has not requested video metadata, pop it
+            if "return_metadata" not in kwargs:
+                video_metadata = videos_inputs.pop("video_metadata")
+            else:
+                video_metadata = videos_inputs["video_metadata"]
+        else:
+            videos_inputs = {}
+            video_grid_thw = None
+            video_metadata = None
+
+        # Step 3: Process text with placeholder replacement
+        if text is None or (isinstance(text, str) and len(text.strip()) == 0):
+            raise ValueError("Text input is required for MossVL processor and cannot be empty.")
+        
+        if not isinstance(text, list):
+            text = [text]
+        
+        text = text.copy()  # Copy to avoid in-place modifications
+        
+        # Prepare labels_spans if provided
+        # labels_spans format: List[List[List[int]]] - batch of samples, each sample has multiple spans
+        # Each span is [start, end] (list, not tuple) so it can be modified in place
+        should_create_labels = labels_spans is not None
+        if should_create_labels:
+            # Ensure batch format: convert single sample spans to batch format
+            # Single sample: [[start, end], [start, end], ...]
+            # Batch: [[[start, end], ...], [[start, end], ...], ...]
+            if labels_spans and isinstance(labels_spans[0], list) and len(labels_spans[0]) == 2 and isinstance(labels_spans[0][0], int):
+                labels_spans = [labels_spans]
+
+        # Step 3.0-pre: Check if we need to reorder (when both images and videos exist)
+        # If only one media type exists, we can skip the expensive split+reorder+concat
+        has_images = images is not None and "pixel_values" in image_inputs
+        has_videos = videos is not None and "pixel_values_videos" in videos_inputs
+        needs_reorder = has_images and has_videos
+        
+        image_pixel_values_list = []
+        video_pixel_values_list = []
+        
+        # Step 3.0: Record the order of media in original text (before replacement)
+        # This will be used later to correctly order pixel_values and grid_thw
+        media_order_per_sample = []
+        for i in range(len(text)):
+            media_order = []
+            temp_text = text[i]
+            pos = 0
+            while pos < len(temp_text):
+                img_pos = temp_text.find(self.image_placeholder, pos)
+                vid_pos = temp_text.find(self.video_placeholder, pos)
+                
+                if img_pos == -1 and vid_pos == -1:
+                    break
+                
+                if img_pos != -1 and (vid_pos == -1 or img_pos < vid_pos):
+                    media_order.append(("image", img_pos))
+                    pos = img_pos + len(self.image_placeholder)
+                elif vid_pos != -1:
+                    media_order.append(("video", vid_pos))
+                    pos = vid_pos + len(self.video_placeholder)
+            
+            media_order_per_sample.append(media_order)
+        
+        # Step 3.0.1: Check if any sample has no media (empty samples need blank image)
+        # If there are empty samples, we need to enter slow path to handle them properly
+        has_empty_samples = any(len(order) == 0 for order in media_order_per_sample)
+        if has_empty_samples:
+            needs_reorder = True
+        
+        # Split pixel values for reordering if needed
+        if needs_reorder:
+            if has_images:
+                flat_pixel_values = image_inputs["pixel_values"]
+                flat_grid_thw = image_inputs["image_grid_thw"]
+                # grid_thw is (t, h, w), num_patches = t * h * w
+                patch_counts = [int(np.prod(_to_numpy(grid))) for grid in flat_grid_thw]
+                if len(patch_counts) == 1:
+                    # Single image case: no need to split
+                    image_pixel_values_list = [flat_pixel_values]
+                elif len(patch_counts) > 1:
+                    # Multiple images: split by cumulative counts
+                    split_indices = np.cumsum(patch_counts)[:-1]
+                    image_pixel_values_list = _split_array_or_tensor(
+                        flat_pixel_values, split_indices
+                    )
+
+            if has_videos:
+                flat_video_values = videos_inputs["pixel_values_videos"]
+                flat_video_grid = videos_inputs["video_grid_thw"]
+                video_patch_counts = [int(np.prod(_to_numpy(grid))) for grid in flat_video_grid]
+                if len(video_patch_counts) == 1:
+                    # Single video case: no need to split
+                    video_pixel_values_list = [flat_video_values]
+                elif len(video_patch_counts) > 1:
+                    # Multiple videos: split by cumulative counts
+                    split_indices = np.cumsum(video_patch_counts)[:-1]
+                    video_pixel_values_list = _split_array_or_tensor(
+                        flat_video_values, split_indices
+                    )
+        
+        # Step 3.1: Replace placeholders (simple replacement, no expansion yet)
+        # In MossVL, one image placeholder = one image token
+        # One video placeholder = one video token (will be expanded later)
+        for i in range(len(text)):
+            if should_create_labels:
+                # Replace and update spans for image placeholders
+                text[i], labels_spans[i] = self._replace_and_update_spans(
+                    text[i], self.image_placeholder, self.image_token, labels_spans[i]
+                )
+                # Replace and update spans for video placeholders
+                text[i], labels_spans[i] = self._replace_and_update_spans(
+                    text[i], self.video_placeholder, self.video_token, labels_spans[i]
+                )
+            else:
+                text[i] = text[i].replace(self.image_placeholder, self.image_token)
+                text[i] = text[i].replace(self.video_placeholder, self.video_token)
+        
+        # Step 3.2: Validate token counts 
+        n_images_in_text = [t.count(self.image_token) for t in text]
+        n_videos_in_text = [t.count(self.video_token) for t in text]
+        
+        # Count placeholders in text
+        total_images_in_text = sum(n_images_in_text)
+        total_videos_in_text = sum(n_videos_in_text)
+        
+        # Count actual images and videos provided
+        total_images_provided = len(image_grid_thw) if image_grid_thw is not None else 0
+        total_videos_provided = len(video_grid_thw) if video_grid_thw is not None else 0
+        
+        # Validate image counts
+        if total_images_in_text != total_images_provided:
+            raise ValueError(
+                "Number of image tokens does not match number of images provided. "
+                f"Found {total_images_in_text} image tokens in text and {total_images_provided} images."
+            )
+        
+        # Validate video counts
+        if total_videos_in_text != total_videos_provided:
+            raise ValueError(
+                "Number of video tokens does not match number of videos provided. "
+                f"Found {total_videos_in_text} video tokens in text and {total_videos_provided} videos."
+            )
+        
+        # Step 3.3: Expand video tokens with timestamps
+        # Now expand each video token to multiple tokens (one per frame) with timestamps
+        if video_grid_thw is not None:
+            index = 0
+            for i in range(len(text)):
+                while self.video_token in text[i]:
+                    metadata = video_metadata[index]
+                    if metadata.fps is None:
+                        logger.warning_once(
+                            "MossVL requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. "
+                            "Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. "
+                            "Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results."
+                        )
+                        metadata.fps = 24 if metadata.fps is None else metadata.fps
+
+                    # Calculate timestamps
+                    # Use actual_timestamps if available (for segments), otherwise use frames_indices
+                    actual_timestamps = getattr(metadata, 'actual_timestamps', None)
+                    curr_timestamp = self._calculate_timestamps(
+                        metadata.frames_indices,
+                        metadata.total_num_frames,
+                        metadata.fps,
+                        metadata.duration,
+                        self.video_processor.temporal_patch_size,
+                        actual_timestamps=actual_timestamps,
+                    )
+
+                    # Build video placeholder: one video token per frame with timestamp
+                    # video_grid_thw[index][0] is the temporal dimension (number of frames after merging)
+
+                    video_tokens = []
+                    for frame_idx in range(video_grid_thw[index][0]):
+                        curr_time = curr_timestamp[frame_idx]
+                        # Format: <|time_start|>X.X seconds<|time_end|><|image_pad|>
+                        video_tokens.append(
+                            f"{self.time_start_token}{curr_time:.1f} seconds{self.time_end_token}{self.image_token}"
+                        )
+                    
+                    # Wrap the entire video sequence with vision_start and vision_end tokens
+                    video_placeholder = f"{self.vision_start_token}{''.join(video_tokens)}{self.vision_end_token}"
+                    
+                    # Replace the video token with expanded sequence and update spans if needed
+                    if should_create_labels:
+                        text[i], labels_spans[i] = self._replace_and_update_spans(
+                            text[i], self.video_token, video_placeholder, labels_spans[i], replace_count=1
+                        )
+                    else:
+                        text[i] = text[i].replace(self.video_token, video_placeholder, 1)
+                    index += 1
+                
+
+
+        # Step 4: Tokenize text
+        return_tensors = output_kwargs["text_kwargs"].pop("return_tensors", None)
+        return_mm_token_type_ids = output_kwargs["text_kwargs"].pop("return_mm_token_type_ids", None)
+        
+        # Request offset_mapping if we need to create labels
+        if should_create_labels:
+            output_kwargs["text_kwargs"]["return_offsets_mapping"] = True
+        
+        text_inputs = self.tokenizer(text, **output_kwargs["text_kwargs"])
+        
+        # ignore check_special_mm_tokens nums in test and input ids.
+        # self._check_special_mm_tokens(text, text_inputs, modalities=["image", "video"])
+        
+        # Create labels if labels_spans was provided
+        if should_create_labels:
+            offset_mapping = text_inputs.pop("offset_mapping")
+            labels = self._create_labels_from_spans(
+                text_inputs["input_ids"],
+                offset_mapping,
+                labels_spans,
+                ignore_index
+            )
+
+        if return_mm_token_type_ids:
+            array_ids = np.array(text_inputs["input_ids"])
+            mm_token_type_ids = np.zeros_like(text_inputs["input_ids"])
+            mm_token_type_ids[array_ids == self.image_token_id] = 1
+            text_inputs["mm_token_type_ids"] = mm_token_type_ids.tolist()
+
+        # Step 5: Concatenate pixel_values and grid_thw in sequence order
+        # Prepare output
+        output_data = {**text_inputs}
+        
+        if not needs_reorder:
+            # Fast path: only one media type, no reordering needed
+            final_pixel_values = []
+            final_grid_thw = []
+            
+            if has_images:
+                final_pixel_values.append(image_inputs["pixel_values"])
+                final_grid_thw.extend(image_grid_thw)
+            
+            if has_videos:
+                final_pixel_values.append(videos_inputs["pixel_values_videos"])
+                final_grid_thw.extend(video_grid_thw)
+            
+            if final_pixel_values:
+                output_data["pixel_values"] = np.concatenate(final_pixel_values, axis=0) if len(final_pixel_values) > 1 else final_pixel_values[0]
+            
+            if final_grid_thw:
+                output_data["grid_thw"] = np.stack(final_grid_thw, axis=0)
+            
+            # Calculate media_nums_per_sample
+            media_nums_per_sample = []
+            for batch_idx in range(len(text)):
+                media_order = media_order_per_sample[batch_idx]
+                media_nums_per_sample.append(len(media_order) if len(media_order) > 0 else 1)
+            
+            # Don't add media_nums_per_sample to output_data yet
+            # Will add it after BatchFeature to keep it as list
+            
+        else:
+            # Slow path: both images and videos exist, need reordering
+            final_pixel_values = []
+            final_grid_thw = []
+            media_nums_per_sample = []
+            
+            # Global indices to track position in flattened image/video arrays
+            global_image_idx = 0
+            global_video_idx = 0
+            
+            for batch_idx in range(len(text)):
+                # Use the recorded media order from Step 3.0
+                media_order = media_order_per_sample[batch_idx]
+                
+                if len(media_order) == 0:
+                    # If no media provided for this sample, add a blank image
+                    media_nums_per_sample.append(1)
+                    min_pixels = 128 * 128
+                    patch_size = getattr(self.image_processor, "patch_size", None) or 16
+                    temporal_patch_size = getattr(self.image_processor, "temporal_patch_size", None) or 1
+                    merge_size = getattr(self.image_processor, "merge_size", None) or 2
+                    
+                    factor = patch_size * merge_size
+                    side = int(np.ceil(np.sqrt(min_pixels) / factor) * factor)
+                    grid_h = side // patch_size
+                    grid_w = side // patch_size
+                    grid_t = 1
+                    
+                    # Channel = 3 (RGB)
+                    channel = 3
+                    dim = channel * temporal_patch_size * patch_size * patch_size
+                    num_patches = grid_t * grid_h * grid_w
+                    
+                    blank_pixel_values = np.zeros((num_patches, dim), dtype=np.float32)
+                    blank_grid_thw = np.array([grid_t, grid_h, grid_w], dtype=np.int64)
+                    
+                    final_pixel_values.append(blank_pixel_values)
+                    final_grid_thw.append(blank_grid_thw)
+                else:
+                    media_nums_per_sample.append(len(media_order))
+                    
+                    # Collect media data according to the recorded order
+                    for media_type, _ in media_order:
+                        if media_type == "image" and image_grid_thw is not None:
+                            # Get image data
+                            if image_pixel_values_list:
+                                final_pixel_values.append(image_pixel_values_list[global_image_idx])
+                            final_grid_thw.append(image_grid_thw[global_image_idx])
+                            global_image_idx += 1
+                        elif media_type == "video" and video_grid_thw is not None:
+                            # Get video data
+                            if video_pixel_values_list:
+                                final_pixel_values.append(video_pixel_values_list[global_video_idx])
+                            final_grid_thw.append(video_grid_thw[global_video_idx])
+                            global_video_idx += 1
+            
+            # Concatenate/stack to unified format
+            if final_pixel_values:
+                output_data["pixel_values"] = _concat_array_or_tensor(
+                    final_pixel_values, axis=0
+                )
+            
+            if final_grid_thw:
+                output_data["grid_thw"] = _stack_array_or_tensor(
+                    final_grid_thw, axis=0
+                )
+            
+            # Don't add media_nums_per_sample to output_data yet
+            # Will add it after BatchFeature to keep it as list
+
+        # Create cross_attention_mask using media_nums_per_sample
+        if "input_ids" in output_data and "grid_thw" in output_data and media_nums_per_sample:
+            cross_attention_mask = self._create_cross_attention_mask(
+                output_data["input_ids"],
+                output_data["grid_thw"],
+                media_nums_per_sample,
+                output_data.get("attention_mask", None)
+            )
+            output_data["cross_attention_mask"] = cross_attention_mask
+        
+        # Add labels to output if created
+        if should_create_labels:
+            output_data["labels"] = labels
+
+        # BatchFeature will handle conversion to pt/tf/jax/np based on tensor_type
+        batch_feature = BatchFeature(data=output_data, tensor_type=return_tensors)
+        
+        # Add media_nums_per_sample after BatchFeature to keep it as list (not tensor)
+        if media_nums_per_sample:
+            batch_feature["media_nums_per_sample"] = media_nums_per_sample
+        
+        return batch_feature
+
+    def _create_cross_attention_mask(self, input_ids, grid_thw, media_nums_per_sample, attention_mask=None):
+        """
+        Create cross_attention_mask of shape (batch_size, 1, text_len, num_images).
+        Video frames are treated as individual images.
+        Mask values: True for masked, False for visible.
+        Causal masking: text can see images that appear at or before the text position.
+        
+        Args:
+            input_ids: List of token ids
+            grid_thw: Grid sizes for each media item
+            media_nums_per_sample: Number of media items per sample
+            attention_mask: Optional attention mask to filter out padding positions
+        """
+        batch_size = len(input_ids)
+        max_text_len = max(len(ids) for ids in input_ids)
+        
+        # Calculate total frames per sample to find max_num_frames
+        total_frames_per_sample = []
+        media_idx = 0
+        for b in range(batch_size):
+            num_media = media_nums_per_sample[b]
+            if num_media == 0:
+                total_frames_per_sample.append(0)
+                continue
+                
+            sample_frames = 0
+            for _ in range(num_media):
+                # grid_thw is (N, 3) where first dim is t (num_frames)
+                t = grid_thw[media_idx][0]
+                if isinstance(t, torch.Tensor):
+                    t = int(t.item())
+                else:
+                    t = int(t)
+                sample_frames += t
+                media_idx += 1
+            total_frames_per_sample.append(sample_frames)
+            
+        max_num_frames = max(total_frames_per_sample) if total_frames_per_sample else 0
+        
+        if max_num_frames == 0:
+            return None
+            
+        # Vectorized implementation for speed
+        
+        # 1. Pad input_ids to create a tensor
+        # We use -1 as pad value since token ids are positive
+        input_ids_tensor = torch.full((batch_size, max_text_len), -1, dtype=torch.long)
+        for b, ids in enumerate(input_ids):
+            l = len(ids)
+            input_ids_tensor[b, :l] = torch.tensor(ids, dtype=torch.long)
+            
+        # 2. Identify image tokens
+        is_image_token = (input_ids_tensor == self.image_token_id)
+        
+        # 3. Compute cumulative image tokens (how many image tokens appeared up to position t)
+        # shape: (batch_size, text_len)
+        cum_image_tokens = is_image_token.cumsum(dim=1)
+        
+        # 4. Create frame indices
+        # shape: (1, 1, max_num_frames)
+        frame_indices = torch.arange(max_num_frames).reshape(1, 1, -1)
+        
+        # 5. Determine visibility based on causal relationship
+        # Text at `t` sees frame `i` if `cum_image_tokens[t] > i`
+        # Because if frame `i` is the (i+1)-th image token, it becomes visible when count reaches i+1
+        # shape: (batch_size, text_len, max_num_frames)
+        visible_mask = cum_image_tokens.unsqueeze(-1) > frame_indices
+        
+        # 6. Apply attention_mask if provided
+        if attention_mask is not None:
+            # Convert to tensor if needed
+            if isinstance(attention_mask, torch.Tensor):
+                 attn_mask_tensor = attention_mask
+            else:
+                 # List of lists
+                 attn_mask_tensor = torch.zeros((batch_size, max_text_len), dtype=torch.long)
+                 for b, mask_row in enumerate(attention_mask):
+                     l = len(mask_row)
+                     attn_mask_tensor[b, :l] = torch.tensor(mask_row, dtype=torch.long)
+            
+            # shape: (batch_size, text_len, 1)
+            valid_text = (attn_mask_tensor.unsqueeze(-1) == 1)
+            visible_mask = visible_mask & valid_text
+            
+        # 7. Mask out frames that don't exist for a sample
+        # shape: (batch_size, 1, 1)
+        total_frames_tensor = torch.tensor(total_frames_per_sample).reshape(batch_size, 1, 1)
+        # shape: (batch_size, 1, max_num_frames)
+        valid_frames = frame_indices < total_frames_tensor
+        
+        visible_mask = visible_mask & valid_frames
+        
+        # 8. Create final mask (True for masked, False for visible)
+        mask = ~visible_mask
+        
+        # 9. Add channel dimension: (batch_size, 1, text_len, max_num_frames)
+        mask = mask.unsqueeze(1)
+        
+        return mask
+
+    def _replace_and_update_spans(
+        self,
+        text: str,
+        old_str: str,
+        new_str: str,
+        spans: List[List[int]],
+        replace_count: int = -1
+    ) -> tuple:
+        """
+        Replace occurrences of old_str with new_str and update spans accordingly.
+        
+        Args:
+            text: The text to perform replacement on
+            old_str: String to be replaced
+            new_str: String to replace with
+            spans: List of [start, end] spans to update (modified in place)
+            replace_count: Maximum number of replacements (-1 for all)
+            
+        Returns:
+            Tuple of (new_text, updated_spans)
+        """
+        delta = len(new_str) - len(old_str)
+        result_text = text
+        count = 0
+        search_start = 0
+        
+        while True:
+            pos = result_text.find(old_str, search_start)
+            if pos == -1:
+                break
+            if replace_count != -1 and count >= replace_count:
+                break
+            
+            # Update all spans that come after this position
+            for span in spans:
+                if span[0] > pos:
+                    # Span starts after replacement point
+                    span[0] += delta
+                    span[1] += delta
+                elif span[1] > pos:
+                    # Span ends after replacement point (spans the replacement)
+                    span[1] += delta
+            
+            # Perform the replacement
+            result_text = result_text[:pos] + new_str + result_text[pos + len(old_str):]
+            search_start = pos + len(new_str)
+            count += 1
+        
+        return result_text, spans
+
+    def _create_labels_from_spans(
+        self,
+        input_ids: List[List[int]],
+        offset_mapping: List[List[tuple]],
+        labels_spans: List[List[List[int]]],
+        ignore_index: int = -100,
+        mask_token_ids: Optional[set] = None
+    ) -> List[List[int]]:
+        """
+        Create labels from spans and offset_mapping.
+        
+        Args:
+            input_ids: Tokenized input ids
+            offset_mapping: Character offsets for each token from tokenizer (special tokens included)
+            labels_spans: Updated spans indicating assistant regions (after text transformations)
+            ignore_index: Value for masked positions
+            mask_token_ids: Set of token ids that should always be masked (set to ignore_index)
+                in labels, regardless of whether they fall inside a span.
+                Defaults to self.mask_token_ids if not provided.
+            
+        Returns:
+            labels: List of label ids, same shape as input_ids
+        
+        Note:
+            - Tokenizer's offset_mapping already includes correct offsets for special tokens in text
+            - Only need to mask tokens inside <|vision_start|>...<|vision_end|>
+            - Tokens whose id is in mask_token_ids are always masked
+            - All other tokens in spans (including special tokens like <|im_end|>) get labels
+        """
+        if mask_token_ids is None:
+            mask_token_ids = self.mask_token_ids
+        
+        batch_labels = []
+        
+        for batch_idx in range(len(input_ids)):
+            ids = input_ids[batch_idx]
+            offsets = offset_mapping[batch_idx]
+            spans = labels_spans[batch_idx]
+            
+            labels = [ignore_index] * len(ids)
+            
+            # Process each span: find token range and set labels
+            for span_start, span_end in spans:
+                in_vision = False
+                
+                # Find tokens that overlap with this span
+                for token_idx, (token_id, (char_start, char_end)) in enumerate(zip(ids, offsets)):
+                    # Skip tokens completely before this span
+                    if char_end <= span_start:
+                        continue
+                    # Stop when tokens are completely after this span
+                    if char_start >= span_end:
+                        break
+                    
+                    # Token overlaps with span, process it
+                    # Track vision region: <|vision_start|> ... <|vision_end|>
+                    if token_id == self.vision_start_token_id:
+                        in_vision = True
+                        continue
+                    if token_id == self.vision_end_token_id:
+                        in_vision = False
+                        continue
+                    
+                    # Skip tokens inside vision region
+                    if in_vision:
+                        continue
+                    
+                    # Always mask special tokens that should never have labels
+                    if token_id in mask_token_ids:
+                        continue
+                    
+                    # Set label for this token
+                    labels[token_idx] = token_id
+            
+            batch_labels.append(labels)
+        
+        return batch_labels
+
+    def _calculate_timestamps(
+        self, 
+        frames_indices: Optional[Union[List[int], np.ndarray]], 
+        total_num_frames: int, 
+        video_fps: float, 
+        duration: float, 
+        merge_size: int = 1,
+        actual_timestamps: Optional[List[float]] = None
+    ):
+        """
+        Calculate timestamps for video frames.
+        
+        Args:
+            frames_indices: Actual frame indices extracted (if available)
+            total_num_frames: Total number of sampled frames
+            video_fps: Video frames per second
+            duration: Video duration in seconds
+            merge_size: Temporal merge size
+            actual_timestamps: Pre-calculated actual timestamps (for segments)
+            
+        Returns:
+            List of timestamps (one per merged temporal patch)
+        """
+        # If actual timestamps are provided (from segment), use them directly
+        if actual_timestamps is not None:
+            timestamps = list(actual_timestamps)
+            
+            # Pad timestamps to be multiple of merge_size
+            if len(timestamps) % merge_size != 0:
+                timestamps.extend([timestamps[-1]] * (merge_size - len(timestamps) % merge_size))
+            
+            # Frames are merged by merge_size, so we average the timestamps within each temporal patch
+            timestamps = [
+                (timestamps[i] + timestamps[i + merge_size - 1]) / 2 
+                for i in range(0, len(timestamps), merge_size)
+            ]
+            return timestamps
+        
+        # Use frames_indices if available, otherwise generate uniformly sampled indices
+        if frames_indices is not None:
+            if isinstance(frames_indices, np.ndarray):
+                indices = frames_indices.tolist()
+            else:
+                indices = list(frames_indices)
+        else:
+            # Generate uniformly sampled frame indices
+            if total_num_frames <= 1:
+                indices = [0]
+            else:
+                # Uniformly sample frames across the video duration
+                indices = np.linspace(0, duration * video_fps - 1, total_num_frames).astype(np.int32).tolist()
+        
+        # Pad indices to be multiple of merge_size
+        if len(indices) % merge_size != 0:
+            indices.extend([indices[-1]] * (merge_size - len(indices) % merge_size))
+        
+        # Convert frame indices to timestamps
+        timestamps = [idx / video_fps for idx in indices]
+        
+        # Frames are merged by merge_size, so we average the timestamps within each temporal patch
+        timestamps = [
+            (timestamps[i] + timestamps[i + merge_size - 1]) / 2 
+            for i in range(0, len(timestamps), merge_size)
+        ]
+        return timestamps
+
+    def batch_decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to the tokenizer's batch_decode. 
+        Please refer to the docstring of this method for more information.
+        """
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to the tokenizer's decode.
+        Please refer to the docstring of this method for more information.
+        """
+        return self.tokenizer.decode(*args, **kwargs)
+
+    def post_process_image_text_to_text(
+        self, generated_outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False, **kwargs
+    ):
+        """
+        Post-process the output of the model to decode the text.
+
+        Args:
+            generated_outputs (`torch.Tensor` or `np.ndarray`):
+                The output of the model `generate` function. The output is expected to be a tensor 
+                of shape `(batch_size, sequence_length)` or `(sequence_length,)`.
+            skip_special_tokens (`bool`, *optional*, defaults to `True`):
+                Whether or not to remove special tokens in the output.
+            clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
+                Whether or not to clean up the tokenization spaces.
+            **kwargs:
+                Additional arguments to be passed to the tokenizer's `batch_decode` method.
+
+        Returns:
+            `list[str]`: The decoded text.
+        """
+        return self.tokenizer.batch_decode(
+            generated_outputs,
+            skip_special_tokens=skip_special_tokens,
+            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+            **kwargs,
+        )
+
+
+__all__ = ["MossVLProcessor", "MossVLImageProcessorFast"]

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7bbd0f9784004df51aca562befc3c7a8f294b4045aa8685536c35804c9aa493
+size 11423411

tokenizer_config.json

@@ -0,0 +1,276 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<|time_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151670": {
+      "content": "<|time_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151671": {
+      "content": "<|silence|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151672": {
+      "content": "<|response|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<|time_start|>",
+    "<|time_end|>",
+    "<|silence|>",
+    "<|response|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 262144,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' }}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- for message in messages %}\n    {%- if message.role == \"user\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|image|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|video|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content_item in message.content %}\n                {%- if 'text' in content_item %}\n                    {{- content_item.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and message.content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|image|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|video|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n"
+}

video_preprocessor_config.json

@@ -0,0 +1,30 @@
+{
+    "auto_map": {
+        "AutoProcessor": "processing_moss_vl.MossVLProcessor",
+        "AutoVideoProcessor": "video_processing_moss_vl.MossVLVideoProcessor"
+    },
+    "size": {
+        "longest_edge": 16777216,
+        "shortest_edge": 4096
+    },
+    "video_max_pixels": 65536000,
+    "patch_size": 16,
+    "temporal_patch_size": 1,
+    "merge_size": 2,
+    "video_fps": 1.0,
+    "min_frames": 1,
+    "max_frames": 256,
+    "num_extract_threads": 4,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "MossVLProcessor",
+    "video_processor_type": "MossVLVideoProcessor"
+}

video_processing_moss_vl.py

@@ -0,0 +1,1248 @@
+# coding=utf-8
+# Copyright 2025 The FNLP Vision Team and The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""video processor class for Moss-VL."""
+
+import json
+import logging as system_logging
+import math
+import os
+import re
+import subprocess
+import traceback
+from functools import lru_cache
+from typing import Any, Dict, List, Optional, Union
+
+import numpy as np
+import torch
+from joblib import Parallel, delayed
+from torchcodec.decoders import VideoDecoder
+
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.image_utils import ChannelDimension, PILImageResampling, SizeDict, get_image_size, validate_kwargs
+from transformers.processing_utils import Unpack, VideosKwargs
+from transformers.utils import TensorType, add_start_docstrings, logging
+from transformers.video_processing_utils import BASE_VIDEO_PROCESSOR_DOCSTRING, BaseVideoProcessor
+from transformers.video_utils import VideoMetadata, group_videos_by_shape, reorder_videos
+
+
+logger = logging.get_logger(__name__)
+
+TORCHCODEC_TIMESTAMP_EPSILON = 1e-6
+
+
+# -----------------------------------------------------------------------------
+# Torchcodec video frame extraction utilities
+# -----------------------------------------------------------------------------
+
+def check_video_for_extra_streams_and_errors(video_path: str) -> dict:
+    """
+    Check if video file has abnormal streams or errors reported by ffprobe.
+    
+    Args:
+        video_path: Path to the video file.
+        
+    Returns:
+        A dictionary containing:
+        - 'has_extra_streams': bool, whether there are streams other than video and audio.
+        - 'unsupported_codec_errors': list, all "Unsupported codec" error messages.
+        - 'ffprobe_output_error': str, other errors/warnings from ffprobe stderr.
+        - 'ffprobe_successful': bool, whether ffprobe command executed successfully (return code 0).
+        - 'stream_details': list, codec_type and index for each stream.
+        - 'num_streams': int, total number of streams identified in the video file.
+    """
+    result = {
+        'has_extra_streams': False,
+        'unsupported_codec_errors': [],
+        'ffprobe_output_error': '',
+        'ffprobe_successful': False,
+        'stream_details': [],
+        'num_streams': 0
+    }
+    
+    command = [
+        "ffprobe",
+        "-v", "error",
+        "-show_streams",
+        "-show_format",
+        "-of", "json",
+        video_path
+    ]
+    
+    try:
+        process = subprocess.run(
+            command,
+            capture_output=True,
+            text=True,
+            check=False
+        )
+        result['ffprobe_successful'] = (process.returncode == 0)
+        
+        if process.stderr:
+            result['ffprobe_output_error'] = process.stderr
+            unsupported_codec_pattern = re.compile(r"Unsupported codec with id \d+ for input stream \d+")
+            result['unsupported_codec_errors'] = unsupported_codec_pattern.findall(process.stderr)
+        
+        if process.stdout:
+            ffprobe_data = json.loads(process.stdout)
+            if 'streams' in ffprobe_data:
+                result['num_streams'] = len(ffprobe_data['streams'])
+                for stream in ffprobe_data['streams']:
+                    stream_type = stream.get('codec_type')
+                    stream_index = stream.get('index')
+                    result['stream_details'].append({'index': stream_index, 'codec_type': stream_type})
+                    if stream_type not in ['video', 'audio']:
+                        result['has_extra_streams'] = True
+                        
+            if 'format' in ffprobe_data and 'nb_streams' in ffprobe_data['format']:
+                if result['num_streams'] == 0:
+                    result['num_streams'] = ffprobe_data['format']['nb_streams']
+                elif result['num_streams'] != ffprobe_data['format']['nb_streams']:
+                    logger.warning(
+                        f"Number of streams in 'streams' list ({result['num_streams']}) "
+                        f"differs from 'nb_streams' in 'format' ({ffprobe_data['format']['nb_streams']})."
+                    )
+    except FileNotFoundError:
+        result['ffprobe_output_error'] = "ffprobe command not found. Please ensure FFmpeg is installed and in your PATH."
+        result['ffprobe_successful'] = False
+    except json.JSONDecodeError:
+        result['ffprobe_output_error'] = "Failed to parse ffprobe JSON output. Check ffprobe installation or video file."
+        result['ffprobe_successful'] = False
+    except Exception as e:
+        result['ffprobe_output_error'] = f"An unexpected error occurred: {e}"
+        result['ffprobe_successful'] = False
+        
+    return result
+
+
+def remove_video_extra_stream_ffmpeg(input_video: str, output_video: str) -> bool:
+    """
+    Remove extra streams from video using ffmpeg.
+    
+    Args:
+        input_video: Path to input video.
+        output_video: Path to output video.
+        
+    Returns:
+        bool: True if successful, False otherwise.
+    """
+    command_list = [
+        "ffmpeg", "-y", "-i", input_video,
+        "-map", "0:v:0",
+        "-c", "copy",
+        "-an",
+        "-sn",
+        "-dn",
+        "-map_metadata", "-1",
+        "-map_chapters", "-1",
+        "-movflags", "faststart",
+        output_video,
+    ]
+    
+    try:
+        subprocess.run(command_list, shell=False, check=True, capture_output=True)
+        return True
+    except subprocess.CalledProcessError as e:
+        system_logging.error(f"Command execution failed with return code: {e.returncode}, video: {input_video}")
+        system_logging.error(f"Error output:\n{e.stderr}")
+        return False
+    except FileNotFoundError:
+        system_logging.error("Error: ffmpeg command not found. Please ensure ffmpeg is installed and in PATH.")
+        return False
+    except Exception as e:
+        system_logging.error(f"Unexpected error executing command: {e}, video: {input_video}", exc_info=True)
+        return False
+
+
+def clean_video_streams(video_path: str) -> str:
+    """
+    Clean video streams if extra streams are detected.
+    
+    Args:
+        video_path: Path to the video file.
+        
+    Returns:
+        str: Path to cleaned video (or original if no cleaning needed).
+    """
+    ffprobe_res = check_video_for_extra_streams_and_errors(video_path)
+    if ffprobe_res['has_extra_streams']:
+        base_name = os.path.basename(video_path)
+        output_folder = os.path.dirname(video_path)
+        file_name_without_ext, file_ext = os.path.splitext(base_name)
+        new_base_name = f"{file_name_without_ext}_fix{file_ext}"
+        video_path_output = os.path.join(output_folder, new_base_name)
+        
+        process_flag = remove_video_extra_stream_ffmpeg(video_path, video_path_output)
+        if not process_flag:
+            logger.warning("Failed to remove extra streams with ffmpeg")
+            return video_path
+        return video_path_output
+    return video_path
+
+
+@lru_cache(maxsize=8192)
+def cached_clean_video_streams(video_path: str) -> str:
+    return clean_video_streams(video_path)
+
+
+def clamp_timestamps_for_torchcodec(timestamps: List[float], torchcodec_metadata) -> List[float]:
+    if not timestamps:
+        return timestamps
+
+    min_pts = torchcodec_metadata.begin_stream_seconds_from_content
+    if min_pts is None:
+        min_pts = 0.0
+
+    max_pts_candidates = []
+    if torchcodec_metadata.num_frames_from_content and torchcodec_metadata.average_fps:
+        max_pts_candidates.append(
+            (torchcodec_metadata.num_frames_from_content - 1) / torchcodec_metadata.average_fps + min_pts
+        )
+    if torchcodec_metadata.end_stream_seconds_from_content is not None:
+        # TorchCodec requires requested PTS to be strictly smaller than the content end.
+        max_pts_candidates.append(torchcodec_metadata.end_stream_seconds_from_content - TORCHCODEC_TIMESTAMP_EPSILON)
+    if not max_pts_candidates and torchcodec_metadata.duration_seconds is not None:
+        max_pts_candidates.append(torchcodec_metadata.duration_seconds - TORCHCODEC_TIMESTAMP_EPSILON)
+
+    if max_pts_candidates:
+        max_pts = max(min_pts, min(max_pts_candidates))
+        return [max(min_pts, min(float(t), max_pts)) for t in timestamps]
+    if min_pts > 0:
+        return [max(min_pts, float(t)) for t in timestamps]
+    return [float(t) for t in timestamps]
+
+
+def split_indices(indices: List[Union[int, float]], num_chunks: int) -> List[List[Union[int, float]]]:
+    """
+    Split an index list into roughly equal chunks.
+    
+    Args:
+        indices: List of indices to split.
+        num_chunks: Number of chunks to create.
+        
+    Returns:
+        List of index chunks.
+    """
+    chunk_size = len(indices) // num_chunks
+    chunks = []
+    for i in range(num_chunks - 1):
+        chunks.append(indices[i * chunk_size:(i + 1) * chunk_size])
+    chunks.append(indices[(num_chunks - 1) * chunk_size:])
+    return chunks
+
+
+def decode_sequentially(indices: List[int], video_path: str, ffmpeg_threads: int = 0):
+    """
+    Decode frames sequentially from a video.
+    
+    Args:
+        indices: List of frame indices to decode.
+        video_path: Path to the video file.
+        ffmpeg_threads: Number of ffmpeg threads to use.
+        
+    Returns:
+        FrameBatch from torchcodec.
+    """
+    decoder = VideoDecoder(video_path, num_ffmpeg_threads=ffmpeg_threads)
+    try:
+        return decoder.get_frames_at(indices)
+    finally:
+        del decoder
+
+
+def decode_with_multithreading(indices: List[int], num_threads: int, video_path: str) -> dict:
+    """
+    Decode frames using multithreading with joblib.
+    
+    Args:
+        indices: List of frame indices to decode.
+        num_threads: Number of threads to use.
+        video_path: Path to the video file.
+        
+    Returns:
+        dict: Contains 'data', 'duration_seconds', 'pts_seconds' tensors.
+    """
+    chunks = split_indices(indices, num_chunks=num_threads)
+    results = Parallel(n_jobs=num_threads, prefer="threads", verbose=0)(
+        delayed(decode_sequentially)(chunk, video_path) for chunk in chunks
+    )
+    
+    return {
+        "data": torch.cat([frame_batch.data for frame_batch in results], dim=0),
+        "duration_seconds": torch.cat([frame_batch.duration_seconds for frame_batch in results], dim=0),
+        "pts_seconds": torch.cat([frame_batch.pts_seconds for frame_batch in results], dim=0)
+    }
+
+
+def decode_sequentially_timestamp(timestamp_list: List[float], video_path: str, ffmpeg_threads: int = 0):
+    """
+    Decode frames sequentially from a video based on timestamps.
+    
+    Args:
+        timestamp_list: List of timestamps (in seconds) to decode.
+        video_path: Path to the video file.
+        ffmpeg_threads: Number of ffmpeg threads to use.
+        
+    Returns:
+        FrameBatch from torchcodec.
+    """
+    decoder = VideoDecoder(video_path, num_ffmpeg_threads=ffmpeg_threads)
+    try:
+        metadata = decoder.metadata
+        timestamp_list = clamp_timestamps_for_torchcodec(timestamp_list, metadata)
+
+        return decoder.get_frames_played_at(timestamp_list)
+    finally:
+        del decoder
+
+
+def timestamp_decode_with_multithreading(timestamp_list: List[float], num_threads: int, video_path: str) -> dict:
+    """
+    Decode frames using multithreading based on timestamps.
+    
+    Args:
+        timestamp_list: List of timestamps (in seconds) to decode.
+        num_threads: Number of threads to use.
+        video_path: Path to the video file.
+        
+    Returns:
+        dict: Contains 'data', 'duration_seconds', 'pts_seconds' tensors.
+    """
+    chunks = split_indices(timestamp_list, num_chunks=num_threads)
+    results = Parallel(n_jobs=num_threads, prefer="threads", verbose=0)(
+        delayed(decode_sequentially_timestamp)(chunk, video_path) for chunk in chunks
+    )
+    
+    # Concatenate results from all threads
+    data_list = [frame_batch.data for frame_batch in results]
+    duration_list = [frame_batch.duration_seconds for frame_batch in results]
+    pts_list = [frame_batch.pts_seconds for frame_batch in results]
+    
+    if not data_list:
+        logger.warning("No frames were successfully decoded.")
+        return {"data": torch.empty(0), "duration_seconds": torch.empty(0), "pts_seconds": torch.empty(0)}
+    
+    return {
+        "data": torch.cat(data_list, dim=0),
+        "duration_seconds": torch.cat(duration_list, dim=0),
+        "pts_seconds": torch.cat(pts_list, dim=0)
+    }
+
+
+def extract_frames_with_torchcodec(
+    video_path: str,
+    sample_frames_count: int,
+    num_threads: int = 4,
+
+) -> Optional[dict]:
+    """
+    Extract frames from video using torchcodec with multithreading.
+    
+    Args:
+        video_path: Path to the video file.
+        sample_frames_count: Number of frames to sample.
+        num_threads: Number of threads to use for extraction.
+        sampling_method: Sampling method, either "index" (uniform frame indices) or "timestamp" (uniform timestamps).
+        
+    Returns:
+        dict: Contains 'data' (N, C, H, W), 'duration_seconds' (N,), 'pts_seconds' (N,) tensors.
+              Returns None if extraction fails.
+    """
+    try:
+        video_path = cached_clean_video_streams(video_path)
+        decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+        metadata = decoder.metadata
+
+
+        total_frames_in_video = metadata.num_frames_from_content
+        
+        effective_sample_count = min(sample_frames_count, total_frames_in_video)
+        if effective_sample_count == 0:
+            logger.error("Cannot extract frames: video has 0 frames or specified frame count is 0")
+            return None
+        
+        # Generate uniform frame indices
+        frame_indices = np.linspace(0, total_frames_in_video - 1, effective_sample_count).astype(np.int32)
+        # Ensure indices are valid and remove duplicates
+        frame_indices = np.unique(np.clip(frame_indices, 0, total_frames_in_video - 1))
+        
+        result = decode_with_multithreading(frame_indices.tolist(), num_threads=num_threads, video_path=video_path)
+        # Add frame_indices to the result for later use
+        result["frame_indices"] = frame_indices
+        return result
+
+
+            
+    except Exception:
+        traceback.print_exc()
+        return None
+
+
+def smart_resize(
+    num_frames: int,
+    height: int,
+    width: int,
+    temporal_factor: int = 1,
+    factor: int = 32,
+    min_pixels: int = 128 * 128,
+    max_pixels: int = 16 * 16 * 2 * 2 * 2 * 6144,
+    per_frame_min_pixels: int = None,
+    per_frame_max_pixels: int = None,
+):
+    if num_frames < temporal_factor:
+        raise ValueError(f"t:{num_frames} must be larger than temporal_factor:{temporal_factor}")
+    if height < factor or width < factor:
+        raise ValueError(f"height:{height} or width:{width} must be larger than factor:{factor}")
+    elif max(height, width) / min(height, width) > 200:
+        raise ValueError(
+            f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}"
+        )
+    h_bar = round(height / factor) * factor
+    w_bar = round(width / factor) * factor
+    t_bar = round(num_frames / temporal_factor) * temporal_factor
+
+    # Step 1: Apply per-frame upper limit constraint
+    if per_frame_max_pixels is not None and h_bar * w_bar > per_frame_max_pixels:
+        beta = math.sqrt((height * width) / per_frame_max_pixels)
+        h_bar = max(factor, math.floor(height / beta / factor) * factor)
+        w_bar = max(factor, math.floor(width / beta / factor) * factor)
+
+    # Step 2: Apply 3D volume constraints (frames * height * width)
+    if t_bar * h_bar * w_bar > max_pixels:
+        beta = math.sqrt((num_frames * height * width) / max_pixels)
+        h_bar = max(factor, math.floor(height / beta / factor) * factor)
+        w_bar = max(factor, math.floor(width / beta / factor) * factor)
+    elif t_bar * h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (num_frames * height * width))
+        h_bar = math.ceil(height * beta / factor) * factor
+        w_bar = math.ceil(width * beta / factor) * factor
+
+    # Step 3: Ensure per-frame lower limit is respected (after volume constraint)
+    # This guarantees single frame stays within [per_frame_min_pixels, per_frame_max_pixels]
+    if per_frame_min_pixels is not None and h_bar * w_bar < per_frame_min_pixels:
+        beta = math.sqrt(per_frame_min_pixels / (height * width))
+        h_bar = math.ceil(height * beta / factor) * factor
+        w_bar = math.ceil(width * beta / factor) * factor
+
+    return h_bar, w_bar
+
+
+class MossVLVideoProcessorInitKwargs(VideosKwargs):
+    patch_size: Optional[int]
+    temporal_patch_size: Optional[int]
+    merge_size: Optional[int]
+    min_frames: Optional[int]
+    max_frames: Optional[int]
+    video_fps: Optional[Union[int, float]]
+    num_extract_threads: Optional[int]
+    # Total 3D volume budget across all videos; distributed proportionally per video by T*H*W
+    video_max_pixels: Optional[int]
+
+
+@add_start_docstrings(
+    "Constructs a fast Moss-VL video processor that dynamically resizes videos based on the original videos.",
+    BASE_VIDEO_PROCESSOR_DOCSTRING,
+    """
+        patch_size (`int`, *optional*, defaults to 16):
+            The spacial patch size of the vision encoder.
+        temporal_patch_size (`int`, *optional*, defaults to 1):
+            The temporal patch size of the vision encoder.
+        merge_size (`int`, *optional*, defaults to 2):
+            The merge size of the vision encoder to llm encoder.
+        video_fps (`float`, *optional*, defaults to 1.0):
+            Target frames per second for video sampling.
+        min_frames (`int`, *optional*, defaults to 1):
+            Minimum number of frames to sample from a video.
+        max_frames (`int`, *optional*, defaults to 256):
+            Maximum number of frames to sample from a video.
+        num_extract_threads (`int`, *optional*, defaults to 4):
+            Number of threads to use for frame extraction.
+    """,
+)
+class MossVLVideoProcessor(BaseVideoProcessor):
+    resample = PILImageResampling.BICUBIC
+    size = {"shortest_edge": 128 * 32 * 32, "longest_edge": 32 * 32 * 768}
+    image_mean = [0.5, 0.5, 0.5]
+    image_std = [0.5, 0.5, 0.5]
+    do_resize = True
+    do_rescale = True
+    do_normalize = True
+    do_convert_rgb = True
+    patch_size = 16
+    temporal_patch_size = 1
+    merge_size = 2
+    video_fps = 1.0
+    min_frames = 1
+    max_frames = 256
+    num_extract_threads = 4
+    do_sample_frames = True
+    # Total 3D volume budget across all videos; distributed proportionally per video by T*H*W
+    video_max_pixels = None  # read from config
+    valid_kwargs = MossVLVideoProcessorInitKwargs
+    model_input_names = ["pixel_values_videos", "video_grid_thw"]
+
+    def __init__(self, **kwargs: Unpack[MossVLVideoProcessorInitKwargs]):
+        super().__init__(**kwargs)
+        if self.size is not None and (
+            self.size.get("shortest_edge", None) is None or self.size.get("longest_edge", None) is None
+        ):
+            raise ValueError("size must contain 'shortest_edge' and 'longest_edge' keys.")
+
+    def _further_process_kwargs(
+        self,
+        size: Optional[SizeDict] = None,
+        **kwargs,
+    ) -> dict:
+        """
+        Update kwargs that need further processing before being validated
+        Can be overridden by subclasses to customize the processing of kwargs.
+        """
+        if size is not None and ("shortest_edge" not in size or "longest_edge" not in size):
+            raise ValueError("size must contain 'shortest_edge' and 'longest_edge' keys.")
+
+        return super()._further_process_kwargs(size=size, **kwargs)
+
+    def _get_video_path_from_input(self, video_input: Union[str, Dict[str, Any]]) -> str:
+        """Normalize a video input into a video path."""
+        if isinstance(video_input, dict):
+            return video_input["video_path"]
+        return video_input
+
+    def _get_video_duration_seconds(self, video_input: Union[str, Dict[str, Any]]) -> float:
+        """Get video duration in seconds for weighted frame-budget allocation."""
+        video_path = cached_clean_video_streams(self._get_video_path_from_input(video_input))
+        decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+        try:
+            metadata = decoder.metadata
+            duration = None
+            if (
+                metadata.end_stream_seconds_from_content is not None
+                and metadata.begin_stream_seconds_from_content is not None
+            ):
+                duration = metadata.end_stream_seconds_from_content - metadata.begin_stream_seconds_from_content
+            if duration is None or duration <= 0:
+                duration = metadata.duration_seconds
+            return max(0.0, float(duration or 0.0))
+        finally:
+            del decoder
+
+    def _allocate_max_frames_for_multiple_videos(
+        self,
+        video_inputs: List[Union[str, Dict[str, Any]]],
+        total_max_frames: Optional[int],
+    ) -> List[Optional[int]]:
+        """
+        Treat max_frames as a total budget for multi-video input and allocate it by duration.
+
+        The returned values are per-video max_frames. Segment dict inputs still keep their
+        existing per-segment weighting logic after receiving the video-level allocation.
+        """
+        if not video_inputs:
+            return []
+        if total_max_frames is None or len(video_inputs) == 1:
+            return [total_max_frames] * len(video_inputs)
+
+        total_max_frames = int(total_max_frames)
+        num_videos = len(video_inputs)
+        if total_max_frames < num_videos:
+            logger.warning(
+                "Received max_frames=%s for %s videos. At least one frame per video is required, "
+                "so falling back to 1 frame per video.",
+                total_max_frames,
+                num_videos,
+            )
+            return [1] * num_videos
+
+        video_durations = [self._get_video_duration_seconds(video_input) for video_input in video_inputs]
+        total_duration = sum(video_durations)
+
+        # Reserve one frame per video first, then distribute the remaining budget by duration.
+        allocations = [1] * num_videos
+        remaining_budget = total_max_frames - num_videos
+        if remaining_budget == 0:
+            return allocations
+
+        if total_duration <= 0:
+            raw_extra_allocations = [remaining_budget / num_videos] * num_videos
+        else:
+            raw_extra_allocations = [
+                remaining_budget * (duration / total_duration) for duration in video_durations
+            ]
+
+        base_extra_allocations = [int(math.floor(value)) for value in raw_extra_allocations]
+        allocations = [base + extra for base, extra in zip(allocations, base_extra_allocations)]
+
+        remainder = remaining_budget - sum(base_extra_allocations)
+        if remainder > 0:
+            fractional_parts = [
+                (raw_value - base_value, index)
+                for index, (raw_value, base_value) in enumerate(zip(raw_extra_allocations, base_extra_allocations))
+            ]
+            fractional_parts.sort(key=lambda item: (-item[0], item[1]))
+            for _, index in fractional_parts[:remainder]:
+                allocations[index] += 1
+
+        return allocations
+
+    def calculate_num_frames(
+        self,
+        metadata: VideoMetadata,
+        num_frames: Optional[int] = None,
+        fps: Optional[Union[int, float]] = None,
+        min_frames: Optional[int] = None,
+        max_frames: Optional[int] = None,
+        **kwargs,
+    ) -> int:
+        """
+        Calculate the number of frames to sample using fps-based logic with min/max constraints.
+        
+        Logic:
+        1. Calculate target_frames based on fps and video duration
+        2. Apply min_frames and max_frames constraints
+        3. Apply max_allowed_frames protection (rough cap from total video_max_pixels budget)
+        4. Return the number of frames to sample
+        
+        Args:
+            metadata (`VideoMetadata`):
+                Metadata of the video containing information about total duration, fps and total number of frames.
+            num_frames (`int`, *optional*):
+                Maximum number of frames to sample. If provided, overrides fps-based calculation.
+            fps (`int` or `float`, *optional*):
+                Target frames to sample per second. Defaults to `self.video_fps`.
+            min_frames (`int`, *optional*):
+                Minimum number of frames to sample. If None, uses self.min_frames.
+            max_frames (`int`, *optional*):
+                Maximum number of frames to sample. If None, uses self.max_frames.
+        Returns:
+            int:
+                Number of frames to sample.
+        """
+        if fps is not None and num_frames is not None:
+            raise ValueError("`num_frames` and `fps` are mutually exclusive arguments, please use only one!")
+
+        total_num_frames = metadata.total_num_frames
+        
+        # Use provided min/max or fall back to defaults
+        effective_min_frames = min_frames if min_frames is not None else self.min_frames
+        effective_max_frames = max_frames if max_frames is not None else self.max_frames
+        
+        # Rough per-video frame cap derived from the multi-video total budget
+        # (exact allocation happens later in _preprocess via weighted distribution)
+        per_frame_min_pixels = self.size.get("shortest_edge", None) if self.size else None
+        video_max_pixels = getattr(self, "video_max_pixels", None)
+        if per_frame_min_pixels is not None and video_max_pixels is not None and per_frame_min_pixels > 0:
+            max_allowed_frames = video_max_pixels // per_frame_min_pixels
+            effective_max_frames = min(effective_max_frames, max_allowed_frames)
+        
+        # Get video duration
+        if hasattr(metadata, 'duration') and metadata.duration is not None:
+            duration = metadata.duration
+        else:
+            video_fps = metadata.fps
+            if video_fps is not None and video_fps > 0:
+                duration = total_num_frames / video_fps
+            else:
+                # Fallback: assume 24 fps
+                video_fps = 24.0
+                duration = total_num_frames / video_fps
+                logger.warning_once(
+                    "Could not determine video fps from metadata, defaulting to 24 fps for duration calculation."
+                )
+
+        # Use provided fps or default
+        target_fps = fps if fps is not None else self.video_fps
+        
+        # Calculate target frames based on fps and duration
+        if num_frames is None:
+            # Calculate how many frames we should sample based on target fps
+            target_total_frames = int(math.ceil(duration * target_fps - 1e-6))
+            
+            # Apply min/max constraints
+            sample_frames = max(target_total_frames, effective_min_frames)
+            sample_frames = min(sample_frames, effective_max_frames, total_num_frames)
+        else:
+            # If num_frames is explicitly provided, use it directly with constraints
+            sample_frames = min(max(num_frames, effective_min_frames), effective_max_frames, total_num_frames)
+
+        return sample_frames
+
+
+    def _decode_timestamps_with_decoder(
+        self,
+        decoder: VideoDecoder,
+        timestamps: List[float],
+        chunk_size: int = 128,
+    ) -> torch.Tensor:
+        if not timestamps:
+            return torch.empty(0)
+
+        frame_chunks = []
+        for start in range(0, len(timestamps), chunk_size):
+            frame_batch = decoder.get_frames_played_at(timestamps[start:start + chunk_size])
+            frame_chunks.append(frame_batch.data)
+
+        if len(frame_chunks) == 1:
+            return frame_chunks[0]
+        return torch.cat(frame_chunks, dim=0)
+
+    def _clamp_timestamps_for_decoder(
+        self,
+        timestamps: List[float],
+        torchcodec_metadata,
+    ) -> List[float]:
+        return clamp_timestamps_for_torchcodec(timestamps, torchcodec_metadata)
+
+    def _fetch_video_segments_batched(
+        self,
+        video_path: str,
+        segments: List[List[float]],
+        min_frames: Optional[int] = None,
+        max_frames: Optional[int] = None,
+        video_fps: Optional[float] = None,
+    ):
+        min_frames = max(1, min_frames if min_frames is not None else self.min_frames)
+        max_frames = max(1, max_frames if max_frames is not None else self.max_frames)
+        target_video_fps = video_fps if video_fps is not None else self.video_fps
+
+        video_path = cached_clean_video_streams(video_path)
+        decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+        try:
+            torchcodec_metadata = decoder.metadata
+            source_video_fps = torchcodec_metadata.average_fps
+
+            duration = None
+            if (
+                torchcodec_metadata.end_stream_seconds_from_content is not None
+                and torchcodec_metadata.begin_stream_seconds_from_content is not None
+            ):
+                duration = (
+                    torchcodec_metadata.end_stream_seconds_from_content
+                    - torchcodec_metadata.begin_stream_seconds_from_content
+                )
+            if duration is None or duration <= 0:
+                duration = torchcodec_metadata.duration_seconds
+
+            segment_durations = [
+                segment[1] - segment[0] if len(segment) == 2 else None
+                for segment in segments
+            ]
+            total_segment_duration = sum(d for d in segment_durations if d is not None)
+            num_range_segments = sum(1 for d in segment_durations if d is not None)
+
+            segment_timestamps = []
+            decode_timestamps = []
+            for i, segment in enumerate(segments):
+                if len(segment) == 1:
+                    actual_timestamps = self._clamp_timestamps_for_decoder([segment[0]], torchcodec_metadata)
+                    segment_timestamps.append(actual_timestamps)
+                    decode_timestamps.extend(actual_timestamps)
+                    continue
+
+                start_time, end_time = segment
+                segment_duration = end_time - start_time
+                target_frames = int(math.ceil(segment_duration * target_video_fps))
+
+                if total_segment_duration > 0:
+                    weight = segment_durations[i] / total_segment_duration
+                else:
+                    weight = 1.0 / num_range_segments if num_range_segments > 0 else 1.0
+
+                weighted_min_frames = max(1, int(round(min_frames * weight)))
+                weighted_max_frames = max(1, int(round(max_frames * weight)))
+                target_frames = max(target_frames, weighted_min_frames)
+                target_frames = min(target_frames, weighted_max_frames)
+
+                if target_frames == 1:
+                    actual_timestamps = [start_time]
+                else:
+                    actual_timestamps = np.linspace(
+                        start_time,
+                        end_time,
+                        target_frames,
+                        endpoint=False,
+                    ).tolist()
+
+                actual_timestamps = self._clamp_timestamps_for_decoder(actual_timestamps, torchcodec_metadata)
+                segment_timestamps.append(actual_timestamps)
+                decode_timestamps.extend(actual_timestamps)
+
+            flat_frames = self._decode_timestamps_with_decoder(decoder, decode_timestamps)
+
+            videos = []
+            metadata = []
+            frame_offset = 0
+            for actual_timestamps in segment_timestamps:
+                sample_count = len(actual_timestamps)
+                video_tensor = flat_frames[frame_offset:frame_offset + sample_count]
+                frame_offset += sample_count
+
+                video_metadata = VideoMetadata(
+                    total_num_frames=sample_count,
+                    fps=source_video_fps,
+                    duration=duration,
+                    video_backend="torchcodec",
+                    height=torchcodec_metadata.height,
+                    width=torchcodec_metadata.width,
+                    frames_indices=None,
+                )
+                video_metadata.actual_timestamps = actual_timestamps
+
+                videos.append(video_tensor)
+                metadata.append(video_metadata)
+
+            return videos, metadata
+        finally:
+            del decoder
+
+
+    def _fetch_video_segment(
+        self,
+        video_path: str,
+        segment: List[float],
+        min_frames: Optional[int] = None,
+        max_frames: Optional[int] = None,
+        video_fps: Optional[float] = None,
+    ):
+        """
+        Fetch video frames for a specific segment.
+        
+        Args:
+            video_path: Path to the video file
+            segment: [start, end] for a segment (left-closed, right-open) or [time] for a single frame
+            min_frames: Minimum frames for this segment (weighted). Defaults to self.min_frames. Must be >= 1.
+            max_frames: Maximum frames for this segment (weighted). Defaults to self.max_frames. Must be >= 1.
+            video_fps: Target frames per second for video sampling. If None, uses self.video_fps.
+        
+        Returns:
+            Tuple of (video_tensor, video_metadata)
+        """
+        # Use provided min/max or fall back to defaults, ensure >= 1
+        min_frames = max(1, min_frames if min_frames is not None else self.min_frames)
+        max_frames = max(1, max_frames if max_frames is not None else self.max_frames)
+        # Use provided video_fps or fall back to self.video_fps
+        target_video_fps = video_fps if video_fps is not None else self.video_fps
+        
+        video_path = clean_video_streams(video_path)
+        decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+        try:
+            torchcodec_metadata = decoder.metadata
+            
+            video_fps = torchcodec_metadata.average_fps
+            
+            # Calculate duration
+            duration = None
+            if torchcodec_metadata.end_stream_seconds_from_content is not None and torchcodec_metadata.begin_stream_seconds_from_content is not None:
+                duration = torchcodec_metadata.end_stream_seconds_from_content - torchcodec_metadata.begin_stream_seconds_from_content
+            if duration is None or duration <= 0:
+                duration = torchcodec_metadata.duration_seconds
+            
+            if len(segment) == 1:
+                # Single frame at specified time
+                actual_timestamps = self._clamp_timestamps_for_decoder([segment[0]], torchcodec_metadata)
+                frame_batch = decoder.get_frames_played_at(actual_timestamps)
+                video_tensor = frame_batch.data
+                sample_count = 1
+            else:
+                # Segment [start, end) - left-closed, right-open interval
+                start_time, end_time = segment
+                segment_duration = end_time - start_time
+                
+                # Calculate number of frames to sample for this segment
+                target_frames = int(math.ceil(segment_duration * target_video_fps))
+                target_frames = max(target_frames, min_frames)
+                target_frames = min(target_frames, max_frames)
+                
+                # Generate timestamps for uniform sampling within segment
+                if target_frames == 1:
+                    actual_timestamps = [start_time]  # Use start_time for single frame
+                else:
+                    # Sample uniformly within [start, end), endpoint=False for left-closed right-open
+                    actual_timestamps = np.linspace(start_time, end_time, target_frames, endpoint=False).tolist()
+                
+                actual_timestamps = self._clamp_timestamps_for_decoder(actual_timestamps, torchcodec_metadata)
+
+                # Use multithreading for extraction
+                result = timestamp_decode_with_multithreading(actual_timestamps, self.num_extract_threads, video_path)
+                video_tensor = result["data"]
+                sample_count = len(actual_timestamps)
+            
+            # Create VideoMetadata
+            video_metadata = VideoMetadata(
+                total_num_frames=sample_count,
+                fps=video_fps,
+                duration=duration,
+                video_backend="torchcodec",
+                height=torchcodec_metadata.height,
+                width=torchcodec_metadata.width,
+                frames_indices=None
+            )
+            
+            # Store actual timestamps as a custom attribute for _calculate_timestamps to use
+            video_metadata.actual_timestamps = actual_timestamps
+            
+            return video_tensor, video_metadata
+        finally:
+            del decoder
+
+    def fetch_videos(
+        self, 
+        video_url_or_urls: Union[str, Dict[str, Any], List[Union[str, Dict[str, Any]]]], 
+        sample_indices_fn=None, 
+        video_fps: Optional[float] = None,
+        min_frames: Optional[int] = None,
+        max_frames: Optional[int] = None,
+    ):
+        """
+        Override fetch_videos to use torchcodec for frame extraction.
+        
+        This method uses torchcodec with multithreading for efficient frame extraction.
+        Frame count is calculated by the calculate_num_frames method
+        (fps-based with min/max constraints).
+        
+        Args:
+            video_url_or_urls: Can be one of:
+                - str: Single video path
+                - Dict: Video with segments {"video_path": str, "segments": List[List[float]]}
+                - List[Union[str, Dict]]: List of video paths or segment dicts
+            sample_indices_fn: (Not used) Kept for compatibility with base class signature.
+            video_fps: Target frames per second for video sampling. If None, uses self.video_fps.
+            min_frames: Minimum number of frames to sample. If None, uses self.min_frames.
+            max_frames: Maximum number of frames to sample. If None, uses self.max_frames.
+        
+        Returns:
+            Tuple of (videos, metadata) where videos are torch.Tensors and metadata are VideoMetadata objects.
+        """
+        # Use provided values or fall back to self defaults
+        effective_video_fps = video_fps if video_fps is not None else self.video_fps
+        effective_min_frames = min_frames if min_frames is not None else self.min_frames
+        effective_max_frames = max_frames if max_frames is not None else self.max_frames
+        # Handle recursive calls for lists
+        if isinstance(video_url_or_urls, list):
+            all_videos = []
+            all_metadata = []
+            if len(video_url_or_urls) == 1:
+                per_video_max_frames = [effective_max_frames]
+            else:
+                per_video_max_frames = self._allocate_max_frames_for_multiple_videos(
+                    video_url_or_urls,
+                    effective_max_frames,
+                )
+            for x, allocated_max_frames in zip(video_url_or_urls, per_video_max_frames):
+                result = self.fetch_videos(
+                    x, 
+                    video_fps=effective_video_fps,
+                    min_frames=effective_min_frames,
+                    max_frames=allocated_max_frames,
+                )
+                # Check if result is from segment expansion (returns lists) or single item
+                if isinstance(result[0], list):
+                    all_videos.extend(result[0])
+                    all_metadata.extend(result[1])
+                else:
+                    all_videos.append(result[0])
+                    all_metadata.append(result[1])
+            return all_videos, all_metadata
+        
+        # Handle dict with segments - returns lists (one per segment)
+        if isinstance(video_url_or_urls, dict):
+            video_path = video_url_or_urls["video_path"]
+            segments = video_url_or_urls["segments"]
+            
+            return self._fetch_video_segments_batched(
+                video_path,
+                segments,
+                min_frames=effective_min_frames,
+                max_frames=effective_max_frames,
+                video_fps=effective_video_fps,
+            )
+        
+        # Single video path
+        video_path = video_url_or_urls
+        
+        # Clean video streams first (remove extra streams if needed)
+        video_path = cached_clean_video_streams(video_path)
+
+        decoder = None
+        try:
+            # Create VideoDecoder only once for both metadata and frame extraction
+            decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+            torchcodec_metadata = decoder.metadata
+
+            duration = None
+            if torchcodec_metadata.end_stream_seconds_from_content is not None and torchcodec_metadata.begin_stream_seconds_from_content is not None:
+                duration = torchcodec_metadata.end_stream_seconds_from_content - torchcodec_metadata.begin_stream_seconds_from_content
+            
+            if duration is None or duration <= 0:
+                duration = torchcodec_metadata.duration_seconds
+            
+            # Use num_frames_from_content for accurate frame count (consistent with extraction)
+            total_frames_in_video = torchcodec_metadata.num_frames_from_content
+            
+            # Create VideoMetadata object for sample_frames method
+            temp_metadata = VideoMetadata(
+                total_num_frames=total_frames_in_video,
+                fps=torchcodec_metadata.average_fps,
+                duration=duration,
+                video_backend="torchcodec",
+                height=torchcodec_metadata.height,
+                width=torchcodec_metadata.width,
+                frames_indices=None
+            )
+            
+            # Use calculate_num_frames method to get the number of frames to sample
+            sample_frames_count = self.calculate_num_frames(
+                temp_metadata, 
+                fps=effective_video_fps,
+                min_frames=effective_min_frames,
+                max_frames=effective_max_frames,
+            )
+            
+            # Ensure sample count is valid
+            effective_sample_count = min(sample_frames_count, total_frames_in_video)
+            if effective_sample_count == 0:
+                raise ValueError(f"Cannot extract frames: video has 0 frames or specified frame count is 0")
+            
+            # Generate uniform frame indices
+            frame_indices = np.linspace(0, total_frames_in_video - 1, effective_sample_count).astype(np.int32)
+            # Ensure indices are valid and remove duplicates
+            frame_indices = np.unique(np.clip(frame_indices, 0, total_frames_in_video - 1))
+            
+            # Extract frames using multithreading (decoder is created inside each thread for thread safety)
+            result = decode_with_multithreading(frame_indices.tolist(), num_threads=self.num_extract_threads, video_path=video_path)
+            
+            # Extract frame tensor (N, C, H, W)
+            frames_tensor = result["data"]
+            
+            # Create final VideoMetadata object
+            video_metadata = VideoMetadata(
+                total_num_frames=len(frame_indices),
+                fps=torchcodec_metadata.average_fps,
+                duration=duration,
+                video_backend="torchcodec",
+                height=torchcodec_metadata.height,
+                width=torchcodec_metadata.width,
+                frames_indices=frame_indices
+            )
+            
+            # Ensure frames are in (T, C, H, W) format
+            if frames_tensor.dim() == 4:  # (N, C, H, W)
+                video_tensor = frames_tensor
+            else:
+                raise ValueError(f"Unexpected frame tensor shape: {frames_tensor.shape}")
+            
+            return video_tensor, video_metadata
+            
+        except Exception as e:
+            logger.error(f"Error loading video {video_path}: {e}")
+            traceback.print_exc()
+            raise ValueError(f"Failed to load video {video_path}: {e}")
+        finally:
+            if decoder is not None:
+                del decoder
+
+    def _preprocess(
+        self,
+        videos: list[torch.Tensor],
+        do_convert_rgb: bool = True,
+        do_resize: bool = True,
+        size: Optional[SizeDict] = None,
+        interpolation: PILImageResampling = PILImageResampling.BICUBIC,
+        do_rescale: bool = True,
+        rescale_factor: float = 1 / 255.0,
+        do_normalize: bool = True,
+        image_mean: Optional[Union[float, list[float]]] = None,
+        image_std: Optional[Union[float, list[float]]] = None,
+        patch_size: Optional[int] = None,
+        temporal_patch_size: Optional[int] = None,
+        merge_size: Optional[int] = None,
+        return_tensors: Optional[Union[str, TensorType]] = None,
+        **kwargs,
+    ):
+        grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
+        resized_videos_grouped = {}
+
+        video_max_pixels = getattr(self, "video_max_pixels", None)
+        if video_max_pixels is not None:
+            total_volume = sum(
+                sv.shape[0] * sv.shape[1] * sv.shape[3] * sv.shape[4]
+                for sv in grouped_videos.values()
+            )
+        else:
+            total_volume = 0
+
+        for shape, stacked_videos in grouped_videos.items():
+            B, T, C, H, W = stacked_videos.shape
+            num_frames, height, width = T, H, W
+            # Convert to RGB if needed (reuse from base class)
+            if do_convert_rgb:
+                stacked_videos = self.convert_to_rgb(stacked_videos)
+            if do_resize:
+                if video_max_pixels is not None and total_volume > 0:
+                    allocated_max_pixels = int(video_max_pixels * (T * H * W) / total_volume)
+                else:
+                    allocated_max_pixels = size.longest_edge
+                resized_height, resized_width = smart_resize(
+                    num_frames=num_frames,
+                    height=height,
+                    width=width,
+                    temporal_factor=temporal_patch_size,
+                    factor=patch_size * merge_size,
+                    min_pixels=size.shortest_edge,
+                    max_pixels=allocated_max_pixels,
+                    per_frame_min_pixels=size.shortest_edge,
+                    per_frame_max_pixels=size.longest_edge,
+                )
+                stacked_videos = stacked_videos.view(B * T, C, H, W)
+                stacked_videos = self.resize(
+                    stacked_videos,
+                    size=SizeDict(height=resized_height, width=resized_width),
+                    interpolation=interpolation,
+                )
+                stacked_videos = stacked_videos.view(B, T, C, resized_height, resized_width)
+            resized_videos_grouped[shape] = stacked_videos
+        resized_videos = reorder_videos(resized_videos_grouped, grouped_videos_index)
+
+        # Group videos by size for further processing
+        # Needed in case do_resize is False, or resize returns videos with different sizes
+        grouped_videos, grouped_videos_index = group_videos_by_shape(resized_videos)
+        processed_videos_grouped = {}
+        processed_grids = {}
+        for shape, stacked_videos in grouped_videos.items():
+            resized_height, resized_width = get_image_size(stacked_videos[0], channel_dim=ChannelDimension.FIRST)
+
+            # Fused rescale and normalize
+            stacked_videos = self.rescale_and_normalize(
+                stacked_videos, do_rescale, rescale_factor, do_normalize, image_mean, image_std
+            )
+            patches = stacked_videos
+
+            # Check that videos have `num_frames` divisible by `temporal_patch_size`
+            if patches.shape[1] % temporal_patch_size != 0:
+                repeats = patches[:, -1:].repeat(1, temporal_patch_size - 1, 1, 1, 1)
+                patches = torch.cat([patches, repeats], dim=1)
+            batch_size, grid_t, channel = patches.shape[:3]
+            grid_t = grid_t // temporal_patch_size
+            grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+
+            patches = patches.view(
+                batch_size,
+                grid_t,
+                temporal_patch_size,
+                channel,
+                grid_h // merge_size,
+                merge_size,
+                patch_size,
+                grid_w // merge_size,
+                merge_size,
+                patch_size,
+            )
+            patches = patches.permute(0, 1, 4, 7, 5, 8, 3, 2, 6, 9)
+            flatten_patches = patches.reshape(
+                batch_size,
+                grid_t * grid_h * grid_w,
+                channel * temporal_patch_size * patch_size * patch_size,
+            )
+
+            processed_videos_grouped[shape] = flatten_patches
+            processed_grids[shape] = [[grid_t, grid_h, grid_w]] * batch_size
+
+        processed_videos = reorder_videos(processed_videos_grouped, grouped_videos_index)
+        processed_grids = reorder_videos(processed_grids, grouped_videos_index)
+        pixel_values_videos = torch.cat(processed_videos, dim=0)
+        video_grid_thw = torch.tensor(processed_grids)
+        data = {
+            "pixel_values_videos": pixel_values_videos,
+            "video_grid_thw": video_grid_thw,
+        }
+
+        return BatchFeature(data=data, tensor_type=return_tensors)
+
+    def preprocess(
+        self,
+        videos: Union[str, Dict[str, Any], List[Union[str, Dict[str, Any]]]],
+        **kwargs,
+    ) -> BatchFeature:
+        """
+        Preprocess videos for the model.
+        
+        This method overrides the base class to handle two video input formats:
+        1. String path: "path/to/video.mp4"
+        2. Dict with segments: {"video_path": "...", "segment": [[start, end], [time], ...]}
+        
+        Args:
+            videos: Video input(s) in one of the supported formats.
+            **kwargs: Additional arguments passed to _preprocess.
+            
+        Returns:
+            BatchFeature with pixel_values_videos, video_grid_thw, and optionally video_metadata.
+        """
+        # Validate kwargs
+        validate_kwargs(
+            captured_kwargs=kwargs.keys(),
+            valid_processor_keys=list(self.valid_kwargs.__annotations__.keys()) + ["return_tensors"],
+        )
+        
+        # Set default kwargs from self
+        for kwarg_name in self.valid_kwargs.__annotations__:
+            kwargs.setdefault(kwarg_name, getattr(self, kwarg_name, None))
+        
+        # Pop kwargs that are handled separately
+        return_tensors = kwargs.pop("return_tensors", None)
+        return_metadata = kwargs.pop("return_metadata", False)
+        input_data_format = kwargs.pop("input_data_format", None)
+        device = kwargs.pop("device", None)
+        kwargs.pop("video_metadata", None)  # We generate our own metadata
+        kwargs.pop("do_sample_frames", None)  # We handle sampling ourselves
+        kwargs.pop("data_format", None)  # Not used
+        
+        # Normalize input to list format
+        if not isinstance(videos, list):
+            videos = [videos]
+        
+        # Get video processing params from kwargs (may be passed explicitly for per-batch configuration)
+        video_fps = kwargs.pop("video_fps", None)
+        min_frames = kwargs.pop("min_frames", None)
+        max_frames = kwargs.pop("max_frames", None)
+        
+        # Use fetch_videos to handle both string and dict formats
+        video_tensors, video_metadata = self.fetch_videos(
+            videos, 
+            video_fps=video_fps,
+            min_frames=min_frames,
+            max_frames=max_frames,
+        )
+        
+        # Prepare video tensors using _prepare_input_videos
+        prepared_videos = self._prepare_input_videos(
+            videos=video_tensors,
+            input_data_format=input_data_format,
+            device=device,
+        )
+        
+        # Process kwargs for _preprocess
+        kwargs = self._further_process_kwargs(**kwargs)
+        self._validate_preprocess_kwargs(**kwargs)
+        
+        # Call _preprocess with prepared videos
+        result = self._preprocess(videos=prepared_videos, return_tensors=return_tensors, **kwargs)
+        
+        # Add metadata if requested
+        if return_metadata:
+            result["video_metadata"] = video_metadata
+        
+        return result
+
+
+__all__ = ["MossVLVideoProcessor"]
+