Instructions to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", trust_remote_code=True, device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModelForImageTextToText model = AutoModelForImageTextToText.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", trust_remote_code=True, dtype="auto", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct
fix(codec): merge per-canvas image_grid_thw into [[N,H,W]] per video
Browse filesThe codec branch in __call__ was emitting per-canvas image_grid_thw rows
(N, 3) = [[1, h, w]]*N, which caused the vision encoder's _build_cu_seqlens
(fixed_t=4) to treat each canvas as an isolated 1-frame sample instead of
grouping canvases into 4-frame attention windows. This silently degraded
video benchmark scores by ~1pt on videoeval_pro (45.07 vs reference 46.4).
Merge same-size canvases per video into a single [[N, H, W]] row so the
encoder restores the cross-canvas self-attention windows that the training
pipeline assumes.
|
@@ -308,7 +308,30 @@ class LlavaOnevision2Processor:
|
|
| 308 |
all_patch_positions.append(patch_positions)
|
| 309 |
|
| 310 |
out["pixel_values"] = torch.cat(all_pixel_values, dim=0)
|
| 311 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 312 |
out["patch_positions"] = torch.cat(all_patch_positions, dim=0)
|
| 313 |
text = rewritten_texts
|
| 314 |
# Codec branch handled the video. Suppress the frame-sampling block below.
|
|
|
|
| 308 |
all_patch_positions.append(patch_positions)
|
| 309 |
|
| 310 |
out["pixel_values"] = torch.cat(all_pixel_values, dim=0)
|
| 311 |
+
# Merge per-canvas grid_thw rows into a single [N, H, W] row per
|
| 312 |
+
# video so the vision encoder's _build_cu_seqlens (fixed_t=4)
|
| 313 |
+
# groups canvases into 4-frame attention windows, matching the
|
| 314 |
+
# training pipeline. Without this merge the encoder treats each
|
| 315 |
+
# canvas as an isolated 1-frame sample (no cross-canvas
|
| 316 |
+
# self-attention), which silently degrades video benchmark
|
| 317 |
+
# scores by ~1pt on videoeval_pro.
|
| 318 |
+
merged_grid_thw_rows = []
|
| 319 |
+
for grid in all_grid_thw:
|
| 320 |
+
if (
|
| 321 |
+
grid.shape[0] > 1
|
| 322 |
+
and bool(torch.all(grid[:, 1] == grid[0, 1]).item())
|
| 323 |
+
and bool(torch.all(grid[:, 2] == grid[0, 2]).item())
|
| 324 |
+
):
|
| 325 |
+
merged_grid_thw_rows.append(
|
| 326 |
+
torch.tensor(
|
| 327 |
+
[[int(grid.shape[0]), int(grid[0, 1]), int(grid[0, 2])]],
|
| 328 |
+
dtype=grid.dtype,
|
| 329 |
+
device=grid.device,
|
| 330 |
+
)
|
| 331 |
+
)
|
| 332 |
+
else:
|
| 333 |
+
merged_grid_thw_rows.append(grid)
|
| 334 |
+
out["image_grid_thw"] = torch.cat(merged_grid_thw_rows, dim=0)
|
| 335 |
out["patch_positions"] = torch.cat(all_patch_positions, dim=0)
|
| 336 |
text = rewritten_texts
|
| 337 |
# Codec branch handled the video. Suppress the frame-sampling block below.
|