Instructions to use trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5", trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained("trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5", trust_remote_code=True, device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5

SGLang

How to use trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5 with Docker Model Runner:
```
docker model run hf.co/trl-internal-testing/tiny-Lfm2VlForConditionalGeneration-2.5
```

qgallouedec HF Staff commited on 30 days ago

Commit

140cb85

verified ·

1 Parent(s): e4b39e2

Upload Lfm2VlForConditionalGeneration

Browse files

Files changed (9) hide show

.gitattributes +1 -0
README.md +9 -0
chat_template.jinja +125 -0
config.json +148 -0
generation_config.json +16 -0
model.safetensors +3 -0
processor_config.json +39 -0
tokenizer.json +3 -0
tokenizer_config.json +18 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md ADDED Viewed

	@@ -0,0 +1,9 @@

+---
+library_name: transformers
+tags:
+- trl
+---
+# Tiny Lfm2VlForConditionalGeneration
+This is a minimal model built for unit tests in the [TRL](https://github.com/huggingface/trl) library.

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,125 @@

+{{- bos_token -}}
+{%- set preserve_thinking = preserve_thinking | default(false) -%}
+{%- macro format_arg_value(arg_value) -%}
+    {%- if arg_value is string -%}
+        {{- "'" + (arg_value | replace("\\", "\\\\") | replace("'", "\\'") | replace("\n", "\\n") | replace("\r", "\\r")) + "'" -}}
+    {%- elif arg_value is mapping or arg_value is iterable -%}
+        {{- arg_value | tojson -}}
+    {%- else -%}
+        {{- arg_value | string -}}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro parse_content(content) -%}
+    {%- if content is string -%}
+        {{- content -}}
+    {%- elif content is mapping -%}
+        {{- content | tojson -}}
+    {%- elif content is iterable -%}
+        {%- set _ns = namespace(result="") -%}
+        {%- for item in content -%}
+            {%- if item is string -%}
+                {%- set _ns.result = _ns.result + item -%}
+            {%- elif item is mapping and item.get("type") == "image" -%}
+                {%- set _ns.result = _ns.result + "<image>" -%}
+            {%- elif item is mapping and item.get("type") == "text" -%}
+                {%- set _ns.result = _ns.result + ((item.get("text") or "") | string) -%}
+            {%- else -%}
+                {%- set _ns.result = _ns.result + (item | tojson) -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {{- _ns.result -}}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro render_tool_calls(tool_calls) -%}
+    {%- set tool_calls_ns = namespace(tool_calls=[]) -%}
+    {%- for tool_call in tool_calls -%}
+        {%- set func = tool_call["function"] if "function" in tool_call else tool_call -%}
+        {%- set func_name = func["name"] -%}
+        {%- set func_args = func.get("arguments") -%}
+        {%- set args_ns = namespace(arg_strings=[]) -%}
+        {%- if func_args is mapping -%}
+            {%- for arg_name, arg_value in func_args.items() -%}
+                {%- set args_ns.arg_strings = args_ns.arg_strings + [arg_name + "=" + format_arg_value(arg_value)] -%}
+            {%- endfor -%}
+        {%- elif func_args is string and (func_args | trim) not in ["", "{}", "null"] -%}
+            {{- raise_exception("Tool call arguments must be a mapping, got a JSON-encoded string: parse arguments with json.loads() before applying the chat template") -}}
+        {%- endif -%}
+        {%- set tool_calls_ns.tool_calls = tool_calls_ns.tool_calls + [func_name + "(" + (args_ns.arg_strings | join(", ")) + ")"] -%}
+    {%- endfor -%}
+    {{- "<|tool_call_start|>[" + (tool_calls_ns.tool_calls | join(", ")) + "]<|tool_call_end|>" -}}
+{%- endmacro -%}
+{%- set ns = namespace(system_prompt="", last_user_index=-1) -%}
+{%- if messages and messages[0]["role"] == "system" -%}
+    {%- if messages[0].get("content") -%}
+        {%- set ns.system_prompt = parse_content(messages[0]["content"]) -%}
+    {%- endif -%}
+    {%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+    {%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
+    {%- for tool in tools -%}
+        {%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+        {%- endif -%}
+        {%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns.system_prompt = ns.system_prompt + "]" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- for message in messages -%}
+    {%- if message["role"] == "user" -%}
+        {%- set ns.last_user_index = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+{%- for message in messages -%}
+    {{- "<|im_start|>" + message.role + "\n" -}}
+    {%- if message.role == "assistant" -%}
+        {%- generation -%}
+        {%- set keep_thinking = preserve_thinking or loop.index0 > ns.last_user_index -%}
+        {%- set thinking = message.thinking or message.reasoning or message.reasoning_content -%}
+        {%- set thinking = thinking if thinking is string else "" -%}
+        {%- if thinking and keep_thinking -%}
+            {{- "<think>" + thinking + "</think>" -}}
+        {%- endif -%}
+        {%- set _cfm_tag = "CONTINUE_FINAL_MESSAGE_TAG " -%}
+        {%- set _has_cfm = false -%}
+        {%- set content = "" -%}
+        {%- if message.get("content") -%}
+            {%- set content = parse_content(message.content) -%}
+        {%- endif -%}
+        {%- if not keep_thinking and "</think>" in content -%}
+            {%- set content = content.split("</think>")[-1] | trim -%}
+        {%- endif -%}
+        {%- if content.endswith(_cfm_tag) -%}
+            {%- set _has_cfm = true -%}
+            {%- set _trunc_len = (content | length) - (_cfm_tag | length) -%}
+            {%- set content = content[:_trunc_len] -%}
+        {%- endif -%}
+        {{- content -}}
+        {%- if message.tool_calls -%}
+            {{- render_tool_calls(message.tool_calls) -}}
+        {%- endif -%}
+        {%- if _has_cfm -%}
+            {{- _cfm_tag -}}
+        {%- endif -%}
+        {{- "<|im_end|>\n" -}}
+        {%- endgeneration -%}
+    {%- else %}
+        {%- if message.get("content") -%}
+            {{- parse_content(message["content"]) -}}
+        {%- endif -%}
+        {{- "<|im_end|>\n" -}}
+    {%- endif %}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{- "<|im_start|>assistant\n" -}}
+{%- endif -%}

config.json ADDED Viewed

	@@ -0,0 +1,148 @@

+{
+  "architectures": [
+    "Lfm2VlForConditionalGeneration"
+  ],
+  "auto_map": {},
+  "bos_token_id": 124894,
+  "do_image_splitting": true,
+  "do_resize": true,
+  "downsample_factor": 2,
+  "dtype": "bfloat16",
+  "encoder_patch_size": 16,
+  "eos_token_id": 124900,
+  "freeze_language_model": false,
+  "freeze_multi_modal_projector": false,
+  "freeze_vision_tower": false,
+  "image_token_id": 124907,
+  "keep_trainable_parameters_fp32": true,
+  "language_model_lr_multiplier": 1.0,
+  "lfm2_attention_backend": "flash_varlen",
+  "lfm2_attention_fusion": "fused_linear",
+  "lfm2_flash_varlen_blhd_fastpath": false,
+  "lfm2_frozen_input_grad_only_linear": true,
+  "lfm2_mlp_fusion": "triton_swiglu",
+  "lfm2_rmsnorm_fusion": "none",
+  "lfm2_short_conv_frozen_recompute_in_proj": 0,
+  "lfm2_torch_compile_clone_outputs": false,
+  "lfm2_torch_compile_disable_cudagraphs": true,
+  "lfm2_torch_compile_dynamic": true,
+  "lfm2_torch_compile_fullgraph": false,
+  "lfm2_torch_compile_layers": "none",
+  "lfm2_torch_compile_mlp": false,
+  "lfm2_torch_compile_mode": "reduce-overhead",
+  "lfm2_vl_direct_image_merge_by_shape": true,
+  "lfm2_vl_forward_dtype": "bf16",
+  "lfm2_vl_frozen_vision_chunk_size": 2048,
+  "lfm2_vl_grouped_image_projector": true,
+  "lfm2_vl_grouped_vision_by_patch_count": false,
+  "lfm2_vl_grouped_vision_by_shape": true,
+  "lfm2_vl_inplace_image_merge": true,
+  "lfm2_vl_siglip_packed_vision": true,
+  "lfm2_vl_torch_compile_vision_encoder": true,
+  "lfm2_vl_torch_compile_vision_encoder_mode": "whole",
+  "lm_head_loss_mask_only": true,
+  "max_image_tokens": 256,
+  "max_num_patches": 1024,
+  "max_pixels_tolerance": 2.0,
+  "max_tiles": 10,
+  "min_image_tokens": 64,
+  "min_tiles": 1,
+  "model_type": "lfm2_vl",
+  "pad_token_id": 124893,
+  "projector_bias": true,
+  "projector_hidden_act": "gelu",
+  "projector_hidden_size": 32,
+  "projector_lr_multiplier": 1.0,
+  "projector_use_layernorm": false,
+  "siglip2_layernorm_fusion": "liger",
+  "text_config": {
+    "architectures": [
+      "Lfm2ForCausalLM"
+    ],
+    "block__name_mlp": "parallel_mlp_merged",
+    "block_auto_adjust_ff_dim": false,
+    "block_dim": 2048,
+    "block_ffn_dim_multiplier": 1.0,
+    "block_ffn_te_autocast": false,
+    "block_ffn_use_quantized_params": false,
+    "block_mlp_init_scale": 1.0,
+    "block_multiple_of": 256,
+    "block_norm_eps": 1e-05,
+    "block_out_init_scale": 1.0,
+    "block_use_swiglu": true,
+    "block_use_xavier_init": true,
+    "bos_token_id": 124894,
+    "conv_L_cache": 3,
+    "conv_bias": false,
+    "conv_dim": 2048,
+    "conv_use_xavier_init": true,
+    "dtype": "bfloat16",
+    "eos_token_id": 124900,
+    "hidden_size": 8,
+    "initializer_range": 0.02,
+    "intermediate_size": 32,
+    "layer_types": [
+      "conv",
+      "full_attention"
+    ],
+    "lfm2_attention_backend": "flash_varlen",
+    "lfm2_flash_varlen_blhd_fastpath": false,
+    "lfm2_frozen_input_grad_only_linear": true,
+    "max_position_embeddings": 128000,
+    "mm_config_image_embedder": {
+      "activation_checkpointing_layer_stride": 0,
+      "attn_implementation": "sdpa",
+      "cuda_sync_points": [],
+      "downsample": 2,
+      "encoder_load_balance": false,
+      "hidden_dim": 2048,
+      "hidden_state_index": -1,
+      "layer_norm": false,
+      "max_batch_size": 1024,
+      "pretrained_model_name_or_path": "google/siglip2-so400m-patch16-naflex",
+      "set_activation_checkpointing": false,
+      "use_image_special_tokens": true,
+      "use_pooling_head": false,
+      "use_slice_special_tokens": false,
+      "use_torch_pixel_unshuffle": false
+    },
+    "model_type": "lfm2",
+    "norm_eps": 1e-05,
+    "num_attention_heads": 4,
+    "num_heads": 32,
+    "num_hidden_layers": 2,
+    "num_key_value_heads": 2,
+    "output_softcap": 0.0,
+    "pad_token_id": 124893,
+    "rope_parameters": {
+      "rope_theta": 1000000.0,
+      "rope_type": "default"
+    },
+    "tie_embedding": true,
+    "tie_word_embeddings": true,
+    "use_cache": true,
+    "use_pos_enc": true,
+    "vocab_size": 128000
+  },
+  "tie_word_embeddings": true,
+  "tile_size": 512,
+  "transformers_version": "5.0.0",
+  "use_image_special_tokens": true,
+  "use_thumbnail": true,
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 16,
+    "intermediate_size": 32,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip2_vision_model",
+    "num_attention_heads": 4,
+    "num_channels": 3,
+    "num_hidden_layers": 2,
+    "num_patches": 256,
+    "patch_size": 16,
+    "vision_use_head": false
+  },
+  "vision_tower_lr_multiplier": 1.0
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,16 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 124894,
+  "do_sample": true,
+  "eos_token_id": [
+    124900
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 124893,
+  "repetition_penalty": 1.0,
+  "temperature": 0.2,
+  "top_k": 50,
+  "transformers_version": "5.0.0",
+  "use_cache": true
+}

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d95dc9b19ca8a18f78d50f40a99f1704e34be272c057d2720a2a93a7d05bf3e3
+size 2106448

processor_config.json ADDED Viewed

	@@ -0,0 +1,39 @@

+{
+  "image_processor": {
+    "data_format": "channels_first",
+    "do_image_splitting": true,
+    "do_normalize": true,
+    "do_pad": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "downsample_factor": 2,
+    "encoder_patch_size": 16,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Lfm2VlImageProcessorFast",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_image_tokens": 256,
+    "max_num_patches": 1024,
+    "max_pixels_tolerance": 2.0,
+    "max_tiles": 10,
+    "min_image_tokens": 64,
+    "min_tiles": 1,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_row_col_info": true,
+    "size": {
+      "height": 512,
+      "width": 512
+    },
+    "tile_size": 512,
+    "use_thumbnail": true
+  },
+  "processor_class": "Lfm2VlProcessor"
+}

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8096ecb9f54599d756c8de728a598a340bc1e43c0deb77ddd62456c38349fcee
+size 17905750

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,18 @@

+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "is_local": false,
+  "legacy": false,
+  "local_files_only": false,
+  "max_length": null,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_to_multiple_of": null,
+  "pad_token": "<|pad|>",
+  "pad_token_type_id": 0,
+  "padding_side": "left",
+  "processor_class": "Lfm2VlProcessor",
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false
+}