Instructions to use openbmb/MiniCPM5-2B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openbmb/MiniCPM5-2B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="openbmb/MiniCPM5-2B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openbmb/MiniCPM5-2B")
model = AutoModelForCausalLM.from_pretrained("openbmb/MiniCPM5-2B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use openbmb/MiniCPM5-2B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "openbmb/MiniCPM5-2B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM5-2B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/openbmb/MiniCPM5-2B

SGLang

How to use openbmb/MiniCPM5-2B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "openbmb/MiniCPM5-2B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM5-2B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "openbmb/MiniCPM5-2B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM5-2B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use openbmb/MiniCPM5-2B with Docker Model Runner:
```
docker model run hf.co/openbmb/MiniCPM5-2B
```

WebLLM / MLC LLM build (q4f16_1) for running MiniCPM5-2B in the browser

by ozhyhinas - opened about 11 hours ago

Discussion

ozhyhinas

about 11 hours ago

Hugging Face CLI

There is now a WebLLM / MLC LLM build of MiniCPM5-2B for running the model in the browser on WebGPU:

https://huggingface.co/ozhyhinas/MiniCPM5-2B-q4f16_1-MLC

q4f16_1 weights (42 shards, 1.42 GB, 4.5 bits per parameter) converted with mlc_llm convert_weight
WebGPU model library compiled against the mlc-llm/TVM pair that WebLLM 0.2.84 ships, so it loads with the current @mlc-ai/web-llm npm package as a custom appConfig entry (snippet on the model card)
ChatML conversation template with this model's stop tokens (</s>, <|im_end|>); enable_thinking: false in WebLLM produces the empty <think>\n\n</think> prefix from chat_template.jinja, so non-thinking mode works as in the reference implementation
SHA-256 manifest for every file and the exact build recipe on the card; Apache 2.0 as the base model

Tested in Chrome on Apple Silicon: cold load about 25 s, 17 to 29 tokens/s decode at 4-bit. A pull request adding the record to WebLLM's prebuilt model list is open at https://github.com/mlc-ai/web-llm/pull/855.

If you would like to link it from the README alongside the GGUF and MLX conversions, or host a copy under the openbmb namespace, happy to help with either.

OpenBMB-LLM

OpenBMB org about 8 hours ago

Thanks for doing this, and for documenting the build recipe and SHA-256 manifest so carefully — that's genuinely useful for anyone who wants to run the model in the browser.
We're keeping the model card's links limited for now, so we won't be adding it there. Leaving this discussion open, though, so people looking for a WebGPU build can find it. Nice work.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment