MiniCPM5-2B (LiteRT-LM)

This repository hosts the LiteRT-LM (LiteRT formerly known as TensorFlow Lite) version of openbmb/MiniCPM5-2B, optimized for fully on-device inference on mobile and edge hardware.

---

Available Models

Recommend to use below both CPU and GPU compatable models, originally from mlboydaisuke/MiniCPM5-2B-LiteRT Requires litert-lm ≥ 0.16 (thought channel + ThinkingConfig); measured here on 0.17.0.

File	Recipe	Size
MiniCPM5-2B_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 embedding	1.55 GB
MiniCPM5-2B_int8.litertlm	int8 dynamic on linears + embedding; fp32 activations declared (see notes)	2.60 GB

The int4 file is the phone file (smaller, fastest GPU decode on every platform measured) — best used for direct answers or short reasoning; see the thinking-mode note below. int8 is the file for reasoning that has to complete: its thinking chains are ~3–4× shorter than int4's on the same questions and terminate where int4 runs into the token budget. int8's main weight section is 2.33 GB, above the single-section mmap ceiling of default-entitlement iOS apps, so it is a desktop / Android build.

Further, below are some CPU-only models for exploration:

• minicpm_wi4c_wi8_afp32.litertlm: This model features mixed INT4/INT8 weight-only quantization with FP32 activations (afp32). MLP projections use channelwise INT4 with Hadamard rotation; all remaining weights (attention, embedding, and lmhead) use channelwise INT8.
• minicpm_wi8_afp32.litertlm: This model features weight-only INT8 quantization (wi8) with FP32 activations (afp32).

What is MiniCPM?

MiniCPM5-2B is part of the MiniCPM5 series from OpenBMB. It is a dense 2.5B-parameter Transformer built specifically for on-device, local, and resource-constrained deployment.

Highlights

• 🧠 Hybrid Reasoning — a single checkpoint serves as both a fast assistant and a deliberate reasoner via a built-in <think> template (enable_thinking).
• 🛠️ Tool use — native tool-calling support in the chat template.
• 📏 Long context — native 131,072-token context length.
• 📱 Built for the edge — compact footprint designed for local assistants, coding agents, and tool-use workflows.

Model Information

Item	Value
Type	Causal Language Model
Architecture	Standard LlamaForCausalLM
Parameters	2,516,756,480 (~2.5B)
Non-Embedding Parameters	1,981,982,720
Layers	42
Attention Heads (GQA)	16 (Q) / 2 (KV), head dim 128
Hidden / Intermediate Size	2048 / 6144
Vocabulary Size	130,560
Context Length	131,072

---

Use the model

Edge Gallery App (Android)

1. Get the App: Install the app from Google Play or download the latest APK from the GitHub releases page.
2. Importing the Model: Navigate to the Model manager within the app and click the "+" (plus) icon in the bottom-right corner. Two options will appear:
   • Import from HF (Recommended): Select this option, and a dialog box will appear showing an example Hugging Face model URL. Enter the HF link for the desired .litertlm model and click submit. The model will then appear in your list, and you can proceed to download it (a Hugging Face account login is required).
   • From local model file: First, download the .litertlm model directly to your Android device, OR download it to your computer and push it via ADB (e.g., adb push MiniCPM5-2B_int4.litertlm /sdcard/Download/). Then, select this option, choose the downloaded file from your storage, configure your preferred parameters, and tap "Import".

For full details on importing models and other features, see the Edge Gallery App Wiki.

To build the demo app from source, please follow the instructions from the GitHub repository.

Try It (Desktop/CLI)

Install uv and run the model directly from the LiteRT-LM command line:

uv tool install litert-lm
uvx litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm --prompt="What is the capital of France?"

Performance

litert-lm benchmark (litert-lm 0.17.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, serialized, ≥300 s rest before each GPU reading; each backend confirmed to generate real text before its number was recorded:

File	Backend	Prefill (256)	Decode	TTFT	Init
int4	GPU (Metal)	1699 tok/s	92.8 tok/s	0.16 s	3.7 s
int4	CPU	149 tok/s	31.1 tok/s	1.76 s	4.5 s
int8 (fp32 activations)	GPU (Metal)	1405 tok/s	74.7 tok/s	0.20 s	3.0 s
int8 (fp32 activations)	CPU	161 tok/s	30.0 tok/s	1.62 s	15.0 s

Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno; litert_lm_advanced_main from the litert-lm v0.16.0 release kit, 205-token prompt with --benchmark, 2 runs per cell, ranges shown; a reasoning model decodes its own full response, so decode-turn lengths vary):

File	Backend	Prefill (205)	Decode	TTFT	Init	Peak RSS
int4	GPU (OpenCL)	401–411 tok/s	16.1–18.6 tok/s	0.56 s	11.2–13.1 s	1.14 GB
int4	CPU	39–72 tok/s	15.6–15.8 tok/s	2.9–5.3 s	3.1–5.7 s	2.12 GB
int8 (fp32 activations)	GPU (OpenCL)	150–160 tok/s	10.9–12.8 tok/s	1.4 s	4.0–6.3 s	1.10 GB
int8 (fp32 activations)	CPU	103–157 tok/s	11.7 tok/s	1.4–2.1 s	0.3 s	2.90 GB

GPU wins prefill (5–10× on the phone, 11× on the Mac) and time-to-first-token everywhere; on Adreno the int4 GPU decode edge over the same-device CPU is modest (~1.1×), while the int8 file's fp32 activations bring its GPU decode level with its CPU.

Links

• 🤗 Original model (BF16): openbmb/MiniCPM5-2B
• 🤗 Smaller sibling: litert-community/MiniCPM5-1B
• 📦 GitHub: OpenBMB/MiniCPM
• 🛠️ LiteRT docs: ai.google.dev/edge/litert

---

License

Released under the Apache-2.0 License, consistent with the upstream openbmb/MiniCPM5-2B.

Citation

@article{minicpm4,
  title={MiniCPM4: Ultra-efficient LLMs on end devices},
  author={MiniCPM, Team},
  journal={arXiv preprint arXiv:2506.07900},
  year={2025}
}