README.md

---
license: apache-2.0
base_model: openbmb/MiniCPM5-2B
pipeline_tag: text-generation
library_name: coreai
tags:
- coreai
- core-ai
- coreml
- apple
- on-device
- iphone
- metal
- system-one
- typed-decisions
base_model_relation: quantized
---

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's `coreai-torch` (LLMs: `coreai.llm.export`) into `.aimodel` bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol ([apple-silicon-llm-bench](https://github.com/john-rocky/apple-silicon-llm-bench), macOS 27 beta 26A5353q, 2026-06-11).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) -->
This model has no row on [DeviceMark](https://devicemark.github.io/), the on-device LLM leaderboard.
<!-- gen-cards:devicemark end -->

# MiniCPM5-2B — Core AI (int8 block-32, runs on iPhone)

Apple **Core AI** (`.aimodel`) conversion of [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) —
OpenBMB's 2.5B on-device LLM (released 2026-09-06, the 1B's 42-layer sibling) with **hybrid
Think / No-Think reasoning**, native tool calling and **128K** context; OpenBMB reports it as
2B-class open-source SOTA (LiveCodeBench v6 69.1, AIME 2026 86.5, BFCL v4 66.6, SWE-bench Verified
46.4 on their card). Runs fully on-device on **iPhone** and Apple Silicon Macs (GPU, pipelined engine).

Part of the community Core AI model zoo: **https://github.com/john-rocky/coreai-model-zoo**

<!-- gen-cards:use-it begin id=minicpm5-2b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->
## Use it

**New to Core AI? [Start with CoreAIKit 0.7.1](https://github.com/john-rocky/coreai-kit#readme).** Follow its requirements and first-run steps for `qwen3-0.6b`, then open the same release's [ChatDemo](https://github.com/john-rocky/coreai-kit/tree/0.7.1/Examples/ChatDemo). The README records the tested OS/SDK and download size; model and device coverage is stated per example.

⚡ **One line** — run the kit's task op on this model
(`import CoreAIOps`; no session, no model plumbing, downloads on first use):

```swift
let tldr = try await CoreAI.summarize(text, options: .model("minicpm5-2b"))
```

Every op, one shape — [Cookbook](https://github.com/john-rocky/coreai-kit/blob/0.7.1/docs/COOKBOOK.md).

▶️ **Run it (source)** — the [ChatDemo runner](https://github.com/john-rocky/coreai-kit/tree/0.7.1/Examples/ChatDemo)
(GUI + CLI, one app for every chat model in the catalog):

```bash
git clone --branch 0.7.1 --depth 1 https://github.com/john-rocky/coreai-kit
export DEVELOPER_DIR=/Applications/Xcode-27.0.0-RC.app/Contents/Developer
open -a /Applications/Xcode-27.0.0-RC.app coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "MiniCPM5 2B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run -c release chat-cli --model minicpm5-2b --prompt "What can you do, offline?"
```

Use Xcode build **27A266a** from the release's `.xcode-pin`; adjust the app path if your installation is named differently.

💻 **Build with it** — complete; the glue is kit API, copy-paste runs:

```swift
import CoreAIKit

let id = "minicpm5-2b"
let chat: ChatSession
if id == "qwen3-0.6b" {
    // Freeze the release starter; other selections retain the live catalog's
    // model-specific dispatch (including paired Gemma bundles).
    guard let model = ModelCatalog.builtin.entry(id: id)?.modelID else {
        throw CoreAIKitError.modelNotAvailableOnPlatform(id: id)
    }
    chat = try await ChatSession(model: model)
} else {
    chat = try await ChatSession(catalog: "minicpm5-2b")
}
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device
```

The take-home is [`Examples/ChatDemo/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/0.7.1/Examples/ChatDemo/Sources/QuickStart.swift)
— this exact code as one typed function, no UI; the CLI is an argument shell over it, and
the GUI drives the same `ChatSession` across turns for its transcript.
Multi-turn? Hold the `ChatSession` and call `respond(to:)` per turn — it keeps the
conversation history; `streamResponse(to:)` yields tokens as they decode.

**Integration checklist**

- SPM: `https://github.com/john-rocky/coreai-kit` (exact **0.7.1**) → product **CoreAIKit**
- Info.plist: none needed
- Entitlements: none on Mac; iPhone needs `com.apple.developer.kernel.increased-memory-limit` (the 2.7 GB cold specialization passes the default jetsam limit)
- First run downloads the model — ~2,685 MB (Mac) / ~2,685 MB (iPhone) — then it loads from the
  local cache (Application Support; progress via the `downloadProgress` callback)
- Measure in Release — Debug is ~3× slower on per-token host work
<!-- gen-cards:use-it end -->

## Neural Engine bundle (iOS static export, AOT h18p) — 2026-09-16

`ios-ane-h18p/` is Apple's stock `coreai.llm.export --platform iOS` static export of this checkpoint at
**6-bit k-means palettization, group 8** (the shape of Apple's own iOS preset for Qwen3-1.7B;
`conversion/minicpm5_pal6_g8.yaml` in the zoo; embeddings int8; static graphs `prompt_opt`/`extend` ×
contexts {256, 512, 1024, 2048, 4096} × query {8, 16, 64}), AOT-compiled with
`xcrun coreai-build compile --platform iOS --preferred-compute neural-engine --architecture h18p`
(31/31 ANE regions, 2.5 GB). It loads through Apple's `EngineFactory` → `StaticShapeEngine` unchanged
(iPhone 17-class devices). `ios-static/` is the same export before AOT — the portable IR; compile it
for another chip yourself.

Why 6-bit: the 2026-09-15 bundle was Apple's iOS default (4-bit, group 32). It passed the three short
gate prompts but diverged from fp32 at the first token of a 109-token free-form answer (and got the
physics wrong). At 6-bit the same phone matches fp32 token-for-token on the three short prompts (24/24,
8/8 including the stop, 16/16) and for 33 tokens of the long answer, then diverges on a step where fp32
itself is nearly tied (top-2 margin 0.17); 5 of its 109 teacher-forced steps differ — transcript
`models/minicpm5-2b/gate-minicpm5-2b-ane-6bit-device.json` in the zoo. An 8-bit shape is exact in an
fp32 simulation of the recipe, but the 2B at 8 bits (2.16 GB of weights) loads on the ANE and never
returns its first token, and 4-bit group 8 does not compile for the ANE at all — zoo
`knowledge/ane-vs-gpu-iphone-2026-09.md` §9.

Speed on an iPhone 17 Pro (iOS 27.0), same day, engine released and idle until the thermal state was
back at `fair`, then 60 s of 128-token-prompt / 256-token trials: **38.5 tok/s decode** (flat over the
minute), prefill 1679 tok/s, footprint 2.6 GB. In the same run the `int8/` GPU bundle decoded 23.8 tok/s;
the previous 4-bit ANE bundle did ~55 (`models/minicpm5-2b/bench-iphone-ane-6bit-vs-gpu-2026-09-16.json`).
**The first launch on a phone builds the ANE programs: about 23 minutes** (0.2 s afterwards; the cache
lives in the app's container and is invalidated by an iOS update).

Task accuracy, GSM8K test (first 200 questions, 0-shot CoT, greedy, no-think, max 640 new tokens; same
prompt and scoring as the litertlm-convert evals) — `models/minicpm5-2b/gsm8k-200-2026-09-17.json` in the zoo:

| model | correct / 200 |
|---|---:|
| fp32 checkpoint (bf16 on a Mac) | 172 (86.0 %) |
| **`ios-ane-h18p/` 6-bit, on the iPhone 17 Pro** | **173 (86.5 %)** |
| the same 6-bit recipe applied to the fp32 weights (Mac) | 172 (86.0 %) |
| the `int8/` recipe applied to the fp32 weights (Mac) | 171 (85.5 %) |
| the replaced 4-bit bundle, on the iPhone 17 Pro | 131 (65.5 %) |

## Measured

| | decode | prefill | numerics | size |
|---|---:|---:|---|---:|
| **iPhone 17 Pro** (A19 Pro, `PipelinedBench`, Release) | **22.4 tok/s** | 27.3 tok/s | **24/24 + 24/24 token-exact** vs HF fp32 (nat + oracle, the margin-clean alphabet prompt); engine ready 28.9 s cold | **2.7 GB** |
| **M4 Max** (macOS 27, `llm-benchmark`) | **127.6 tok/s** | 2654 tok/s | **16/16 token-exact** vs the fp32 oracle (margin-aware gate, min margin 0.925) | |

Free-run check (4 prompts × 30 greedy tokens vs fp32 HF, `verify_minicpm5.py`): **3/4 exact**; the one miss is a name at fp32 probability 0.2126 vs 0.2065 (`Emma`/`Lily`, top-2 margin 0.006) — a tie any precision may flip. The fp16 control export scores 4/4, and the **per-channel** int8 sibling of this bundle scored 2/4 with a real 0.245-margin flip (`,`→` and`), which is why this repo ships per-**block-32** scales instead (same recipe, three YAML lines; see *Quantization*).

⚠️ **iPhone context cap: prompt + generated tokens must stay under 1024.** The bundle declares a 131072
dynamic KV, and the shipped `CoreAIPipelinedEngine` caps iOS growing-KV capacity at 1024 (its guard
against the iOS compiler miscompiling growing-KV specializations at seq ≥ 2048) — so a phone
conversation truncates at absolute position 1024. Chunk or trim the history on iOS; macOS has no cap.

Same recipe as the published [MiniCPM5-1B](https://huggingface.co/mlboydaisuke/MiniCPM5-1B-CoreAI)
(int8 66.8 tok/s on the same phone) with one YAML changed — **per-block-32 scales instead of
per-channel**. Measured on the Mac before picking it (`llm-benchmark`, 512p/1024g): int8
per-channel 25.6 tok/s, fp16 80.0, int8 per-block-32 **127.6**. Per-channel int8 lowers to a
slow dequant path on the Mac GPU; block-32 lands on the fast quantized-matmul path, 5× the
per-channel decode and 1.6× fp16's, at +155 MB. On the phone the two decode the same
(bandwidth-bound), so block-32 wins on both.

### JevBench public 231 (Mac, 2026-09-24)

| easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max |
|---:|---:|---:|---:|---:|---:|---:|
| 0.979 | 0.708 | 0.459 | 0.241 | 0.14 s | 1.19 s | 1.8 s |

The benchmark's own harness ([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) `2fa63fa`, v1.4.0, `typesafe` adapter) ran the 231 public items against coreai-kit `adbc755` `decide-cli serve` with the `int8/` bundle, one question per request. This chat model was asked zero-shot under the kit's JSON decision prompt, with the catalog's calibration temperature 2.93 (fit on SemIf perturbations108). Accuracy per tier and the hard tier's ECE are JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are per-request latency over all 231 requests, hard max the maximum over the hard tier. Latency was measured without an exclusive GPU window (contended), with this model's server running alone. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534 items, sealed ones included, and are not comparable to these accuracies.

**iPhone 17 Pro (2026-09-24)**

| | easy 48 | standard 72 | hard 111 |
|---|---:|---:|---:|
| accuracy | 0.979 | 0.708 | 0.459 |
| p50 | 0.22 s | 0.17 s | 0.63 s |
| p95 | 0.25 s | 0.25 s | 3.49 s |
| p50 / p95 over | 48 rows, nominal | 72 rows, nominal | 111 rows, nominal |

The phone (iOS 27.0 24A437) received the same request bodies as the Mac run, one question per request. A headless harness app answered each with coreai-kit 0.7.1, through the call the kit's System One server makes. It ran the same `int8/` bundle as the Mac run, not `ios-ane-h18p/`. Every bundle file on the phone matched the Hub revision by hash. Every answer's argmax equals the Mac run's. p50 and p95 are the kit's time per request. A nominal row started and ended with the phone on its battery at thermal state nominal.

## Quantization

Weight-only **symmetric int8, per-block-32** (a scale per 32-wide block along the input dim; no
clipping), applied as a torch pre-export pass via `coreai-opt`; SDPA / RoPE / RMSNorm stay full
precision. The 1B ships the per-channel version of the same config.

```bash
uv run coreai.llm.export openbmb/MiniCPM5-2B --experimental --compute-precision float16 \
  --compression-config minicpm5_int8sym_b32.yaml
# minicpm5_int8sym_b32.yaml: quantization_config → op_state_spec.weight = {dtype: int8,
#   qscheme: symmetric, granularity: {type: per_block, block_size: 32}}
```

## Conversion notes

- **`llama → mistral` remap.** MiniCPM5-2B's `model_type` is `llama` (a plain `LlamaForCausalLM`:
  42 layers × hidden 2048, GQA 16:2, `head_dim` 128, RoPE θ 5e6, untied 130560-vocab head); the
  stock exporter has no `llama` graph family, but Mistral's builder is architecturally identical for
  this config (GQA, no qkv bias, no qk-norm, explicit `head_dim` honored). One-line remap in the
  model registry — the same line that ships the 1B.
- **Chat EOS.** Base `eos_token` is `</s>`, but the chat template ends turns with `<|im_end|>`
  (id 130073). The bundle's tokenizer `eos_token` is set to `<|im_end|>` (as Qwen ships) so
  generation halts cleanly — checked through the engine with the chat template applied: the model
  thinks, answers, and stops (124-token reply, 131.6 tok/s short-context on the M4 Max, on the
  published bundle).
- **Dynamic-shape bundle** → the Core AI pipelined engine (the iPhone path); a static iOS export
  routes to the static-shape engine instead, which this FM-format bundle doesn't target. The 2.67 GB
  single-file bundle cold-specializes on the phone in 28.9 s (no AOT); that step needs the
  increased-memory entitlement and ~3 GB of free phone storage.
- **Thinking.** The model thinks by default (`<think>…</think>` before the answer); pass
  `enable_thinking=False` through the chat template for a direct answer. Give generation a
  generous budget (the kit caps at 4096) — the think trace alone can run several hundred tokens.

## Run

```swift
import FoundationModels
import CoreAILanguageModels
let model = try await CoreAILanguageModel(resourcesAt: int8BundleURL)   // …/int8
let session = LanguageModelSession(model: model)
print(try await session.respond(to: "Explain on-device AI in one sentence."))
```

Or in the zoo's **CoreAIChat** app / the kit's **ChatDemo** (Model → "MiniCPM5 2B").

## Reproduce

Exporter, gate, card and port notes live in the
[Core AI model zoo](https://github.com/john-rocky/coreai-model-zoo):
[`models/minicpm5-2b/`](https://github.com/john-rocky/coreai-model-zoo/tree/main/models/minicpm5-2b),
[`conversion/export_minicpm5.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/export_minicpm5.py),
[`knowledge/minicpm5-1b.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/minicpm5-1b.md).

```bash
python3 conversion/zoo_convert.py show minicpm5-2b
python3 conversion/zoo_convert.py run  minicpm5-2b
```

## Credits

Model: **MiniCPM5-2B** by **OpenBMB** ([openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B), Apache-2.0).
Core AI conversion: the Core AI model zoo.