Hugging Face's logo

mlboydaisuke
/
MiniCPM5-2B-CoreAI

Text Generation
Core ML
coreai
coreai-aimodel
core-ai
apple
on-device
iphone
metal
system-one
typed-decisions
Model card Files
xet
 Community
MiniCPM5-2B-CoreAI / README.md
mlboydaisuke's picture
mlboydaisuke
JevBench public 231: iPhone 17 Pro rows (2026-09-24)
b418cf7 verified
preview code
|
Raw
History Blame Contribute Delete
14.9 kB
metadata 
license: apache-2.0
base_model: openbmb/MiniCPM5-2B
pipeline_tag: text-generation
library_name: coreai
tags:
  - coreai
  - core-ai
  - coreml
  - apple
  - on-device
  - iphone
  - metal
  - system-one
  - typed-decisions
base_model_relation: quantized

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

This model has no row on DeviceMark, the on-device LLM leaderboard.

MiniCPM5-2B β€” Core AI (int8 block-32, runs on iPhone)

Apple Core AI (.aimodel) conversion of openbmb/MiniCPM5-2B β€” OpenBMB's 2.5B on-device LLM (released 2026-09-06, the 1B's 42-layer sibling) with hybrid Think / No-Think reasoning, native tool calling and 128K context; OpenBMB reports it as 2B-class open-source SOTA (LiveCodeBench v6 69.1, AIME 2026 86.5, BFCL v4 66.6, SWE-bench Verified 46.4 on their card). Runs fully on-device on iPhone and Apple Silicon Macs (GPU, pipelined engine).

Part of the community Core AI model zoo: https://github.com/john-rocky/coreai-model-zoo

Use it

New to Core AI? Start with CoreAIKit 0.7.1. Follow its requirements and first-run steps for qwen3-0.6b, then open the same release's ChatDemo. The README records the tested OS/SDK and download size; model and device coverage is stated per example.

⚑ One line β€” run the kit's task op on this model (import CoreAIOps; no session, no model plumbing, downloads on first use):

let tldr = try await CoreAI.summarize(text, options: .model("minicpm5-2b"))

Every op, one shape β€” Cookbook.

▢️ Run it (source) β€” the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone --branch 0.7.1 --depth 1 https://github.com/john-rocky/coreai-kit
export DEVELOPER_DIR=/Applications/Xcode-27.0.0-RC.app/Contents/Developer
open -a /Applications/Xcode-27.0.0-RC.app coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# β†’ Run, then pick "MiniCPM5 2B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run -c release chat-cli --model minicpm5-2b --prompt "What can you do, offline?"

Use Xcode build 27A266a from the release's .xcode-pin; adjust the app path if your installation is named differently.

πŸ’» Build with it β€” complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let id = "minicpm5-2b"
let chat: ChatSession
if id == "qwen3-0.6b" {
    // Freeze the release starter; other selections retain the live catalog's
    // model-specific dispatch (including paired Gemma bundles).
    guard let model = ModelCatalog.builtin.entry(id: id)?.modelID else {
        throw CoreAIKitError.modelNotAvailableOnPlatform(id: id)
    }
    chat = try await ChatSession(model: model)
} else {
    chat = try await ChatSession(catalog: "minicpm5-2b")
}
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

The take-home is Examples/ChatDemo/Sources/QuickStart.swift β€” this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn β€” it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

  • SPM: https://github.com/john-rocky/coreai-kit (exact 0.7.1) β†’ product CoreAIKit
  • Info.plist: none needed
  • Entitlements: none on Mac; iPhone needs com.apple.developer.kernel.increased-memory-limit (the 2.7 GB cold specialization passes the default jetsam limit)
  • First run downloads the model β€” ~2,685 MB (Mac) / ~2,685 MB (iPhone) β€” then it loads from the local cache (Application Support; progress via the downloadProgress callback)
  • Measure in Release β€” Debug is ~3Γ— slower on per-token host work

Neural Engine bundle (iOS static export, AOT h18p) β€” 2026-09-16

ios-ane-h18p/ is Apple's stock coreai.llm.export --platform iOS static export of this checkpoint at 6-bit k-means palettization, group 8 (the shape of Apple's own iOS preset for Qwen3-1.7B; conversion/minicpm5_pal6_g8.yaml in the zoo; embeddings int8; static graphs prompt_opt/extend Γ— contexts {256, 512, 1024, 2048, 4096} Γ— query {8, 16, 64}), AOT-compiled with xcrun coreai-build compile --platform iOS --preferred-compute neural-engine --architecture h18p (31/31 ANE regions, 2.5 GB). It loads through Apple's EngineFactory β†’ StaticShapeEngine unchanged (iPhone 17-class devices). ios-static/ is the same export before AOT β€” the portable IR; compile it for another chip yourself.

Why 6-bit: the 2026-09-15 bundle was Apple's iOS default (4-bit, group 32). It passed the three short gate prompts but diverged from fp32 at the first token of a 109-token free-form answer (and got the physics wrong). At 6-bit the same phone matches fp32 token-for-token on the three short prompts (24/24, 8/8 including the stop, 16/16) and for 33 tokens of the long answer, then diverges on a step where fp32 itself is nearly tied (top-2 margin 0.17); 5 of its 109 teacher-forced steps differ β€” transcript models/minicpm5-2b/gate-minicpm5-2b-ane-6bit-device.json in the zoo. An 8-bit shape is exact in an fp32 simulation of the recipe, but the 2B at 8 bits (2.16 GB of weights) loads on the ANE and never returns its first token, and 4-bit group 8 does not compile for the ANE at all β€” zoo knowledge/ane-vs-gpu-iphone-2026-09.md Β§9.

Speed on an iPhone 17 Pro (iOS 27.0), same day, engine released and idle until the thermal state was back at fair, then 60 s of 128-token-prompt / 256-token trials: 38.5 tok/s decode (flat over the minute), prefill 1679 tok/s, footprint 2.6 GB. In the same run the int8/ GPU bundle decoded 23.8 tok/s; the previous 4-bit ANE bundle did ~55 (models/minicpm5-2b/bench-iphone-ane-6bit-vs-gpu-2026-09-16.json). The first launch on a phone builds the ANE programs: about 23 minutes (0.2 s afterwards; the cache lives in the app's container and is invalidated by an iOS update).

Task accuracy, GSM8K test (first 200 questions, 0-shot CoT, greedy, no-think, max 640 new tokens; same prompt and scoring as the litertlm-convert evals) β€” models/minicpm5-2b/gsm8k-200-2026-09-17.json in the zoo:

model correct / 200
fp32 checkpoint (bf16 on a Mac) 172 (86.0 %)
ios-ane-h18p/ 6-bit, on the iPhone 17 Pro 173 (86.5 %)
the same 6-bit recipe applied to the fp32 weights (Mac) 172 (86.0 %)
the int8/ recipe applied to the fp32 weights (Mac) 171 (85.5 %)
the replaced 4-bit bundle, on the iPhone 17 Pro 131 (65.5 %)

Measured

decode prefill numerics size
iPhone 17 Pro (A19 Pro, PipelinedBench, Release) 22.4 tok/s 27.3 tok/s 24/24 + 24/24 token-exact vs HF fp32 (nat + oracle, the margin-clean alphabet prompt); engine ready 28.9 s cold 2.7 GB
M4 Max (macOS 27, llm-benchmark) 127.6 tok/s 2654 tok/s 16/16 token-exact vs the fp32 oracle (margin-aware gate, min margin 0.925)

Free-run check (4 prompts Γ— 30 greedy tokens vs fp32 HF, verify_minicpm5.py): 3/4 exact; the one miss is a name at fp32 probability 0.2126 vs 0.2065 (Emma/Lily, top-2 margin 0.006) β€” a tie any precision may flip. The fp16 control export scores 4/4, and the per-channel int8 sibling of this bundle scored 2/4 with a real 0.245-margin flip (,β†’ and), which is why this repo ships per-block-32 scales instead (same recipe, three YAML lines; see Quantization).

⚠️ iPhone context cap: prompt + generated tokens must stay under 1024. The bundle declares a 131072 dynamic KV, and the shipped CoreAIPipelinedEngine caps iOS growing-KV capacity at 1024 (its guard against the iOS compiler miscompiling growing-KV specializations at seq β‰₯ 2048) β€” so a phone conversation truncates at absolute position 1024. Chunk or trim the history on iOS; macOS has no cap.

Same recipe as the published MiniCPM5-1B (int8 66.8 tok/s on the same phone) with one YAML changed β€” per-block-32 scales instead of per-channel. Measured on the Mac before picking it (llm-benchmark, 512p/1024g): int8 per-channel 25.6 tok/s, fp16 80.0, int8 per-block-32 127.6. Per-channel int8 lowers to a slow dequant path on the Mac GPU; block-32 lands on the fast quantized-matmul path, 5Γ— the per-channel decode and 1.6Γ— fp16's, at +155 MB. On the phone the two decode the same (bandwidth-bound), so block-32 wins on both.

JevBench public 231 (Mac, 2026-09-24)

easy 48 standard 72 hard 111 ECE hard p50 p95 hard max
0.979 0.708 0.459 0.241 0.14 s 1.19 s 1.8 s

The benchmark's own harness (fstandhartinger/jevbench 2fa63fa, v1.4.0, typesafe adapter) ran the 231 public items against coreai-kit adbc755 decide-cli serve with the int8/ bundle, one question per request. This chat model was asked zero-shot under the kit's JSON decision prompt, with the catalog's calibration temperature 2.93 (fit on SemIf perturbations108). Accuracy per tier and the hard tier's ECE are JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are per-request latency over all 231 requests, hard max the maximum over the hard tier. Latency was measured without an exclusive GPU window (contended), with this model's server running alone. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534 items, sealed ones included, and are not comparable to these accuracies.

iPhone 17 Pro (2026-09-24)

easy 48 standard 72 hard 111
accuracy 0.979 0.708 0.459
p50 0.22 s 0.17 s 0.63 s
p95 0.25 s 0.25 s 3.49 s
p50 / p95 over 48 rows, nominal 72 rows, nominal 111 rows, nominal

The phone (iOS 27.0 24A437) received the same request bodies as the Mac run, one question per request. A headless harness app answered each with coreai-kit 0.7.1, through the call the kit's System One server makes. It ran the same int8/ bundle as the Mac run, not ios-ane-h18p/. Every bundle file on the phone matched the Hub revision by hash. Every answer's argmax equals the Mac run's. p50 and p95 are the kit's time per request. A nominal row started and ended with the phone on its battery at thermal state nominal.

Quantization

Weight-only symmetric int8, per-block-32 (a scale per 32-wide block along the input dim; no clipping), applied as a torch pre-export pass via coreai-opt; SDPA / RoPE / RMSNorm stay full precision. The 1B ships the per-channel version of the same config.

uv run coreai.llm.export openbmb/MiniCPM5-2B --experimental --compute-precision float16 \
  --compression-config minicpm5_int8sym_b32.yaml
# minicpm5_int8sym_b32.yaml: quantization_config β†’ op_state_spec.weight = {dtype: int8,
#   qscheme: symmetric, granularity: {type: per_block, block_size: 32}}

Conversion notes

  • llama β†’ mistral remap. MiniCPM5-2B's model_type is llama (a plain LlamaForCausalLM: 42 layers Γ— hidden 2048, GQA 16:2, head_dim 128, RoPE ΞΈ 5e6, untied 130560-vocab head); the stock exporter has no llama graph family, but Mistral's builder is architecturally identical for this config (GQA, no qkv bias, no qk-norm, explicit head_dim honored). One-line remap in the model registry β€” the same line that ships the 1B.
  • Chat EOS. Base eos_token is </s>, but the chat template ends turns with <|im_end|> (id 130073). The bundle's tokenizer eos_token is set to <|im_end|> (as Qwen ships) so generation halts cleanly β€” checked through the engine with the chat template applied: the model thinks, answers, and stops (124-token reply, 131.6 tok/s short-context on the M4 Max, on the published bundle).
  • Dynamic-shape bundle β†’ the Core AI pipelined engine (the iPhone path); a static iOS export routes to the static-shape engine instead, which this FM-format bundle doesn't target. The 2.67 GB single-file bundle cold-specializes on the phone in 28.9 s (no AOT); that step needs the increased-memory entitlement and ~3 GB of free phone storage.
  • Thinking. The model thinks by default (<think>…</think> before the answer); pass enable_thinking=False through the chat template for a direct answer. Give generation a generous budget (the kit caps at 4096) β€” the think trace alone can run several hundred tokens.

Run 

import FoundationModels
import CoreAILanguageModels
let model = try await CoreAILanguageModel(resourcesAt: int8BundleURL)   // …/int8
let session = LanguageModelSession(model: model)
print(try await session.respond(to: "Explain on-device AI in one sentence."))

Or in the zoo's CoreAIChat app / the kit's ChatDemo (Model β†’ "MiniCPM5 2B").

Reproduce

Exporter, gate, card and port notes live in the Core AI model zoo: models/minicpm5-2b/, conversion/export_minicpm5.py, knowledge/minicpm5-1b.md.

python3 conversion/zoo_convert.py show minicpm5-2b
python3 conversion/zoo_convert.py run  minicpm5-2b

Credits

Model: MiniCPM5-2B by OpenBMB (openbmb/MiniCPM5-2B, Apache-2.0). Core AI conversion: the Core AI model zoo.