mlboydaisuke/MiniCPM-V-4.6-CoreAI

MiniCPM-V-4.6 — Core AI

On-device vision-language model for iPhone / Apple Silicon. A Core AI port of openbmb/MiniCPM-V-4.6 — the strongest sub-2B open VLM — running fully local on the GPU via the Core AI pipelined engine: pick a photo, ask about it, stream the answer.

Verified on iPhone 17 Pro: image → grounded answer at ~51.5 tok/s decode, all local.

Fridge photo → recipe ideas, fully on-device on an iPhone 17 Pro (CoreAIChat).

Architecture

MiniCPM-V-4.6 (1.3B) = a SigLIP So400m vision tower (980px / patch 14 / 27 layers, with a window-attention insert-merger @ layer 6 + a downsample-MLP merger → ÷16 = 64 visual tokens per 448px slice) + a Qwen3.5-hybrid text backbone (qwen3_5_text: 0.8B, 24 layers, GatedDeltaNet linear attention ×3 : full attention ×1, head_dim 256, vocab 248094, tied head). Connector = 2×2 spatial merges + MLP, spliced into the text embeddings at <image> positions (masked_scatter).

Bundles

Recommended (optimized, 2026-06-25):

path	what	dtype	size
gpu-pipelined/minicpmv46_vlm_decode_int8hu/	VLM text decoder (input_ids → logits + static image_embeds[64,1024]; in-graph gather ids ≥ V ? image_embeds[ids-V] : embed[ids])	int8 body + untied int8 head	~1.2 GB
gpu-pipelined/minicpmv46_vision_int8lin/	fixed-grid SigLIP vision encoder (pixel_values[1,3,448,448] → image_features[64,1024])	int8	~0.6 GB

The int8 head quantizes the big-vocab LM head (fp16 in int8lin = ~half the per-token read) → +48% decode on iPhone 17 Pro (46→68 tok/s). The int8 vision halves the encoder's size (the encode is compute-bound, so this is a size/memory win); pair it with a one-shot vision-graph warmup at load to hide the ~2.7 s first-photo cold compile. Original …_int8lin decoder + fp16 minicpmv46_vision remain for compatibility.

The decoder is a complete qwen3.5-hybrid text LLM when image_embeds is zero — same bundle, no image needed.

How a VLM rides the text-only engine

The pipelined engine knows nothing about images. The whole multimodal state rides the static-input hook (image_embeds buffer) + an id-space trick — the graph stays ids + positions → logits:

• The host runs the vision encoder once per image (resize 448, normalize x/127.5−1) and writes image_embeds [64,1024] into one owned MTLBuffer the engine binds on every step.
• The prompt's <|image_pad|> ids are rewritten to extension ids V + slot (slot 0..63). In-graph: embed = ids < V ? table[ids] : image_embeds[ids − V].
• Positions are plain 1D (no M-RoPE / no rope-shift), the qwen3.5-hybrid KV + conv + recurrent states are the engine's; nothing else changes.

Simpler than the Qwen3-VL port (no deepstack, no M-RoPE).

Measured (iPhone 17 Pro, iOS 27 beta, release)

• iPhone 17 Pro decode (int8 head): VLM in-app A/B, same conditions, back-to-back — int8lin 51.5 → int8hu 70.0 tok/s = +36% (the VLM bundle binds image_embeds every step, which dilutes the head gain; the text core alone is 46 → 68 = +48%). ~64–70 tok/s in practice by device temperature. · M4 Max text core ~224 tok/s (llm-benchmark), engine cold-spec ~2–4 s, ~1.5 GB resident (jetsam-safe).
• Vision: int8 encoder ≈ fp16 encode time (compute-bound) at 0.6 GB (half); the ~2.7 s first-image latency is the SigLIP graph's cold compile — run a dummy encode at load to make the user's first photo warm (tens of ms).
• Numerics: fp32-torch parity bit-exact (vision cos 1.000000, full overlay logits cos 1.00004); Core AI engine ≡ python ≡ HF (text 24/24; image path reproduces the HF description modulo one int8 near-tie token, then reconverges).
• Real-photo example (kakigōri): "a bowl of shaved ice ... chunks of mango ... a dark blue saucer ... a menu or a book, hinting at a café ... a wooden table" — accurate, fully on-device.

Use it

apps/CoreAIChat and the standalone MiniCPMVLM app have a MiniCPM-V 4.6 mode with a photo picker: pick an image, ask, stream. The vision tower runs once per image (~hundreds of ms); each turn re-prefills (S=1).

Conversion + gates: see coreai-model-zoo / minicpm-v-4.6.

License: Apache-2.0 (inherited from openbmb/MiniCPM-V-4.6).