mlboydaisuke commited on
Commit
2dd14f7
·
verified ·
1 Parent(s): 0f2d7d7

card: Neural Engine bundle section

Browse files
Files changed (1) hide show
  1. README.md +15 -0
README.md CHANGED
@@ -81,6 +81,21 @@ conversation history; `streamResponse(to:)` yields tokens as they decode.
81
  - Measure in Release — Debug is ~3× slower on per-token host work
82
  <!-- gen-cards:use-it end -->
83
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
84
  ## Measured
85
 
86
  | | decode | prefill | numerics | size |
 
81
  - Measure in Release — Debug is ~3× slower on per-token host work
82
  <!-- gen-cards:use-it end -->
83
 
84
+ ## Neural Engine bundle (iOS static export, AOT h18p) — 2026-09-15
85
+
86
+ `ios-ane-h18p/` is Apple's stock `coreai.llm.export --platform iOS` static export of this checkpoint
87
+ (k-means **4-bit palettized, group 32** (Apple's iOS default preset); embeddings int8; static graphs `prompt_opt`/`extend` ×
88
+ contexts {256, 512, 1024, 2048, 4096} × query {8, 16, 64}), AOT-compiled with
89
+ `xcrun coreai-build compile --platform iOS --preferred-compute neural-engine --architecture h18p`
90
+ (31/31 ANE regions, 1.4 GB). It loads through Apple's `EngineFactory` → `StaticShapeEngine` unchanged
91
+ (iPhone 17-class devices). `ios-static/` is the same export before AOT — the portable IR; compile it
92
+ for another chip yourself.
93
+
94
+ Gated on an iPhone 17 Pro (iOS 27.0) against the fp32 HF oracle: teacher-forced single-step sweep 24/24 + 8/8 + 16/16 and free-run greedy token-exact including the stop — transcript in the zoo,
95
+ `models/minicpm5-2b/gate-minicpm5-2b-ane-device.json`. Speed on the same phone (Apple llm-benchmark
96
+ method, 512-token prompt, 1024 generated, 5 trials, two back-to-back runs): decode **48.0 / 38.2 tok/s**, prefill 1858 / 1494 tok/s; footprint 2.0 GB. The Neural Engine is the power-efficient lane; these are
97
+ not a GPU comparison (different protocol from the `int8/` row above).
98
+
99
  ## Measured
100
 
101
  | | decode | prefill | numerics | size |