mlboydaisuke commited on
Commit
b418cf7
·
verified ·
1 Parent(s): b0960e6

JevBench public 231: iPhone 17 Pro rows (2026-09-24)

Browse files
Files changed (1) hide show
  1. README.md +11 -0
README.md CHANGED
@@ -168,6 +168,17 @@ per-channel decode and 1.6× fp16's, at +155 MB. On the phone the two decode the
168
 
169
  The benchmark's own harness ([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) `2fa63fa`, v1.4.0, `typesafe` adapter) ran the 231 public items against coreai-kit `adbc755` `decide-cli serve` with the `int8/` bundle, one question per request. This chat model was asked zero-shot under the kit's JSON decision prompt, with the catalog's calibration temperature 2.93 (fit on SemIf perturbations108). Accuracy per tier and the hard tier's ECE are JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are per-request latency over all 231 requests, hard max the maximum over the hard tier. Latency was measured without an exclusive GPU window (contended), with this model's server running alone. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534 items, sealed ones included, and are not comparable to these accuracies.
170
 
 
 
 
 
 
 
 
 
 
 
 
171
  ## Quantization
172
 
173
  Weight-only **symmetric int8, per-block-32** (a scale per 32-wide block along the input dim; no
 
168
 
169
  The benchmark's own harness ([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) `2fa63fa`, v1.4.0, `typesafe` adapter) ran the 231 public items against coreai-kit `adbc755` `decide-cli serve` with the `int8/` bundle, one question per request. This chat model was asked zero-shot under the kit's JSON decision prompt, with the catalog's calibration temperature 2.93 (fit on SemIf perturbations108). Accuracy per tier and the hard tier's ECE are JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are per-request latency over all 231 requests, hard max the maximum over the hard tier. Latency was measured without an exclusive GPU window (contended), with this model's server running alone. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534 items, sealed ones included, and are not comparable to these accuracies.
170
 
171
+ **iPhone 17 Pro (2026-09-24)**
172
+
173
+ | | easy 48 | standard 72 | hard 111 |
174
+ |---|---:|---:|---:|
175
+ | accuracy | 0.979 | 0.708 | 0.459 |
176
+ | p50 | 0.22 s | 0.17 s | 0.63 s |
177
+ | p95 | 0.25 s | 0.25 s | 3.49 s |
178
+ | p50 / p95 over | 48 rows, nominal | 72 rows, nominal | 111 rows, nominal |
179
+
180
+ The phone (iOS 27.0 24A437) received the same request bodies as the Mac run, one question per request. A headless harness app answered each with coreai-kit 0.7.1, through the call the kit's System One server makes. It ran the same `int8/` bundle as the Mac run, not `ios-ane-h18p/`. Every bundle file on the phone matched the Hub revision by hash. Every answer's argmax equals the Mac run's. p50 and p95 are the kit's time per request. A nominal row started and ended with the phone on its battery at thermal state nominal.
181
+
182
  ## Quantization
183
 
184
  Weight-only **symmetric int8, per-block-32** (a scale per 32-wide block along the input dim; no