JevBench public 231: iPhone 17 Pro rows (2026-09-24)
Browse files
README.md
CHANGED
|
@@ -130,6 +130,24 @@ graph's prefill is S=1, and at that kit commit the state was prefilled again for
|
|
| 130 |
question. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534
|
| 131 |
items, sealed ones included, and are not comparable to these accuracies.
|
| 132 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
## Through the kit
|
| 134 |
|
| 135 |
**Measured through coreai-kit**, using its sequential engine and tokenizer: `decide-cli parity`
|
|
|
|
| 130 |
question. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534
|
| 131 |
items, sealed ones included, and are not comparable to these accuracies.
|
| 132 |
|
| 133 |
+
**iPhone 17 Pro (2026-09-24)**
|
| 134 |
+
|
| 135 |
+
| | easy 48 | standard 72 | hard 20 |
|
| 136 |
+
|---|---:|---:|---:|
|
| 137 |
+
| accuracy | 1.000 | 0.819 | 0.200 |
|
| 138 |
+
| p50 | 0.88 s | 0.99 s | 5.77 s |
|
| 139 |
+
| p95 | 1.29 s | 1.28 s | 7.28 s |
|
| 140 |
+
| p50 / p95 over | 48 rows, hot | 24 of 72 rows, nominal | 20 rows, nominal |
|
| 141 |
+
|
| 142 |
+
The phone (iOS 27.0 24A437) received the same request bodies as the Mac run, one question
|
| 143 |
+
per request. A headless harness app answered each with coreai-kit 0.7.1, through the call the
|
| 144 |
+
kit's System One server makes. Every bundle file on the phone matched the Hub revision by
|
| 145 |
+
hash. Every answer's argmax equals the Mac run's. The hard column is the middle 20 of the 111
|
| 146 |
+
hard items by state length. The Mac run scored 0.200 on the same 20. p50 and p95 are the kit's
|
| 147 |
+
time per request: the state's prefill plus the decision. A nominal row started and ended with
|
| 148 |
+
the phone on its battery at thermal state nominal. A hot row started or ended at fair or worse.
|
| 149 |
+
The easy tier and the first 48 standard rows ran on the charger.
|
| 150 |
+
|
| 151 |
## Through the kit
|
| 152 |
|
| 153 |
**Measured through coreai-kit**, using its sequential engine and tokenizer: `decide-cli parity`
|