mlboydaisuke commited on
Commit
825e5d7
·
verified ·
1 Parent(s): ff1e102

JevBench public 231: iPhone 17 Pro rows (2026-09-24)

Browse files
Files changed (1) hide show
  1. README.md +18 -0
README.md CHANGED
@@ -130,6 +130,24 @@ graph's prefill is S=1, and at that kit commit the state was prefilled again for
130
  question. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534
131
  items, sealed ones included, and are not comparable to these accuracies.
132
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
133
  ## Through the kit
134
 
135
  **Measured through coreai-kit**, using its sequential engine and tokenizer: `decide-cli parity`
 
130
  question. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534
131
  items, sealed ones included, and are not comparable to these accuracies.
132
 
133
+ **iPhone 17 Pro (2026-09-24)**
134
+
135
+ | | easy 48 | standard 72 | hard 20 |
136
+ |---|---:|---:|---:|
137
+ | accuracy | 1.000 | 0.819 | 0.200 |
138
+ | p50 | 0.88 s | 0.99 s | 5.77 s |
139
+ | p95 | 1.29 s | 1.28 s | 7.28 s |
140
+ | p50 / p95 over | 48 rows, hot | 24 of 72 rows, nominal | 20 rows, nominal |
141
+
142
+ The phone (iOS 27.0 24A437) received the same request bodies as the Mac run, one question
143
+ per request. A headless harness app answered each with coreai-kit 0.7.1, through the call the
144
+ kit's System One server makes. Every bundle file on the phone matched the Hub revision by
145
+ hash. Every answer's argmax equals the Mac run's. The hard column is the middle 20 of the 111
146
+ hard items by state length. The Mac run scored 0.200 on the same 20. p50 and p95 are the kit's
147
+ time per request: the state's prefill plus the decision. A nominal row started and ended with
148
+ the phone on its battery at thermal state nominal. A hot row started or ended at fair or worse.
149
+ The easy tier and the first 48 standard rows ran on the charger.
150
+
151
  ## Through the kit
152
 
153
  **Measured through coreai-kit**, using its sequential engine and tokenizer: `decide-cli parity`