JevBench public 231 accuracy and latency (Mac, 2026-09-24)
Browse filesSame subsection as coreai-model-zoo a708c2e (models/openthai-systemone/README.md).
README.md
CHANGED
|
@@ -113,6 +113,23 @@ No other Core AI, Python or Swift engine job appeared in the before/after proces
|
|
| 113 |
benchmark needed a local CLI option to select `EngineOptions.variant`; the trial loop was
|
| 114 |
unchanged. [Trials, load times and environment](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/openthai-systemone/llm-benchmark.json).
|
| 115 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
## Through the kit
|
| 117 |
|
| 118 |
**Measured through coreai-kit**, using its sequential engine and tokenizer: `decide-cli parity`
|
|
|
|
| 113 |
benchmark needed a local CLI option to select `EngineOptions.variant`; the trial loop was
|
| 114 |
unchanged. [Trials, load times and environment](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/openthai-systemone/llm-benchmark.json).
|
| 115 |
|
| 116 |
+
### JevBench public 231 (Mac, 2026-09-24)
|
| 117 |
+
|
| 118 |
+
| easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max |
|
| 119 |
+
|---:|---:|---:|---:|---:|---:|---:|
|
| 120 |
+
| 1.000 | 0.819 | 0.324 | 0.449 | 0.62 s | 25.93 s | 39.0 s |
|
| 121 |
+
|
| 122 |
+
The benchmark's own harness
|
| 123 |
+
([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) `2fa63fa`, v1.4.0,
|
| 124 |
+
`typesafe` adapter) ran the 231 public items against coreai-kit `adbc755` `decide-cli serve`
|
| 125 |
+
with the ship bundle, one question per request. Accuracy per tier and the hard tier's ECE are
|
| 126 |
+
JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are per-request
|
| 127 |
+
latency over all 231 requests, hard max the maximum over the hard tier. Latency was measured
|
| 128 |
+
without an exclusive GPU window (contended), with this model's server running alone; the
|
| 129 |
+
graph's prefill is S=1, and at that kit commit the state was prefilled again for every
|
| 130 |
+
question. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534
|
| 131 |
+
items, sealed ones included, and are not comparable to these accuracies.
|
| 132 |
+
|
| 133 |
## Through the kit
|
| 134 |
|
| 135 |
**Measured through coreai-kit**, using its sequential engine and tokenizer: `decide-cli parity`
|