mlboydaisuke commited on
Commit
ff1e102
·
verified ·
1 Parent(s): 9f10bc2

JevBench public 231 accuracy and latency (Mac, 2026-09-24)

Browse files

Same subsection as coreai-model-zoo a708c2e (models/openthai-systemone/README.md).

Files changed (1) hide show
  1. README.md +17 -0
README.md CHANGED
@@ -113,6 +113,23 @@ No other Core AI, Python or Swift engine job appeared in the before/after proces
113
  benchmark needed a local CLI option to select `EngineOptions.variant`; the trial loop was
114
  unchanged. [Trials, load times and environment](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/openthai-systemone/llm-benchmark.json).
115
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
  ## Through the kit
117
 
118
  **Measured through coreai-kit**, using its sequential engine and tokenizer: `decide-cli parity`
 
113
  benchmark needed a local CLI option to select `EngineOptions.variant`; the trial loop was
114
  unchanged. [Trials, load times and environment](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/openthai-systemone/llm-benchmark.json).
115
 
116
+ ### JevBench public 231 (Mac, 2026-09-24)
117
+
118
+ | easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max |
119
+ |---:|---:|---:|---:|---:|---:|---:|
120
+ | 1.000 | 0.819 | 0.324 | 0.449 | 0.62 s | 25.93 s | 39.0 s |
121
+
122
+ The benchmark's own harness
123
+ ([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) `2fa63fa`, v1.4.0,
124
+ `typesafe` adapter) ran the 231 public items against coreai-kit `adbc755` `decide-cli serve`
125
+ with the ship bundle, one question per request. Accuracy per tier and the hard tier's ECE are
126
+ JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are per-request
127
+ latency over all 231 requests, hard max the maximum over the hard tier. Latency was measured
128
+ without an exclusive GPU window (contended), with this model's server running alone; the
129
+ graph's prefill is S=1, and at that kit commit the state was prefilled again for every
130
+ question. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534
131
+ items, sealed ones included, and are not comparable to these accuracies.
132
+
133
  ## Through the kit
134
 
135
  **Measured through coreai-kit**, using its sequential engine and tokenizer: `decide-cli parity`