Pivot performance
All figures here refer to checkpoint 14bf8c26bf344ebdf88e22a4b6152dc5f75f3578, public JevBench v1.4.1 commit 24b9b5c1609a7a9e8fa14f49e5985a836c9dc842, FP32 and the same frozen 512/128-token input contract.
Accuracy on public tasks
| Tier | Correct | Tasks | Accuracy | Top-label ECE, 10 bins |
|---|---|---|---|---|
| Original | 27 | 72 | 37.50% | 0.5247 |
| Easy | 39 | 48 | 81.25% | 0.1139 |
| Hard | 41 | 111 | 36.94% | 0.3792 |
| Total | 107 | 231 | 46.32% | — |
The official JevBench composite score is unavailable because the sealed and judge tasks and official cost input were not measured.
Local speed
| Warm local FP32 measure | H200 GPU | Xeon CPU, 4 threads |
|---|---|---|
| Single decision p50, 32 measured | 15.7668 ms | 797.5558 ms |
| Single decision p95, 32 measured | 19.8708 ms | 1089.0266 ms |
| Single decision mean | 16.1614 ms | 770.7159 ms |
| Batch size for throughput | 32 | 4 |
| Throughput, median of 3 × 64 decisions | 545.2833 decisions/s | 3.7708 decisions/s |
The CPU p50 single-decision time is 50.6× the H200 p50 for these two machines. CPU and GPU batch throughput used different batch sizes and should not be read as a same-batch comparison. The timings cover tokenizer + inference + scoring with 5 warmup singles and 2 warmup bulk passes. Reproduce them on your own hardware using the CPU script or full public runner.
Source: full structured summary, original public benchmark result, and original CPU measurement.