Leading the System One Mosaic Benchmark: What Darwin-27B-ZTC-v2's #1 Means
β’ 6
AI & ML interests
Contact: arxivgpt@gmail.com
Recent Activity
View all activity
Papers
View all Papers Articles
SeaWolf-AIΒ
updated a
dataset about 2 hours ago
SeaWolf-AIΒ
published an article about 5 hours ago
Article
Leading the System One Mosaic Benchmark: What Darwin-27B-ZTC-v2's #1 Means
FINAL-Bench
β’ β’ 6 SeaWolf-AIΒ
posted an update about 8 hours ago
Post
1087
π Darwin-27B-ZTC-v2 just took #1 on the System One Mosaic Benchmark (S1MB).
S1MB compares 102 models across 137 specialized benchmarks, in three task types: Noul (assess a condition), Choice (select an option), Score (rate on a scale). Ranking is by overall Borda score.
π Top of the board
π₯ Darwin ZTC v2 (FINAL-Bench) 89.58
π₯ OpenJev-27B 87.50
π₯ AutoJev-27B 87.07
4οΈβ£ Eikos 27B 85.43
5οΈβ£ Jev 1.13 85.05
π Ranks 2 to 5 are all the JEV family (TypeSafe AI's System One model, from ex-OpenAI researchers). S1MB exists to compare these System One judges, so leading it is the headline.
βοΈ Why a zero-token judge wins here
πΉ It does not generate. It reads the input and typed questions and returns a calibrated distribution in a single forward pass.
πΉ Zero generated tokens, no decoding loop, so latency and cost stay low.
πΉ Holds up out of distribution too: General Noul 96.00, General Choice 99.34.
It is also #1 on the typed-decisions leaderboard (0.743, zero-shot). Same message from both: a deterministic, calibrated judge at one forward pass per call.
π Model: FINAL-Bench/Darwin-27B-ZTC-v2
π Leaderboard: hotchpotch/S1MB-leaderboard
Standings move as new models are added. Numbers reflect the board at the time of writing. π
S1MB compares 102 models across 137 specialized benchmarks, in three task types: Noul (assess a condition), Choice (select an option), Score (rate on a scale). Ranking is by overall Borda score.
π Top of the board
π₯ Darwin ZTC v2 (FINAL-Bench) 89.58
π₯ OpenJev-27B 87.50
π₯ AutoJev-27B 87.07
4οΈβ£ Eikos 27B 85.43
5οΈβ£ Jev 1.13 85.05
π Ranks 2 to 5 are all the JEV family (TypeSafe AI's System One model, from ex-OpenAI researchers). S1MB exists to compare these System One judges, so leading it is the headline.
βοΈ Why a zero-token judge wins here
πΉ It does not generate. It reads the input and typed questions and returns a calibrated distribution in a single forward pass.
πΉ Zero generated tokens, no decoding loop, so latency and cost stay low.
πΉ Holds up out of distribution too: General Noul 96.00, General Choice 99.34.
It is also #1 on the typed-decisions leaderboard (0.743, zero-shot). Same message from both: a deterministic, calibrated judge at one forward pass per call.
π Model: FINAL-Bench/Darwin-27B-ZTC-v2
π Leaderboard: hotchpotch/S1MB-leaderboard
Standings move as new models are added. Numbers reflect the board at the time of writing. π
SeaWolf-AIΒ
updated 2
collections about 11 hours ago
SeaWolf-AIΒ
updated a
model about 11 hours ago
SeaWolf-AIΒ
published a
model about 23 hours ago
SeaWolf-AIΒ
updated a
dataset 1 day ago
SeaWolf-AIΒ
published a
dataset 1 day ago
SeaWolf-AIΒ
updated a
collection 1 day ago
SeaWolf-AIΒ
published an article 1 day ago
Article
Darwin-27B-ZTC: A Single-Pass Judge and a Quantitative Look at Its Calibration
FINAL-Bench
β’ β’ 12