Hugging Face's logo

FluidInference
/
verdict-coreml

Text Classification
Core ML
GLiClass
apple-silicon
decision-model
verdict
Model card Files
xet
 Community
alexwengg commited on
Commit
3ee6a05
·
verified ·
1 Parent(s): 3f4623d

Document failed Verdict LUT8 parity without publishing package

Browse files
Files changed (5) hide show
  1. README.md +3 -1
  2. compression-lut8_kmeans_per_tensor-L128.json +13 -0
  3. lut8-L128-suite-parity.json +1251 -0
  4. verification-verdict_lut8_kmeans_per_tensor_L128_candidates25.json +177 -0
  5. verify-compression.py +165 -0
README.md CHANGED
@@ -18,6 +18,8 @@ Inputs are `input_ids` and `attention_mask` of shape `[1,L]`, and `class_marker_
18
 
19
  Verified on Apple M5 Pro/macOS 27.0 against the pinned PyTorch model: L128 passed 4/4 smoke decisions (including two native abstentions), worst calibrated probability difference 0.00341. L512 passed 5/5 (including one 283-token public Decision Index row), worst difference 0.00070. The 20-iteration L128 `coreml-cli` median was 3.627 ms CPU+GPU, 3.710 ms CPU+ANE, 3.915 ms automatic, and 11.721 ms CPU only. L512 median automatic model call time across its five parity requests was 8.15 ms. Rendering and tokenization add time. The verification and profile JSON files include the details.
20
 
21
- The public Decision Index tracker reports Verdict at 13.38. Its historical checkpoint and renderer have not been authenticated against this pinned release, so this artifact does **not** claim to reproduce that score. The two buckets cover its 512-token public limit, but full-suite correctness and quantized parity are still pending.
 
 
22
 
23
  The Hub repo includes the runtime renderer, calibration helper, tokenizer, verification report and SHA-256 asset lock. Conversion source is maintained in [FluidInference/mobius](https://github.com/FluidInference/mobius). Upstream model and code: [Verdict model](https://huggingface.co/heman10x/rlcd-modernbert-151m), [Verdict source](https://github.com/Heman10x-NGU/Verdict-open-jev), Apache-2.0. Base architecture: [`knowledgator/gliclass-modern-base-v2.0`](https://huggingface.co/knowledgator/gliclass-modern-base-v2.0).
 
18
 
19
  Verified on Apple M5 Pro/macOS 27.0 against the pinned PyTorch model: L128 passed 4/4 smoke decisions (including two native abstentions), worst calibrated probability difference 0.00341. L512 passed 5/5 (including one 283-token public Decision Index row), worst difference 0.00070. The 20-iteration L128 `coreml-cli` median was 3.627 ms CPU+GPU, 3.710 ms CPU+ANE, 3.915 ms automatic, and 11.721 ms CPU only. L512 median automatic model call time across its five parity requests was 8.15 ms. Rendering and tokenization add time. The verification and profile JSON files include the details.
20
 
21
+ The public Decision Index tracker reports Verdict at 13.38. Its historical checkpoint and renderer have not been authenticated against this pinned release, so this artifact does **not** claim to reproduce that score. The two FP16 buckets cover its 512-token public limit.
22
+
23
+ An experimental L128 8-bit k-means LUT cut package size to 151,878,624 bytes, but **failed** the predeclared 100-request FP16 comparison: 96/100 calibrated selection and abstention agreement (four abstention flips), versus a 99/100 gate. P95 calibrated probability error was 0.0197; worst was 0.0297. Median measured model-call time was 3.48 ms LUT8 versus 3.34 ms FP16. The LUT8 package is **not included** here as a validated model. See `lut8-L128-suite-parity.json` and `verification-verdict_lut8_kmeans_per_tensor_L128_candidates25.json` for the reports.
24
 
25
  The Hub repo includes the runtime renderer, calibration helper, tokenizer, verification report and SHA-256 asset lock. Conversion source is maintained in [FluidInference/mobius](https://github.com/FluidInference/mobius). Upstream model and code: [Verdict model](https://huggingface.co/heman10x/rlcd-modernbert-151m), [Verdict source](https://github.com/Heman10x-NGU/Verdict-open-jev), Apache-2.0. Base architecture: [`knowledgator/gliclass-modern-base-v2.0`](https://huggingface.co/knowledgator/gliclass-modern-base-v2.0).
compression-lut8_kmeans_per_tensor-L128.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "source": "build/verdict_fp16_L128_candidates25.mlpackage",
3
+ "output": "build/verdict_lut8_kmeans_per_tensor_L128_candidates25.mlpackage",
4
+ "length": 128,
5
+ "bits": 8,
6
+ "mode": "kmeans",
7
+ "granularity": "per_tensor",
8
+ "group_size": 32,
9
+ "source_bytes": 303210832,
10
+ "output_bytes": 151878624,
11
+ "compression_seconds": 63.7199045419693,
12
+ "coremltools": "9.0"
13
+ }
lut8-L128-suite-parity.json ADDED
@@ -0,0 +1,1251 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "suite_sha256": "340c61073530a7095b147ce1a4b3c36fb57983d051f431e1cdfba25521e17d93",
3
+ "selection_protocol": "First 10 eligible rows per family in suite order, 100 total; no gold labels used",
4
+ "selected_row_ids": [
5
+ "Humicroedit:test:14007-588",
6
+ "Humicroedit:test:2170-12284",
7
+ "Humicroedit:test:8918-14097",
8
+ "Humicroedit:test:5634-12024",
9
+ "Humicroedit:test:8836-5189",
10
+ "Humicroedit:test:11461-13681",
11
+ "Humicroedit:test:1312-1055",
12
+ "Humicroedit:test:12190-13632",
13
+ "Humicroedit:test:9222-5353",
14
+ "Humicroedit:test:4995-6921",
15
+ "MMLU:test:6188",
16
+ "MMLU:test:8782",
17
+ "MMLU:test:7230",
18
+ "MMLU:test:9510",
19
+ "MMLU:test:12666",
20
+ "MMLU:test:8088",
21
+ "MMLU:test:7273",
22
+ "MMLU:test:8647",
23
+ "MMLU:test:3800",
24
+ "MMLU:test:5414",
25
+ "GPQA-Diamond:test:169",
26
+ "GPQA-Diamond:test:79",
27
+ "GPQA-Diamond:test:36",
28
+ "GPQA-Diamond:test:29",
29
+ "GPQA-Diamond:test:182",
30
+ "GPQA-Diamond:test:13",
31
+ "GPQA-Diamond:test:65",
32
+ "GPQA-Diamond:test:184",
33
+ "GPQA-Diamond:test:31",
34
+ "GPQA-Diamond:test:47",
35
+ "GSM8K-10:test:1104",
36
+ "GSM8K-4:test:639",
37
+ "GSM8K-4:test:872",
38
+ "GSM8K-4:test:782",
39
+ "GSM8K-4:test:1015",
40
+ "GSM8K-4:test:18",
41
+ "GSM8K-4:test:97",
42
+ "GSM8K-4:test:253",
43
+ "GSM8K-4:test:556",
44
+ "GSM8K-4:test:1050",
45
+ "GSM8K-4:test:1298",
46
+ "iSarcasmEval-A-Ar:test:590",
47
+ "iSarcasmEval-A-Ar:test:595",
48
+ "iSarcasmEval-A-Ar:test:208",
49
+ "iSarcasmEval-A-Ar:test:319",
50
+ "iSarcasmEval-A-Ar:test:367",
51
+ "iSarcasmEval-A-Ar:test:928",
52
+ "iSarcasmEval-A-Ar:test:185",
53
+ "iSarcasmEval-A-Ar:test:1281",
54
+ "iSarcasmEval-A-Ar:test:1211",
55
+ "iSarcasmEval-A-Ar:test:258",
56
+ "iSarcasmEval-A-En:test:245",
57
+ "iSarcasmEval-A-En:test:1067",
58
+ "iSarcasmEval-A-En:test:875",
59
+ "iSarcasmEval-A-En:test:1253",
60
+ "iSarcasmEval-A-En:test:1244",
61
+ "iSarcasmEval-A-En:test:750",
62
+ "iSarcasmEval-A-En:test:207",
63
+ "iSarcasmEval-A-En:test:577",
64
+ "iSarcasmEval-A-En:test:599",
65
+ "iSarcasmEval-A-En:test:772",
66
+ "iSarcasmEval-C-Ar:test:78",
67
+ "iSarcasmEval-C-Ar:test:14",
68
+ "iSarcasmEval-C-Ar:test:100",
69
+ "iSarcasmEval-C-Ar:test:110",
70
+ "iSarcasmEval-C-Ar:test:107",
71
+ "iSarcasmEval-C-Ar:test:35",
72
+ "iSarcasmEval-C-Ar:test:122",
73
+ "iSarcasmEval-C-Ar:test:127",
74
+ "iSarcasmEval-C-Ar:test:94",
75
+ "iSarcasmEval-C-Ar:test:10",
76
+ "iSarcasmEval-C-En:test:40",
77
+ "iSarcasmEval-C-En:test:52",
78
+ "iSarcasmEval-C-En:test:195",
79
+ "iSarcasmEval-C-En:test:104",
80
+ "iSarcasmEval-C-En:test:174",
81
+ "iSarcasmEval-C-En:test:16",
82
+ "iSarcasmEval-C-En:test:85",
83
+ "iSarcasmEval-C-En:test:57",
84
+ "iSarcasmEval-C-En:test:187",
85
+ "iSarcasmEval-C-En:test:156",
86
+ "VAST:test:22862",
87
+ "VAST:test:7312",
88
+ "VAST:test:22585",
89
+ "VAST:test:14979",
90
+ "VAST:test:23330",
91
+ "VAST:test:17941",
92
+ "VAST:test:2995",
93
+ "VAST:test:23098",
94
+ "VAST:test:23129",
95
+ "VAST:test:22754",
96
+ "CRUXEval-output-choice:test:sample_406",
97
+ "CRUXEval-output-choice:test:sample_641",
98
+ "CRUXEval-output-choice:test:sample_637",
99
+ "CRUXEval-output-choice:test:sample_735",
100
+ "CRUXEval-output-choice:test:sample_28",
101
+ "CRUXEval-output-choice:test:sample_46",
102
+ "CRUXEval-output-choice:test:sample_561",
103
+ "CRUXEval-output-choice:test:sample_73",
104
+ "CRUXEval-output-choice:test:sample_513"
105
+ ],
106
+ "families": {
107
+ "Humicroedit": 10,
108
+ "MMLU": 10,
109
+ "GPQA-Diamond": 10,
110
+ "GSM8K-10": 1,
111
+ "GSM8K-4": 10,
112
+ "iSarcasmEval-A-Ar": 10,
113
+ "iSarcasmEval-A-En": 10,
114
+ "iSarcasmEval-C-Ar": 10,
115
+ "iSarcasmEval-C-En": 10,
116
+ "VAST": 10,
117
+ "CRUXEval-output-choice": 9
118
+ },
119
+ "skipped_before_limit": {
120
+ "multiple_questions": 655,
121
+ "overlength": 994,
122
+ "invalid_or_overcapacity": 168
123
+ },
124
+ "packages": {
125
+ "fp16": "verdict_fp16_L128_candidates25.mlpackage",
126
+ "lut8": "verdict_lut8_kmeans_per_tensor_L128_candidates25.mlpackage"
127
+ },
128
+ "hardware": {
129
+ "chip": "Apple M5 Pro",
130
+ "macos": "27.0"
131
+ },
132
+ "questions": 100,
133
+ "selection_agreement": 0.96,
134
+ "abstention_agreement": 0.96,
135
+ "p95_probability_error": 0.019697460101399795,
136
+ "worst_probability_error": 0.029712050394336498,
137
+ "median_model_call_ms": {
138
+ "fp16": 3.3446039888076484,
139
+ "lut8": 3.477958496659994
140
+ },
141
+ "gates": {
142
+ "min_selection_agreement": 0.99,
143
+ "min_abstention_agreement": 0.99,
144
+ "max_p95_probability_error": 0.02,
145
+ "max_worst_probability_error": 0.1
146
+ },
147
+ "rows": [
148
+ {
149
+ "id": "Humicroedit:test:14007-588",
150
+ "family": "Humicroedit",
151
+ "tokens": 55,
152
+ "candidates": 3,
153
+ "fp16_selected_id": "__insufficient_evidence__",
154
+ "lut8_selected_id": "__insufficient_evidence__",
155
+ "selection_agrees": true,
156
+ "abstention_agrees": true,
157
+ "max_probability_error": 0.007769925051676008
158
+ },
159
+ {
160
+ "id": "Humicroedit:test:2170-12284",
161
+ "family": "Humicroedit",
162
+ "tokens": 70,
163
+ "candidates": 3,
164
+ "fp16_selected_id": "__insufficient_evidence__",
165
+ "lut8_selected_id": "__insufficient_evidence__",
166
+ "selection_agrees": true,
167
+ "abstention_agrees": true,
168
+ "max_probability_error": 0.0027414398258541195
169
+ },
170
+ {
171
+ "id": "Humicroedit:test:8918-14097",
172
+ "family": "Humicroedit",
173
+ "tokens": 65,
174
+ "candidates": 3,
175
+ "fp16_selected_id": "__insufficient_evidence__",
176
+ "lut8_selected_id": "__insufficient_evidence__",
177
+ "selection_agrees": true,
178
+ "abstention_agrees": true,
179
+ "max_probability_error": 0.004705787043594645
180
+ },
181
+ {
182
+ "id": "Humicroedit:test:5634-12024",
183
+ "family": "Humicroedit",
184
+ "tokens": 64,
185
+ "candidates": 3,
186
+ "fp16_selected_id": "__insufficient_evidence__",
187
+ "lut8_selected_id": "__insufficient_evidence__",
188
+ "selection_agrees": true,
189
+ "abstention_agrees": true,
190
+ "max_probability_error": 0.009245197040519793
191
+ },
192
+ {
193
+ "id": "Humicroedit:test:8836-5189",
194
+ "family": "Humicroedit",
195
+ "tokens": 43,
196
+ "candidates": 3,
197
+ "fp16_selected_id": "__insufficient_evidence__",
198
+ "lut8_selected_id": "__insufficient_evidence__",
199
+ "selection_agrees": true,
200
+ "abstention_agrees": true,
201
+ "max_probability_error": 0.0011339294246149878
202
+ },
203
+ {
204
+ "id": "Humicroedit:test:11461-13681",
205
+ "family": "Humicroedit",
206
+ "tokens": 49,
207
+ "candidates": 3,
208
+ "fp16_selected_id": "__insufficient_evidence__",
209
+ "lut8_selected_id": "__insufficient_evidence__",
210
+ "selection_agrees": true,
211
+ "abstention_agrees": true,
212
+ "max_probability_error": 0.0020556291809581406
213
+ },
214
+ {
215
+ "id": "Humicroedit:test:1312-1055",
216
+ "family": "Humicroedit",
217
+ "tokens": 46,
218
+ "candidates": 3,
219
+ "fp16_selected_id": "__insufficient_evidence__",
220
+ "lut8_selected_id": "__insufficient_evidence__",
221
+ "selection_agrees": true,
222
+ "abstention_agrees": true,
223
+ "max_probability_error": 0.003671344683128158
224
+ },
225
+ {
226
+ "id": "Humicroedit:test:12190-13632",
227
+ "family": "Humicroedit",
228
+ "tokens": 93,
229
+ "candidates": 3,
230
+ "fp16_selected_id": "__insufficient_evidence__",
231
+ "lut8_selected_id": "__insufficient_evidence__",
232
+ "selection_agrees": true,
233
+ "abstention_agrees": true,
234
+ "max_probability_error": 0.0030128346829157993
235
+ },
236
+ {
237
+ "id": "Humicroedit:test:9222-5353",
238
+ "family": "Humicroedit",
239
+ "tokens": 52,
240
+ "candidates": 3,
241
+ "fp16_selected_id": "__insufficient_evidence__",
242
+ "lut8_selected_id": "__insufficient_evidence__",
243
+ "selection_agrees": true,
244
+ "abstention_agrees": true,
245
+ "max_probability_error": 0.007735013722117268
246
+ },
247
+ {
248
+ "id": "Humicroedit:test:4995-6921",
249
+ "family": "Humicroedit",
250
+ "tokens": 71,
251
+ "candidates": 3,
252
+ "fp16_selected_id": "__insufficient_evidence__",
253
+ "lut8_selected_id": "__insufficient_evidence__",
254
+ "selection_agrees": true,
255
+ "abstention_agrees": true,
256
+ "max_probability_error": 0.005057035163770873
257
+ },
258
+ {
259
+ "id": "MMLU:test:6188",
260
+ "family": "MMLU",
261
+ "tokens": 74,
262
+ "candidates": 5,
263
+ "fp16_selected_id": "__insufficient_evidence__",
264
+ "lut8_selected_id": "__insufficient_evidence__",
265
+ "selection_agrees": true,
266
+ "abstention_agrees": true,
267
+ "max_probability_error": 0.010490436657739438
268
+ },
269
+ {
270
+ "id": "MMLU:test:8782",
271
+ "family": "MMLU",
272
+ "tokens": 110,
273
+ "candidates": 5,
274
+ "fp16_selected_id": "D",
275
+ "lut8_selected_id": "D",
276
+ "selection_agrees": true,
277
+ "abstention_agrees": true,
278
+ "max_probability_error": 0.00912150828226932
279
+ },
280
+ {
281
+ "id": "MMLU:test:7230",
282
+ "family": "MMLU",
283
+ "tokens": 79,
284
+ "candidates": 5,
285
+ "fp16_selected_id": "C",
286
+ "lut8_selected_id": "C",
287
+ "selection_agrees": true,
288
+ "abstention_agrees": true,
289
+ "max_probability_error": 0.002089038840056179
290
+ },
291
+ {
292
+ "id": "MMLU:test:9510",
293
+ "family": "MMLU",
294
+ "tokens": 73,
295
+ "candidates": 5,
296
+ "fp16_selected_id": "__insufficient_evidence__",
297
+ "lut8_selected_id": "__insufficient_evidence__",
298
+ "selection_agrees": true,
299
+ "abstention_agrees": true,
300
+ "max_probability_error": 0.0062008828294060625
301
+ },
302
+ {
303
+ "id": "MMLU:test:12666",
304
+ "family": "MMLU",
305
+ "tokens": 70,
306
+ "candidates": 5,
307
+ "fp16_selected_id": "__insufficient_evidence__",
308
+ "lut8_selected_id": "__insufficient_evidence__",
309
+ "selection_agrees": true,
310
+ "abstention_agrees": true,
311
+ "max_probability_error": 0.010176037829099593
312
+ },
313
+ {
314
+ "id": "MMLU:test:8088",
315
+ "family": "MMLU",
316
+ "tokens": 48,
317
+ "candidates": 5,
318
+ "fp16_selected_id": "__insufficient_evidence__",
319
+ "lut8_selected_id": "__insufficient_evidence__",
320
+ "selection_agrees": true,
321
+ "abstention_agrees": true,
322
+ "max_probability_error": 0.011406423391789433
323
+ },
324
+ {
325
+ "id": "MMLU:test:7273",
326
+ "family": "MMLU",
327
+ "tokens": 68,
328
+ "candidates": 5,
329
+ "fp16_selected_id": "C",
330
+ "lut8_selected_id": "C",
331
+ "selection_agrees": true,
332
+ "abstention_agrees": true,
333
+ "max_probability_error": 0.01598426395933683
334
+ },
335
+ {
336
+ "id": "MMLU:test:8647",
337
+ "family": "MMLU",
338
+ "tokens": 123,
339
+ "candidates": 5,
340
+ "fp16_selected_id": "A",
341
+ "lut8_selected_id": "A",
342
+ "selection_agrees": true,
343
+ "abstention_agrees": true,
344
+ "max_probability_error": 0.011774788659825314
345
+ },
346
+ {
347
+ "id": "MMLU:test:3800",
348
+ "family": "MMLU",
349
+ "tokens": 94,
350
+ "candidates": 5,
351
+ "fp16_selected_id": "__insufficient_evidence__",
352
+ "lut8_selected_id": "__insufficient_evidence__",
353
+ "selection_agrees": true,
354
+ "abstention_agrees": true,
355
+ "max_probability_error": 0.023540160925715203
356
+ },
357
+ {
358
+ "id": "MMLU:test:5414",
359
+ "family": "MMLU",
360
+ "tokens": 67,
361
+ "candidates": 5,
362
+ "fp16_selected_id": "__insufficient_evidence__",
363
+ "lut8_selected_id": "__insufficient_evidence__",
364
+ "selection_agrees": true,
365
+ "abstention_agrees": true,
366
+ "max_probability_error": 0.012336045670009121
367
+ },
368
+ {
369
+ "id": "GPQA-Diamond:test:169",
370
+ "family": "GPQA-Diamond",
371
+ "tokens": 118,
372
+ "candidates": 5,
373
+ "fp16_selected_id": "__insufficient_evidence__",
374
+ "lut8_selected_id": "__insufficient_evidence__",
375
+ "selection_agrees": true,
376
+ "abstention_agrees": true,
377
+ "max_probability_error": 0.012852488140051965
378
+ },
379
+ {
380
+ "id": "GPQA-Diamond:test:79",
381
+ "family": "GPQA-Diamond",
382
+ "tokens": 71,
383
+ "candidates": 5,
384
+ "fp16_selected_id": "__insufficient_evidence__",
385
+ "lut8_selected_id": "__insufficient_evidence__",
386
+ "selection_agrees": true,
387
+ "abstention_agrees": true,
388
+ "max_probability_error": 0.019483348102181552
389
+ },
390
+ {
391
+ "id": "GPQA-Diamond:test:36",
392
+ "family": "GPQA-Diamond",
393
+ "tokens": 127,
394
+ "candidates": 5,
395
+ "fp16_selected_id": "D",
396
+ "lut8_selected_id": "D",
397
+ "selection_agrees": true,
398
+ "abstention_agrees": true,
399
+ "max_probability_error": 0.004040489049561946
400
+ },
401
+ {
402
+ "id": "GPQA-Diamond:test:29",
403
+ "family": "GPQA-Diamond",
404
+ "tokens": 65,
405
+ "candidates": 5,
406
+ "fp16_selected_id": "C",
407
+ "lut8_selected_id": "__insufficient_evidence__",
408
+ "selection_agrees": false,
409
+ "abstention_agrees": false,
410
+ "max_probability_error": 0.007356627780481628
411
+ },
412
+ {
413
+ "id": "GPQA-Diamond:test:182",
414
+ "family": "GPQA-Diamond",
415
+ "tokens": 73,
416
+ "candidates": 5,
417
+ "fp16_selected_id": "__insufficient_evidence__",
418
+ "lut8_selected_id": "__insufficient_evidence__",
419
+ "selection_agrees": true,
420
+ "abstention_agrees": true,
421
+ "max_probability_error": 0.005340863715347838
422
+ },
423
+ {
424
+ "id": "GPQA-Diamond:test:13",
425
+ "family": "GPQA-Diamond",
426
+ "tokens": 117,
427
+ "candidates": 5,
428
+ "fp16_selected_id": "__insufficient_evidence__",
429
+ "lut8_selected_id": "__insufficient_evidence__",
430
+ "selection_agrees": true,
431
+ "abstention_agrees": true,
432
+ "max_probability_error": 0.012931709198357288
433
+ },
434
+ {
435
+ "id": "GPQA-Diamond:test:65",
436
+ "family": "GPQA-Diamond",
437
+ "tokens": 84,
438
+ "candidates": 5,
439
+ "fp16_selected_id": "D",
440
+ "lut8_selected_id": "D",
441
+ "selection_agrees": true,
442
+ "abstention_agrees": true,
443
+ "max_probability_error": 0.019544194093023604
444
+ },
445
+ {
446
+ "id": "GPQA-Diamond:test:184",
447
+ "family": "GPQA-Diamond",
448
+ "tokens": 128,
449
+ "candidates": 5,
450
+ "fp16_selected_id": "C",
451
+ "lut8_selected_id": "C",
452
+ "selection_agrees": true,
453
+ "abstention_agrees": true,
454
+ "max_probability_error": 0.0051351520020054275
455
+ },
456
+ {
457
+ "id": "GPQA-Diamond:test:31",
458
+ "family": "GPQA-Diamond",
459
+ "tokens": 105,
460
+ "candidates": 5,
461
+ "fp16_selected_id": "__insufficient_evidence__",
462
+ "lut8_selected_id": "__insufficient_evidence__",
463
+ "selection_agrees": true,
464
+ "abstention_agrees": true,
465
+ "max_probability_error": 0.022686046788916947
466
+ },
467
+ {
468
+ "id": "GPQA-Diamond:test:47",
469
+ "family": "GPQA-Diamond",
470
+ "tokens": 111,
471
+ "candidates": 5,
472
+ "fp16_selected_id": "__insufficient_evidence__",
473
+ "lut8_selected_id": "__insufficient_evidence__",
474
+ "selection_agrees": true,
475
+ "abstention_agrees": true,
476
+ "max_probability_error": 0.009503378973940546
477
+ },
478
+ {
479
+ "id": "GSM8K-10:test:1104",
480
+ "family": "GSM8K-10",
481
+ "tokens": 128,
482
+ "candidates": 11,
483
+ "fp16_selected_id": "__insufficient_evidence__",
484
+ "lut8_selected_id": "__insufficient_evidence__",
485
+ "selection_agrees": true,
486
+ "abstention_agrees": true,
487
+ "max_probability_error": 0.005043460855172099
488
+ },
489
+ {
490
+ "id": "GSM8K-4:test:639",
491
+ "family": "GSM8K-4",
492
+ "tokens": 121,
493
+ "candidates": 5,
494
+ "fp16_selected_id": "__insufficient_evidence__",
495
+ "lut8_selected_id": "__insufficient_evidence__",
496
+ "selection_agrees": true,
497
+ "abstention_agrees": true,
498
+ "max_probability_error": 0.00306730770397462
499
+ },
500
+ {
501
+ "id": "GSM8K-4:test:872",
502
+ "family": "GSM8K-4",
503
+ "tokens": 116,
504
+ "candidates": 5,
505
+ "fp16_selected_id": "option_0",
506
+ "lut8_selected_id": "option_0",
507
+ "selection_agrees": true,
508
+ "abstention_agrees": true,
509
+ "max_probability_error": 0.007077592147996636
510
+ },
511
+ {
512
+ "id": "GSM8K-4:test:782",
513
+ "family": "GSM8K-4",
514
+ "tokens": 122,
515
+ "candidates": 5,
516
+ "fp16_selected_id": "__insufficient_evidence__",
517
+ "lut8_selected_id": "__insufficient_evidence__",
518
+ "selection_agrees": true,
519
+ "abstention_agrees": true,
520
+ "max_probability_error": 0.008950147437850553
521
+ },
522
+ {
523
+ "id": "GSM8K-4:test:1015",
524
+ "family": "GSM8K-4",
525
+ "tokens": 123,
526
+ "candidates": 5,
527
+ "fp16_selected_id": "__insufficient_evidence__",
528
+ "lut8_selected_id": "__insufficient_evidence__",
529
+ "selection_agrees": true,
530
+ "abstention_agrees": true,
531
+ "max_probability_error": 0.014232401908234299
532
+ },
533
+ {
534
+ "id": "GSM8K-4:test:18",
535
+ "family": "GSM8K-4",
536
+ "tokens": 104,
537
+ "candidates": 5,
538
+ "fp16_selected_id": "__insufficient_evidence__",
539
+ "lut8_selected_id": "__insufficient_evidence__",
540
+ "selection_agrees": true,
541
+ "abstention_agrees": true,
542
+ "max_probability_error": 0.004912296386085979
543
+ },
544
+ {
545
+ "id": "GSM8K-4:test:97",
546
+ "family": "GSM8K-4",
547
+ "tokens": 128,
548
+ "candidates": 5,
549
+ "fp16_selected_id": "__insufficient_evidence__",
550
+ "lut8_selected_id": "__insufficient_evidence__",
551
+ "selection_agrees": true,
552
+ "abstention_agrees": true,
553
+ "max_probability_error": 0.015293694776556344
554
+ },
555
+ {
556
+ "id": "GSM8K-4:test:253",
557
+ "family": "GSM8K-4",
558
+ "tokens": 112,
559
+ "candidates": 5,
560
+ "fp16_selected_id": "__insufficient_evidence__",
561
+ "lut8_selected_id": "__insufficient_evidence__",
562
+ "selection_agrees": true,
563
+ "abstention_agrees": true,
564
+ "max_probability_error": 0.004812012840649499
565
+ },
566
+ {
567
+ "id": "GSM8K-4:test:556",
568
+ "family": "GSM8K-4",
569
+ "tokens": 117,
570
+ "candidates": 5,
571
+ "fp16_selected_id": "__insufficient_evidence__",
572
+ "lut8_selected_id": "__insufficient_evidence__",
573
+ "selection_agrees": true,
574
+ "abstention_agrees": true,
575
+ "max_probability_error": 0.0053275976461159835
576
+ },
577
+ {
578
+ "id": "GSM8K-4:test:1050",
579
+ "family": "GSM8K-4",
580
+ "tokens": 117,
581
+ "candidates": 5,
582
+ "fp16_selected_id": "__insufficient_evidence__",
583
+ "lut8_selected_id": "__insufficient_evidence__",
584
+ "selection_agrees": true,
585
+ "abstention_agrees": true,
586
+ "max_probability_error": 0.005315071755024547
587
+ },
588
+ {
589
+ "id": "GSM8K-4:test:1298",
590
+ "family": "GSM8K-4",
591
+ "tokens": 117,
592
+ "candidates": 5,
593
+ "fp16_selected_id": "option_3",
594
+ "lut8_selected_id": "option_3",
595
+ "selection_agrees": true,
596
+ "abstention_agrees": true,
597
+ "max_probability_error": 0.01144584854187225
598
+ },
599
+ {
600
+ "id": "iSarcasmEval-A-Ar:test:590",
601
+ "family": "iSarcasmEval-A-Ar",
602
+ "tokens": 45,
603
+ "candidates": 3,
604
+ "fp16_selected_id": "__insufficient_evidence__",
605
+ "lut8_selected_id": "__insufficient_evidence__",
606
+ "selection_agrees": true,
607
+ "abstention_agrees": true,
608
+ "max_probability_error": 0.007535206563071606
609
+ },
610
+ {
611
+ "id": "iSarcasmEval-A-Ar:test:595",
612
+ "family": "iSarcasmEval-A-Ar",
613
+ "tokens": 49,
614
+ "candidates": 3,
615
+ "fp16_selected_id": "no",
616
+ "lut8_selected_id": "no",
617
+ "selection_agrees": true,
618
+ "abstention_agrees": true,
619
+ "max_probability_error": 0.006617697891030394
620
+ },
621
+ {
622
+ "id": "iSarcasmEval-A-Ar:test:208",
623
+ "family": "iSarcasmEval-A-Ar",
624
+ "tokens": 38,
625
+ "candidates": 3,
626
+ "fp16_selected_id": "no",
627
+ "lut8_selected_id": "no",
628
+ "selection_agrees": true,
629
+ "abstention_agrees": true,
630
+ "max_probability_error": 0.0036362449126985386
631
+ },
632
+ {
633
+ "id": "iSarcasmEval-A-Ar:test:319",
634
+ "family": "iSarcasmEval-A-Ar",
635
+ "tokens": 55,
636
+ "candidates": 3,
637
+ "fp16_selected_id": "no",
638
+ "lut8_selected_id": "no",
639
+ "selection_agrees": true,
640
+ "abstention_agrees": true,
641
+ "max_probability_error": 0.009415075063047063
642
+ },
643
+ {
644
+ "id": "iSarcasmEval-A-Ar:test:367",
645
+ "family": "iSarcasmEval-A-Ar",
646
+ "tokens": 54,
647
+ "candidates": 3,
648
+ "fp16_selected_id": "no",
649
+ "lut8_selected_id": "no",
650
+ "selection_agrees": true,
651
+ "abstention_agrees": true,
652
+ "max_probability_error": 0.004521733373459125
653
+ },
654
+ {
655
+ "id": "iSarcasmEval-A-Ar:test:928",
656
+ "family": "iSarcasmEval-A-Ar",
657
+ "tokens": 47,
658
+ "candidates": 3,
659
+ "fp16_selected_id": "__insufficient_evidence__",
660
+ "lut8_selected_id": "__insufficient_evidence__",
661
+ "selection_agrees": true,
662
+ "abstention_agrees": true,
663
+ "max_probability_error": 0.0060538907337006065
664
+ },
665
+ {
666
+ "id": "iSarcasmEval-A-Ar:test:185",
667
+ "family": "iSarcasmEval-A-Ar",
668
+ "tokens": 48,
669
+ "candidates": 3,
670
+ "fp16_selected_id": "__insufficient_evidence__",
671
+ "lut8_selected_id": "__insufficient_evidence__",
672
+ "selection_agrees": true,
673
+ "abstention_agrees": true,
674
+ "max_probability_error": 0.01015332437915567
675
+ },
676
+ {
677
+ "id": "iSarcasmEval-A-Ar:test:1281",
678
+ "family": "iSarcasmEval-A-Ar",
679
+ "tokens": 43,
680
+ "candidates": 3,
681
+ "fp16_selected_id": "no",
682
+ "lut8_selected_id": "no",
683
+ "selection_agrees": true,
684
+ "abstention_agrees": true,
685
+ "max_probability_error": 0.008593151221636453
686
+ },
687
+ {
688
+ "id": "iSarcasmEval-A-Ar:test:1211",
689
+ "family": "iSarcasmEval-A-Ar",
690
+ "tokens": 53,
691
+ "candidates": 3,
692
+ "fp16_selected_id": "__insufficient_evidence__",
693
+ "lut8_selected_id": "__insufficient_evidence__",
694
+ "selection_agrees": true,
695
+ "abstention_agrees": true,
696
+ "max_probability_error": 0.006962390398277618
697
+ },
698
+ {
699
+ "id": "iSarcasmEval-A-Ar:test:258",
700
+ "family": "iSarcasmEval-A-Ar",
701
+ "tokens": 52,
702
+ "candidates": 3,
703
+ "fp16_selected_id": "no",
704
+ "lut8_selected_id": "no",
705
+ "selection_agrees": true,
706
+ "abstention_agrees": true,
707
+ "max_probability_error": 0.0010695952834968159
708
+ },
709
+ {
710
+ "id": "iSarcasmEval-A-En:test:245",
711
+ "family": "iSarcasmEval-A-En",
712
+ "tokens": 41,
713
+ "candidates": 3,
714
+ "fp16_selected_id": "__insufficient_evidence__",
715
+ "lut8_selected_id": "__insufficient_evidence__",
716
+ "selection_agrees": true,
717
+ "abstention_agrees": true,
718
+ "max_probability_error": 0.007354904008214813
719
+ },
720
+ {
721
+ "id": "iSarcasmEval-A-En:test:1067",
722
+ "family": "iSarcasmEval-A-En",
723
+ "tokens": 59,
724
+ "candidates": 3,
725
+ "fp16_selected_id": "yes",
726
+ "lut8_selected_id": "yes",
727
+ "selection_agrees": true,
728
+ "abstention_agrees": true,
729
+ "max_probability_error": 0.0034244909416177305
730
+ },
731
+ {
732
+ "id": "iSarcasmEval-A-En:test:875",
733
+ "family": "iSarcasmEval-A-En",
734
+ "tokens": 59,
735
+ "candidates": 3,
736
+ "fp16_selected_id": "__insufficient_evidence__",
737
+ "lut8_selected_id": "__insufficient_evidence__",
738
+ "selection_agrees": true,
739
+ "abstention_agrees": true,
740
+ "max_probability_error": 0.002522534667769749
741
+ },
742
+ {
743
+ "id": "iSarcasmEval-A-En:test:1253",
744
+ "family": "iSarcasmEval-A-En",
745
+ "tokens": 59,
746
+ "candidates": 3,
747
+ "fp16_selected_id": "no",
748
+ "lut8_selected_id": "no",
749
+ "selection_agrees": true,
750
+ "abstention_agrees": true,
751
+ "max_probability_error": 0.002583505514246548
752
+ },
753
+ {
754
+ "id": "iSarcasmEval-A-En:test:1244",
755
+ "family": "iSarcasmEval-A-En",
756
+ "tokens": 45,
757
+ "candidates": 3,
758
+ "fp16_selected_id": "yes",
759
+ "lut8_selected_id": "yes",
760
+ "selection_agrees": true,
761
+ "abstention_agrees": true,
762
+ "max_probability_error": 0.003044473280808202
763
+ },
764
+ {
765
+ "id": "iSarcasmEval-A-En:test:750",
766
+ "family": "iSarcasmEval-A-En",
767
+ "tokens": 79,
768
+ "candidates": 3,
769
+ "fp16_selected_id": "__insufficient_evidence__",
770
+ "lut8_selected_id": "__insufficient_evidence__",
771
+ "selection_agrees": true,
772
+ "abstention_agrees": true,
773
+ "max_probability_error": 0.00453627864726977
774
+ },
775
+ {
776
+ "id": "iSarcasmEval-A-En:test:207",
777
+ "family": "iSarcasmEval-A-En",
778
+ "tokens": 47,
779
+ "candidates": 3,
780
+ "fp16_selected_id": "no",
781
+ "lut8_selected_id": "no",
782
+ "selection_agrees": true,
783
+ "abstention_agrees": true,
784
+ "max_probability_error": 0.003737232053770634
785
+ },
786
+ {
787
+ "id": "iSarcasmEval-A-En:test:577",
788
+ "family": "iSarcasmEval-A-En",
789
+ "tokens": 51,
790
+ "candidates": 3,
791
+ "fp16_selected_id": "no",
792
+ "lut8_selected_id": "no",
793
+ "selection_agrees": true,
794
+ "abstention_agrees": true,
795
+ "max_probability_error": 0.008809466848203051
796
+ },
797
+ {
798
+ "id": "iSarcasmEval-A-En:test:599",
799
+ "family": "iSarcasmEval-A-En",
800
+ "tokens": 65,
801
+ "candidates": 3,
802
+ "fp16_selected_id": "no",
803
+ "lut8_selected_id": "no",
804
+ "selection_agrees": true,
805
+ "abstention_agrees": true,
806
+ "max_probability_error": 0.005113272963934878
807
+ },
808
+ {
809
+ "id": "iSarcasmEval-A-En:test:772",
810
+ "family": "iSarcasmEval-A-En",
811
+ "tokens": 51,
812
+ "candidates": 3,
813
+ "fp16_selected_id": "__insufficient_evidence__",
814
+ "lut8_selected_id": "__insufficient_evidence__",
815
+ "selection_agrees": true,
816
+ "abstention_agrees": true,
817
+ "max_probability_error": 0.0004810722181223803
818
+ },
819
+ {
820
+ "id": "iSarcasmEval-C-Ar:test:78",
821
+ "family": "iSarcasmEval-C-Ar",
822
+ "tokens": 52,
823
+ "candidates": 3,
824
+ "fp16_selected_id": "__insufficient_evidence__",
825
+ "lut8_selected_id": "__insufficient_evidence__",
826
+ "selection_agrees": true,
827
+ "abstention_agrees": true,
828
+ "max_probability_error": 0.002704164511572793
829
+ },
830
+ {
831
+ "id": "iSarcasmEval-C-Ar:test:14",
832
+ "family": "iSarcasmEval-C-Ar",
833
+ "tokens": 61,
834
+ "candidates": 3,
835
+ "fp16_selected_id": "text_1",
836
+ "lut8_selected_id": "text_1",
837
+ "selection_agrees": true,
838
+ "abstention_agrees": true,
839
+ "max_probability_error": 0.0018360386448961763
840
+ },
841
+ {
842
+ "id": "iSarcasmEval-C-Ar:test:100",
843
+ "family": "iSarcasmEval-C-Ar",
844
+ "tokens": 87,
845
+ "candidates": 3,
846
+ "fp16_selected_id": "__insufficient_evidence__",
847
+ "lut8_selected_id": "text_0",
848
+ "selection_agrees": false,
849
+ "abstention_agrees": false,
850
+ "max_probability_error": 0.011715809793529208
851
+ },
852
+ {
853
+ "id": "iSarcasmEval-C-Ar:test:110",
854
+ "family": "iSarcasmEval-C-Ar",
855
+ "tokens": 79,
856
+ "candidates": 3,
857
+ "fp16_selected_id": "text_1",
858
+ "lut8_selected_id": "text_1",
859
+ "selection_agrees": true,
860
+ "abstention_agrees": true,
861
+ "max_probability_error": 0.0025496160967801162
862
+ },
863
+ {
864
+ "id": "iSarcasmEval-C-Ar:test:107",
865
+ "family": "iSarcasmEval-C-Ar",
866
+ "tokens": 92,
867
+ "candidates": 3,
868
+ "fp16_selected_id": "text_0",
869
+ "lut8_selected_id": "text_0",
870
+ "selection_agrees": true,
871
+ "abstention_agrees": true,
872
+ "max_probability_error": 0.0038555643183449673
873
+ },
874
+ {
875
+ "id": "iSarcasmEval-C-Ar:test:35",
876
+ "family": "iSarcasmEval-C-Ar",
877
+ "tokens": 68,
878
+ "candidates": 3,
879
+ "fp16_selected_id": "text_1",
880
+ "lut8_selected_id": "text_1",
881
+ "selection_agrees": true,
882
+ "abstention_agrees": true,
883
+ "max_probability_error": 0.0029389524259601507
884
+ },
885
+ {
886
+ "id": "iSarcasmEval-C-Ar:test:122",
887
+ "family": "iSarcasmEval-C-Ar",
888
+ "tokens": 87,
889
+ "candidates": 3,
890
+ "fp16_selected_id": "text_0",
891
+ "lut8_selected_id": "text_0",
892
+ "selection_agrees": true,
893
+ "abstention_agrees": true,
894
+ "max_probability_error": 0.0011127828270222073
895
+ },
896
+ {
897
+ "id": "iSarcasmEval-C-Ar:test:127",
898
+ "family": "iSarcasmEval-C-Ar",
899
+ "tokens": 66,
900
+ "candidates": 3,
901
+ "fp16_selected_id": "text_1",
902
+ "lut8_selected_id": "__insufficient_evidence__",
903
+ "selection_agrees": false,
904
+ "abstention_agrees": false,
905
+ "max_probability_error": 0.00400925698405713
906
+ },
907
+ {
908
+ "id": "iSarcasmEval-C-Ar:test:94",
909
+ "family": "iSarcasmEval-C-Ar",
910
+ "tokens": 92,
911
+ "candidates": 3,
912
+ "fp16_selected_id": "text_1",
913
+ "lut8_selected_id": "text_1",
914
+ "selection_agrees": true,
915
+ "abstention_agrees": true,
916
+ "max_probability_error": 0.0041163496316513615
917
+ },
918
+ {
919
+ "id": "iSarcasmEval-C-Ar:test:10",
920
+ "family": "iSarcasmEval-C-Ar",
921
+ "tokens": 73,
922
+ "candidates": 3,
923
+ "fp16_selected_id": "text_1",
924
+ "lut8_selected_id": "text_1",
925
+ "selection_agrees": true,
926
+ "abstention_agrees": true,
927
+ "max_probability_error": 0.005909693687187523
928
+ },
929
+ {
930
+ "id": "iSarcasmEval-C-En:test:40",
931
+ "family": "iSarcasmEval-C-En",
932
+ "tokens": 73,
933
+ "candidates": 3,
934
+ "fp16_selected_id": "__insufficient_evidence__",
935
+ "lut8_selected_id": "__insufficient_evidence__",
936
+ "selection_agrees": true,
937
+ "abstention_agrees": true,
938
+ "max_probability_error": 0.006713998331879245
939
+ },
940
+ {
941
+ "id": "iSarcasmEval-C-En:test:52",
942
+ "family": "iSarcasmEval-C-En",
943
+ "tokens": 80,
944
+ "candidates": 3,
945
+ "fp16_selected_id": "__insufficient_evidence__",
946
+ "lut8_selected_id": "__insufficient_evidence__",
947
+ "selection_agrees": true,
948
+ "abstention_agrees": true,
949
+ "max_probability_error": 0.008310861776203493
950
+ },
951
+ {
952
+ "id": "iSarcasmEval-C-En:test:195",
953
+ "family": "iSarcasmEval-C-En",
954
+ "tokens": 52,
955
+ "candidates": 3,
956
+ "fp16_selected_id": "__insufficient_evidence__",
957
+ "lut8_selected_id": "__insufficient_evidence__",
958
+ "selection_agrees": true,
959
+ "abstention_agrees": true,
960
+ "max_probability_error": 0.007756577848849755
961
+ },
962
+ {
963
+ "id": "iSarcasmEval-C-En:test:104",
964
+ "family": "iSarcasmEval-C-En",
965
+ "tokens": 52,
966
+ "candidates": 3,
967
+ "fp16_selected_id": "__insufficient_evidence__",
968
+ "lut8_selected_id": "__insufficient_evidence__",
969
+ "selection_agrees": true,
970
+ "abstention_agrees": true,
971
+ "max_probability_error": 0.022609514260547636
972
+ },
973
+ {
974
+ "id": "iSarcasmEval-C-En:test:174",
975
+ "family": "iSarcasmEval-C-En",
976
+ "tokens": 66,
977
+ "candidates": 3,
978
+ "fp16_selected_id": "__insufficient_evidence__",
979
+ "lut8_selected_id": "__insufficient_evidence__",
980
+ "selection_agrees": true,
981
+ "abstention_agrees": true,
982
+ "max_probability_error": 0.01286970918638608
983
+ },
984
+ {
985
+ "id": "iSarcasmEval-C-En:test:16",
986
+ "family": "iSarcasmEval-C-En",
987
+ "tokens": 57,
988
+ "candidates": 3,
989
+ "fp16_selected_id": "__insufficient_evidence__",
990
+ "lut8_selected_id": "__insufficient_evidence__",
991
+ "selection_agrees": true,
992
+ "abstention_agrees": true,
993
+ "max_probability_error": 0.004773977833463339
994
+ },
995
+ {
996
+ "id": "iSarcasmEval-C-En:test:85",
997
+ "family": "iSarcasmEval-C-En",
998
+ "tokens": 68,
999
+ "candidates": 3,
1000
+ "fp16_selected_id": "__insufficient_evidence__",
1001
+ "lut8_selected_id": "__insufficient_evidence__",
1002
+ "selection_agrees": true,
1003
+ "abstention_agrees": true,
1004
+ "max_probability_error": 0.00871297192332332
1005
+ },
1006
+ {
1007
+ "id": "iSarcasmEval-C-En:test:57",
1008
+ "family": "iSarcasmEval-C-En",
1009
+ "tokens": 65,
1010
+ "candidates": 3,
1011
+ "fp16_selected_id": "__insufficient_evidence__",
1012
+ "lut8_selected_id": "__insufficient_evidence__",
1013
+ "selection_agrees": true,
1014
+ "abstention_agrees": true,
1015
+ "max_probability_error": 0.010594482805375893
1016
+ },
1017
+ {
1018
+ "id": "iSarcasmEval-C-En:test:187",
1019
+ "family": "iSarcasmEval-C-En",
1020
+ "tokens": 81,
1021
+ "candidates": 3,
1022
+ "fp16_selected_id": "text_1",
1023
+ "lut8_selected_id": "text_1",
1024
+ "selection_agrees": true,
1025
+ "abstention_agrees": true,
1026
+ "max_probability_error": 0.00431216124624112
1027
+ },
1028
+ {
1029
+ "id": "iSarcasmEval-C-En:test:156",
1030
+ "family": "iSarcasmEval-C-En",
1031
+ "tokens": 81,
1032
+ "candidates": 3,
1033
+ "fp16_selected_id": "text_1",
1034
+ "lut8_selected_id": "text_1",
1035
+ "selection_agrees": true,
1036
+ "abstention_agrees": true,
1037
+ "max_probability_error": 0.005781975723819555
1038
+ },
1039
+ {
1040
+ "id": "VAST:test:22862",
1041
+ "family": "VAST",
1042
+ "tokens": 92,
1043
+ "candidates": 4,
1044
+ "fp16_selected_id": "A",
1045
+ "lut8_selected_id": "A",
1046
+ "selection_agrees": true,
1047
+ "abstention_agrees": true,
1048
+ "max_probability_error": 0.007804000772677122
1049
+ },
1050
+ {
1051
+ "id": "VAST:test:7312",
1052
+ "family": "VAST",
1053
+ "tokens": 118,
1054
+ "candidates": 4,
1055
+ "fp16_selected_id": "C",
1056
+ "lut8_selected_id": "C",
1057
+ "selection_agrees": true,
1058
+ "abstention_agrees": true,
1059
+ "max_probability_error": 0.0074901446729041254
1060
+ },
1061
+ {
1062
+ "id": "VAST:test:22585",
1063
+ "family": "VAST",
1064
+ "tokens": 96,
1065
+ "candidates": 4,
1066
+ "fp16_selected_id": "A",
1067
+ "lut8_selected_id": "A",
1068
+ "selection_agrees": true,
1069
+ "abstention_agrees": true,
1070
+ "max_probability_error": 0.016020381293998853
1071
+ },
1072
+ {
1073
+ "id": "VAST:test:14979",
1074
+ "family": "VAST",
1075
+ "tokens": 99,
1076
+ "candidates": 4,
1077
+ "fp16_selected_id": "A",
1078
+ "lut8_selected_id": "A",
1079
+ "selection_agrees": true,
1080
+ "abstention_agrees": true,
1081
+ "max_probability_error": 0.01327773764760759
1082
+ },
1083
+ {
1084
+ "id": "VAST:test:23330",
1085
+ "family": "VAST",
1086
+ "tokens": 127,
1087
+ "candidates": 4,
1088
+ "fp16_selected_id": "A",
1089
+ "lut8_selected_id": "A",
1090
+ "selection_agrees": true,
1091
+ "abstention_agrees": true,
1092
+ "max_probability_error": 0.004505553297562304
1093
+ },
1094
+ {
1095
+ "id": "VAST:test:17941",
1096
+ "family": "VAST",
1097
+ "tokens": 124,
1098
+ "candidates": 4,
1099
+ "fp16_selected_id": "C",
1100
+ "lut8_selected_id": "C",
1101
+ "selection_agrees": true,
1102
+ "abstention_agrees": true,
1103
+ "max_probability_error": 0.008063342917555505
1104
+ },
1105
+ {
1106
+ "id": "VAST:test:2995",
1107
+ "family": "VAST",
1108
+ "tokens": 113,
1109
+ "candidates": 4,
1110
+ "fp16_selected_id": "__insufficient_evidence__",
1111
+ "lut8_selected_id": "__insufficient_evidence__",
1112
+ "selection_agrees": true,
1113
+ "abstention_agrees": true,
1114
+ "max_probability_error": 0.028016682431132367
1115
+ },
1116
+ {
1117
+ "id": "VAST:test:23098",
1118
+ "family": "VAST",
1119
+ "tokens": 120,
1120
+ "candidates": 4,
1121
+ "fp16_selected_id": "__insufficient_evidence__",
1122
+ "lut8_selected_id": "__insufficient_evidence__",
1123
+ "selection_agrees": true,
1124
+ "abstention_agrees": true,
1125
+ "max_probability_error": 0.005023180604005029
1126
+ },
1127
+ {
1128
+ "id": "VAST:test:23129",
1129
+ "family": "VAST",
1130
+ "tokens": 122,
1131
+ "candidates": 4,
1132
+ "fp16_selected_id": "A",
1133
+ "lut8_selected_id": "A",
1134
+ "selection_agrees": true,
1135
+ "abstention_agrees": true,
1136
+ "max_probability_error": 0.00541524305360469
1137
+ },
1138
+ {
1139
+ "id": "VAST:test:22754",
1140
+ "family": "VAST",
1141
+ "tokens": 119,
1142
+ "candidates": 4,
1143
+ "fp16_selected_id": "A",
1144
+ "lut8_selected_id": "A",
1145
+ "selection_agrees": true,
1146
+ "abstention_agrees": true,
1147
+ "max_probability_error": 0.008606151310190158
1148
+ },
1149
+ {
1150
+ "id": "CRUXEval-output-choice:test:sample_406",
1151
+ "family": "CRUXEval-output-choice",
1152
+ "tokens": 111,
1153
+ "candidates": 3,
1154
+ "fp16_selected_id": "__insufficient_evidence__",
1155
+ "lut8_selected_id": "__insufficient_evidence__",
1156
+ "selection_agrees": true,
1157
+ "abstention_agrees": true,
1158
+ "max_probability_error": 0.008307321950305091
1159
+ },
1160
+ {
1161
+ "id": "CRUXEval-output-choice:test:sample_641",
1162
+ "family": "CRUXEval-output-choice",
1163
+ "tokens": 83,
1164
+ "candidates": 3,
1165
+ "fp16_selected_id": "option_1",
1166
+ "lut8_selected_id": "option_1",
1167
+ "selection_agrees": true,
1168
+ "abstention_agrees": true,
1169
+ "max_probability_error": 0.016792246520733745
1170
+ },
1171
+ {
1172
+ "id": "CRUXEval-output-choice:test:sample_637",
1173
+ "family": "CRUXEval-output-choice",
1174
+ "tokens": 115,
1175
+ "candidates": 3,
1176
+ "fp16_selected_id": "option_0",
1177
+ "lut8_selected_id": "option_0",
1178
+ "selection_agrees": true,
1179
+ "abstention_agrees": true,
1180
+ "max_probability_error": 0.004377568381061847
1181
+ },
1182
+ {
1183
+ "id": "CRUXEval-output-choice:test:sample_735",
1184
+ "family": "CRUXEval-output-choice",
1185
+ "tokens": 125,
1186
+ "candidates": 3,
1187
+ "fp16_selected_id": "__insufficient_evidence__",
1188
+ "lut8_selected_id": "__insufficient_evidence__",
1189
+ "selection_agrees": true,
1190
+ "abstention_agrees": true,
1191
+ "max_probability_error": 0.029712050394336498
1192
+ },
1193
+ {
1194
+ "id": "CRUXEval-output-choice:test:sample_28",
1195
+ "family": "CRUXEval-output-choice",
1196
+ "tokens": 108,
1197
+ "candidates": 3,
1198
+ "fp16_selected_id": "option_1",
1199
+ "lut8_selected_id": "__insufficient_evidence__",
1200
+ "selection_agrees": false,
1201
+ "abstention_agrees": false,
1202
+ "max_probability_error": 0.018939317187460836
1203
+ },
1204
+ {
1205
+ "id": "CRUXEval-output-choice:test:sample_46",
1206
+ "family": "CRUXEval-output-choice",
1207
+ "tokens": 122,
1208
+ "candidates": 4,
1209
+ "fp16_selected_id": "option_0",
1210
+ "lut8_selected_id": "option_0",
1211
+ "selection_agrees": true,
1212
+ "abstention_agrees": true,
1213
+ "max_probability_error": 0.005098614573274901
1214
+ },
1215
+ {
1216
+ "id": "CRUXEval-output-choice:test:sample_561",
1217
+ "family": "CRUXEval-output-choice",
1218
+ "tokens": 112,
1219
+ "candidates": 4,
1220
+ "fp16_selected_id": "option_2",
1221
+ "lut8_selected_id": "option_2",
1222
+ "selection_agrees": true,
1223
+ "abstention_agrees": true,
1224
+ "max_probability_error": 0.0024087652013133898
1225
+ },
1226
+ {
1227
+ "id": "CRUXEval-output-choice:test:sample_73",
1228
+ "family": "CRUXEval-output-choice",
1229
+ "tokens": 125,
1230
+ "candidates": 7,
1231
+ "fp16_selected_id": "__insufficient_evidence__",
1232
+ "lut8_selected_id": "__insufficient_evidence__",
1233
+ "selection_agrees": true,
1234
+ "abstention_agrees": true,
1235
+ "max_probability_error": 0.00522251950075503
1236
+ },
1237
+ {
1238
+ "id": "CRUXEval-output-choice:test:sample_513",
1239
+ "family": "CRUXEval-output-choice",
1240
+ "tokens": 120,
1241
+ "candidates": 3,
1242
+ "fp16_selected_id": "option_0",
1243
+ "lut8_selected_id": "option_0",
1244
+ "selection_agrees": true,
1245
+ "abstention_agrees": true,
1246
+ "max_probability_error": 0.005722456403986886
1247
+ }
1248
+ ],
1249
+ "passed": false,
1250
+ "suite_file": "selected-rows.jsonl.gz"
1251
+ }
verification-verdict_lut8_kmeans_per_tensor_L128_candidates25.json ADDED
@@ -0,0 +1,177 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "source_repo": "heman10x/rlcd-modernbert-151m",
3
+ "source_revision": "8af2496eb63c7fa66d7d234e1f62629380030eb4",
4
+ "package": "verdict_lut8_kmeans_per_tensor_L128_candidates25.mlpackage",
5
+ "chip": "arm",
6
+ "macos": "27.0",
7
+ "max_calibrated_probability_error": 0.007735883267635635,
8
+ "selected_id_agreements": 4,
9
+ "questions": 4,
10
+ "rows": [
11
+ {
12
+ "name": "lost card",
13
+ "type": "choice",
14
+ "tokens": 53,
15
+ "candidates_including_abstention": 3,
16
+ "native": {
17
+ "type": "choice",
18
+ "selected_id": "card_lost",
19
+ "selected_probability": 0.627995456676331,
20
+ "probabilities": {
21
+ "card_lost": 0.627995456676331,
22
+ "pin_reset": 0.12699368118294335,
23
+ "__insufficient_evidence__": 0.2450108621407256
24
+ },
25
+ "concentration": 0.18185863532617996,
26
+ "is_abstention": false,
27
+ "p_abstain": 0.2450108621407256,
28
+ "calibration_status": "calibrated_for_scope",
29
+ "choice": "card_lost"
30
+ },
31
+ "coreml": {
32
+ "type": "choice",
33
+ "selected_id": "card_lost",
34
+ "selected_probability": 0.6284151929178529,
35
+ "probabilities": {
36
+ "card_lost": 0.6284151929178529,
37
+ "pin_reset": 0.12392298940523938,
38
+ "__insufficient_evidence__": 0.24766181767690781
39
+ },
40
+ "concentration": 0.1841022638089621,
41
+ "is_abstention": false,
42
+ "p_abstain": 0.24766181767690781,
43
+ "calibration_status": "calibrated_for_scope",
44
+ "choice": "card_lost"
45
+ },
46
+ "max_calibrated_probability_error": 0.003070691777703971,
47
+ "selected_id_agrees": true,
48
+ "median_model_call_ms": 3.5945209965575486
49
+ },
50
+ {
51
+ "name": "binary inquiry",
52
+ "type": "noul",
53
+ "tokens": 50,
54
+ "candidates_including_abstention": 3,
55
+ "native": {
56
+ "type": "noul",
57
+ "selected_id": "true",
58
+ "selected_probability": 0.46928461522757403,
59
+ "probabilities": {
60
+ "true": 0.46928461522757403,
61
+ "false": 0.24040802067067693,
62
+ "__insufficient_evidence__": 0.29030736410174907
63
+ },
64
+ "concentration": 0.03808313508410699,
65
+ "is_abstention": false,
66
+ "p_abstain": 0.29030736410174907,
67
+ "calibration_status": "calibrated_for_scope",
68
+ "noul": 0.6612505068952915,
69
+ "selected_outcome": "true"
70
+ },
71
+ "coreml": {
72
+ "type": "noul",
73
+ "selected_id": "true",
74
+ "selected_probability": 0.4674474630415731,
75
+ "probabilities": {
76
+ "true": 0.4674474630415731,
77
+ "false": 0.23674314884825226,
78
+ "__insufficient_evidence__": 0.2958093881101746
79
+ },
80
+ "concentration": 0.03798516343649183,
81
+ "is_abstention": false,
82
+ "p_abstain": 0.2958093881101746,
83
+ "calibration_status": "calibrated_for_scope",
84
+ "noul": 0.6638081439158804,
85
+ "selected_outcome": "true"
86
+ },
87
+ "max_calibrated_probability_error": 0.005502024008425532,
88
+ "selected_id_agrees": true,
89
+ "median_model_call_ms": 3.593437490053475
90
+ },
91
+ {
92
+ "name": "urgency score",
93
+ "type": "score",
94
+ "tokens": 55,
95
+ "candidates_including_abstention": 4,
96
+ "native": {
97
+ "type": "score",
98
+ "selected_id": "__insufficient_evidence__",
99
+ "selected_probability": 0.3259147064757007,
100
+ "probabilities": {
101
+ "low": 0.1979320574725608,
102
+ "medium": 0.289823573347529,
103
+ "high": 0.18632966270420953,
104
+ "__insufficient_evidence__": 0.3259147064757007
105
+ },
106
+ "concentration": 0.02039159311536931,
107
+ "is_abstention": true,
108
+ "p_abstain": 0.3259147064757007,
109
+ "calibration_status": "calibrated_for_scope",
110
+ "score": null,
111
+ "selected_level_id": "__insufficient_evidence__",
112
+ "selected_value": null
113
+ },
114
+ "coreml": {
115
+ "type": "score",
116
+ "selected_id": "__insufficient_evidence__",
117
+ "selected_probability": 0.33365058974333633,
118
+ "probabilities": {
119
+ "low": 0.19246977875904386,
120
+ "medium": 0.28663522763043275,
121
+ "high": 0.18724440386718702,
122
+ "__insufficient_evidence__": 0.33365058974333633
123
+ },
124
+ "concentration": 0.02239250899609624,
125
+ "is_abstention": true,
126
+ "p_abstain": 0.33365058974333633,
127
+ "calibration_status": "calibrated_for_scope",
128
+ "score": null,
129
+ "selected_level_id": "__insufficient_evidence__",
130
+ "selected_value": null
131
+ },
132
+ "max_calibrated_probability_error": 0.007735883267635635,
133
+ "selected_id_agrees": true,
134
+ "median_model_call_ms": 3.6040834966115654
135
+ },
136
+ {
137
+ "name": "insufficient evidence",
138
+ "type": "choice",
139
+ "tokens": 41,
140
+ "candidates_including_abstention": 3,
141
+ "native": {
142
+ "type": "choice",
143
+ "selected_id": "__insufficient_evidence__",
144
+ "selected_probability": 0.6003394036234859,
145
+ "probabilities": {
146
+ "visa": 0.19581438090218925,
147
+ "mastercard": 0.20384621547432477,
148
+ "__insufficient_evidence__": 0.6003394036234859
149
+ },
150
+ "concentration": 0.1354395684855455,
151
+ "is_abstention": true,
152
+ "p_abstain": 0.6003394036234859,
153
+ "calibration_status": "calibrated_for_scope",
154
+ "choice": "__insufficient_evidence__"
155
+ },
156
+ "coreml": {
157
+ "type": "choice",
158
+ "selected_id": "__insufficient_evidence__",
159
+ "selected_probability": 0.5987396695679329,
160
+ "probabilities": {
161
+ "visa": 0.19468390741911423,
162
+ "mastercard": 0.20657642301295281,
163
+ "__insufficient_evidence__": 0.5987396695679329
164
+ },
165
+ "concentration": 0.13392959915620617,
166
+ "is_abstention": true,
167
+ "p_abstain": 0.5987396695679329,
168
+ "calibration_status": "calibrated_for_scope",
169
+ "choice": "__insufficient_evidence__"
170
+ },
171
+ "max_calibrated_probability_error": 0.0027302075386280433,
172
+ "selected_id_agrees": true,
173
+ "median_model_call_ms": 3.581458469852805
174
+ }
175
+ ],
176
+ "passed": true
177
+ }
verify-compression.py ADDED
@@ -0,0 +1,165 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Measure Verdict LUT8 against FP16 on fixed Decision Index requests without using gold labels."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import argparse
6
+ import gzip
7
+ import json
8
+ import platform
9
+ import statistics
10
+ import subprocess
11
+ import time
12
+ from collections import Counter
13
+ from pathlib import Path
14
+
15
+ import coremltools as ct
16
+ import numpy as np
17
+ from transformers import AutoTokenizer
18
+
19
+ from assets import ROOT, sha256, verify_assets
20
+ from decision_index_engine import adapt_question, as_text
21
+ from native_reference import build_request, decode, load_calibrator
22
+ from preprocessing import prepare
23
+
24
+
25
+ def selected_requests(rows_path: Path, tokenizer, limit: int, length: int):
26
+ """First ten eligible rows per family, before inspecting either model's output."""
27
+ family_counts = Counter()
28
+ skipped = Counter()
29
+ selected = []
30
+ with gzip.open(rows_path, "rt") as stream:
31
+ for line in stream:
32
+ row = json.loads(line)
33
+ family = row["family"]
34
+ if family_counts[family] >= 10:
35
+ continue
36
+ if len(row["questions"]) != 1:
37
+ skipped["multiple_questions"] += 1
38
+ continue
39
+ question = next(iter(row["questions"].values()))
40
+ if question["type"] not in ("choice", "noul"):
41
+ skipped["unsupported_question_type"] += 1
42
+ continue
43
+ try:
44
+ request = build_request(as_text(row["state"]), adapt_question(question))
45
+ except ValueError:
46
+ skipped["invalid_or_overcapacity"] += 1
47
+ continue
48
+ tokens = len(tokenizer(request.text, truncation=False)["input_ids"])
49
+ if tokens > length:
50
+ skipped["overlength"] += 1
51
+ continue
52
+ selected.append((row["id"], family, request, tokens))
53
+ family_counts[family] += 1
54
+ if len(selected) >= limit:
55
+ break
56
+ return selected, family_counts, skipped
57
+
58
+
59
+ def main() -> None:
60
+ parser = argparse.ArgumentParser(description=__doc__)
61
+ parser.add_argument("--rows", type=Path, required=True, help="pinned public Decision Index selected-rows.jsonl.gz")
62
+ parser.add_argument("--limit", type=int, default=100)
63
+ parser.add_argument("--length", type=int, default=128)
64
+ parser.add_argument("--repeats", type=int, default=5)
65
+ args = parser.parse_args()
66
+ if args.length != 128:
67
+ raise ValueError("the predeclared LUT8 validation protocol is L128 only")
68
+ if args.limit != 100:
69
+ raise ValueError("the predeclared validation manifest uses exactly 100 requests")
70
+ source = verify_assets(required=("config.json", "tokenizer.json", "tokenizer_config.json", "calibrator.json"))
71
+ tokenizer = AutoTokenizer.from_pretrained(source)
72
+ config = json.loads((source / "config.json").read_text())
73
+ calibrator = load_calibrator(source)
74
+ selected, families, skipped = selected_requests(args.rows, tokenizer, args.limit, args.length)
75
+ if len(selected) != args.limit:
76
+ raise ValueError(f"only {len(selected)} eligible fixed requests; expected {args.limit}")
77
+
78
+ packages = {
79
+ "fp16": ROOT / "build" / "verdict_fp16_L128_candidates25.mlpackage",
80
+ "lut8": ROOT / "build" / "verdict_lut8_kmeans_per_tensor_L128_candidates25.mlpackage",
81
+ }
82
+ models = {name: ct.models.MLModel(str(path), compute_units=ct.ComputeUnit.ALL) for name, path in packages.items()}
83
+ for _, _, request, _ in selected[:2]:
84
+ arrays = prepare(tokenizer, config["class_token_index"], request.text, args.length, 25)
85
+ for model in models.values():
86
+ model.predict(arrays)
87
+
88
+ rows = []
89
+ timings = {name: [] for name in models}
90
+ for index, (row_id, family, request, tokens) in enumerate(selected):
91
+ arrays = prepare(tokenizer, config["class_token_index"], request.text, args.length, 25)
92
+ results = {}
93
+ for name, model in models.items():
94
+ output = model.predict(arrays)
95
+ results[name] = decode(output["logits"], request, calibrator)
96
+ if index < 20:
97
+ for _ in range(args.repeats):
98
+ start = time.perf_counter()
99
+ model.predict(arrays)
100
+ timings[name].append((time.perf_counter() - start) * 1000)
101
+ reference = results["fp16"]
102
+ compressed = results["lut8"]
103
+ differences = [abs(reference["probabilities"][key] - compressed["probabilities"][key]) for key in request.ids]
104
+ rows.append(
105
+ {
106
+ "id": row_id,
107
+ "family": family,
108
+ "tokens": tokens,
109
+ "candidates": len(request.ids),
110
+ "fp16_selected_id": reference["selected_id"],
111
+ "lut8_selected_id": compressed["selected_id"],
112
+ "selection_agrees": reference["selected_id"] == compressed["selected_id"],
113
+ "abstention_agrees": reference["is_abstention"] == compressed["is_abstention"],
114
+ "max_probability_error": max(differences),
115
+ }
116
+ )
117
+ errors = np.array([row["max_probability_error"] for row in rows])
118
+ selection_agreement = sum(row["selection_agrees"] for row in rows) / len(rows)
119
+ abstention_agreement = sum(row["abstention_agrees"] for row in rows) / len(rows)
120
+ gates = {
121
+ "min_selection_agreement": 0.99,
122
+ "min_abstention_agreement": 0.99,
123
+ "max_p95_probability_error": 0.02,
124
+ "max_worst_probability_error": 0.10,
125
+ }
126
+ report = {
127
+ "suite_file": args.rows.name,
128
+ "suite_sha256": sha256(args.rows),
129
+ "selection_protocol": "First 10 eligible rows per family in suite order, 100 total; no gold labels used",
130
+ "selected_row_ids": [row["id"] for row in rows],
131
+ "families": dict(families),
132
+ "skipped_before_limit": dict(skipped),
133
+ "packages": {name: path.name for name, path in packages.items()},
134
+ "hardware": {
135
+ "chip": subprocess.run(
136
+ ["sysctl", "-n", "machdep.cpu.brand_string"], capture_output=True, text=True
137
+ ).stdout.strip(),
138
+ "macos": platform.mac_ver()[0],
139
+ },
140
+ "questions": len(rows),
141
+ "selection_agreement": selection_agreement,
142
+ "abstention_agreement": abstention_agreement,
143
+ "p95_probability_error": float(np.percentile(errors, 95)),
144
+ "worst_probability_error": float(errors.max()),
145
+ "median_model_call_ms": {name: statistics.median(values) for name, values in timings.items()},
146
+ "gates": gates,
147
+ "rows": rows,
148
+ }
149
+ report["passed"] = (
150
+ selection_agreement >= gates["min_selection_agreement"]
151
+ and abstention_agreement >= gates["min_abstention_agreement"]
152
+ and report["p95_probability_error"] <= gates["max_p95_probability_error"]
153
+ and report["worst_probability_error"] <= gates["max_worst_probability_error"]
154
+ )
155
+ target = ROOT / "reports" / "lut8-L128-suite-parity.json"
156
+ target.write_text(json.dumps(report, indent=2) + "\n")
157
+ print(
158
+ json.dumps({key: value for key, value in report.items() if key not in ("rows", "selected_row_ids")}, indent=2)
159
+ )
160
+ if not report["passed"]:
161
+ raise SystemExit(1)
162
+
163
+
164
+ if __name__ == "__main__":
165
+ main()