Add verified Verdict Core ML L128 FP16

README.md

@@ -0,0 +1,23 @@
+---
+license: apache-2.0
+library_name: coreml
+pipeline_tag: text-classification
+base_model: heman10x/rlcd-modernbert-151m
+tags:
+  - coreml
+  - apple-silicon
+  - decision-model
+  - verdict
+---
+
+# Verdict Core ML (L128 FP16)
+
+This is a Core ML conversion of the **trained Verdict checkpoint** [`heman10x/rlcd-modernbert-151m`](https://huggingface.co/heman10x/rlcd-modernbert-151m), pinned at `8af2496eb63c7fa66d7d234e1f62629380030eb4`. It has 151,378,177 parameters and retains the full encoder plus trained decision head. The `verdict_fp16_L128_candidates25.mlpackage` file is 303,210,832 bytes and targets iOS 17 / macOS 14 or newer.
+
+Inputs are `input_ids` and `attention_mask` of shape `[1,128]`, and `class_marker_map` of shape `[1,25,128]`. The graph returns raw logits and an uncalibrated softmax. For a native Verdict decision, render the request with `native_reference.py`, append the `__insufficient_evidence__` candidate, and apply `calibrator.json` to raw logits. This means at most **24 substantive candidates**. The released calibrator has a separate temperature for several candidate counts. Calling the graph with arbitrary labels or treating `probabilities` as calibrated changes the model's behavior.
+
+Verified on Apple M5 Pro/macOS 27.0 against the pinned PyTorch model: 3/3 choice, noul and score smoke decisions agreed, with worst calibrated probability difference 0.00341. The 20-iteration `coreml-cli` median was 3.627 ms CPU+GPU, 3.710 ms CPU+ANE, 3.915 ms automatic, and 11.721 ms CPU only. These are model call times; rendering and tokenization add time. The `verification-L128.json` and `profile-L128.json` files include the details.
+
+The public Decision Index tracker reports Verdict at 13.38. Its historical checkpoint and renderer have not been authenticated against this pinned release, so this artifact does **not** claim to reproduce that score. Only the L128 bucket has been verified; larger inputs require a separately exported and tested bucket. Unsupported requests must not be truncated to 128 tokens without stating that behavior.
+
+The Hub repo includes the runtime renderer, calibration helper, tokenizer, verification report and SHA-256 asset lock. Conversion source is maintained in [FluidInference/mobius](https://github.com/FluidInference/mobius). Upstream model and code: [Verdict model](https://huggingface.co/heman10x/rlcd-modernbert-151m), [Verdict source](https://github.com/Heman10x-NGU/Verdict-open-jev), Apache-2.0. Base architecture: [`knowledgator/gliclass-modern-base-v2.0`](https://huggingface.co/knowledgator/gliclass-modern-base-v2.0).

assets.lock.json

@@ -0,0 +1,15 @@
+{
+  "source_repo": "heman10x/rlcd-modernbert-151m",
+  "source_revision": "8af2496eb63c7fa66d7d234e1f62629380030eb4",
+  "source_code": "Heman10x-NGU/Verdict-open-jev",
+  "source_code_revision_audited": "30f15564821626ca5c1ad5b2638c4eb7078787dd",
+  "license": "Apache-2.0",
+  "historical_tracker_checkpoint_verified": false,
+  "files": {
+    "model.safetensors": "d252823994d47a7933217fc86449493299643af6a0c0d83d6bd5a7666d3253ef",
+    "config.json": "303f8eef1009cfdcb0cfba3e653247e625f16a4501e2351bad1a633f1f644695",
+    "tokenizer.json": "8bb449eb0c037aae44115b65905bb339b8f3f74eb37067c19127feb3c0755723",
+    "tokenizer_config.json": "fb54f027372062b2ca52282efb04d178a8b57167a00cd8f4e816515823a2c016",
+    "calibrator.json": "af2a876993148efa0726b6ccf710fe2303897d20c0ce8c7c9036eb50f64d23de"
+  }
+}

calibrator.json

@@ -0,0 +1,41 @@
+{
+  "model_id": "openjev-modernbert-151.4m",
+  "temperature": 2.8039,
+  "log_temperature": 1.0309995577626943,
+  "scope": "open_domain_calibrated_v1",
+  "per_k": {
+    "2": 5.0069,
+    "3": 5.0069,
+    "4": 4.0314,
+    "5": 3.0560,
+    "6": 2.3898,
+    "7": 2.3898,
+    "9": 1.6668,
+    "11": 3.3919,
+    "17": 1.7200,
+    "25": 1.5144
+  },
+  "artifact_hash": "b4742a033bce3fb4707e78d910dc89d53c30656d028b1836a992ba87e0766324",
+  "format_version": "rlcd-calibrator-v1",
+  "provenance": {
+    "fitted_on": [
+      "data/real_banking_cal.jsonl",
+      "data/synthetic_calibration.jsonl",
+      "data/slice_cardinality_k3.jsonl",
+      "data/slice_cardinality_k5.jsonl",
+      "data/slice_cardinality_k9.jsonl",
+      "data/slice_cardinality_k17.jsonl",
+      "data/slice_cardinality_k25.jsonl",
+      "data/slice_distant_oos.jsonl",
+      "data/slice_missing_option.jsonl",
+      "data/gen/T01.jsonl",
+      "data/gen/T02.jsonl",
+      "data/gen/T03.jsonl",
+      "data/gen/T07.jsonl",
+      "data/gen/T12.jsonl"
+    ],
+    "total_calibration_items": 1753,
+    "date": "2026-09-20T14:00:00Z",
+    "notes": "Honest stratified temperature scaling fitted strictly via NLL minimization on held-out calibration data. Zero fitting on JevBench evaluation tasks."
+  }
+}

native_reference.py

@@ -0,0 +1,113 @@
+"""Verdict's pinned formatting, calibration and typed output contract.
+
+The rendering and decoding match Verdict-open-jev/core/{formatting,engine_encoder}.py
+at 30f15564821626ca5c1ad5b2638c4eb7078787dd. The encoder is loaded from
+heman10x/rlcd-modernbert-151m at the revision in assets.lock.json.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+ABSTAIN_ID = "__insufficient_evidence__"
+ABSTAIN_DESCRIPTION = "insufficient evidence"
+MAX_SUBSTANTIVE_OPTIONS = 24
+MAX_CANDIDATES = 25
+
+
+@dataclass(frozen=True)
+class Request:
+    kind: str
+    text: str
+    labels: tuple[str, ...]
+    ids: tuple[str, ...]
+    values: tuple[float, ...] = ()
+
+
+def build_request(context: str, question: dict[str, Any]) -> Request:
+    """Render one native typed query without dropping trained abstention."""
+    kind = question["type"]
+    if kind == "choice":
+        options = question["options"]
+        if not 1 <= len(options) <= MAX_SUBSTANTIVE_OPTIONS:
+            raise ValueError("Verdict supports 1 to 24 substantive choice options")
+        labels = tuple(f"It is {option['description']}" for option in options)
+        ids = tuple(option["id"] for option in options)
+        query = question["question"]
+        text = f"Question: {query}\n\nContext:\n{context}"
+        values: tuple[float, ...] = ()
+    elif kind == "score":
+        levels = question["levels"]
+        if not 1 <= len(levels) <= MAX_SUBSTANTIVE_OPTIONS:
+            raise ValueError("Verdict supports 1 to 24 substantive score levels")
+        labels = tuple(f"{level['description']} (Value: {level['value']})" for level in levels)
+        ids = tuple(level["id"] for level in levels)
+        values = tuple(float(level["value"]) for level in levels)
+        query = question["question"]
+        text = f"Question: {query}\n\nContext:\n{context}"
+    elif kind == "noul":
+        proposition = question["proposition"]
+        labels = (f"true: {proposition}", f"false: not {proposition}")
+        ids = ("true", "false")
+        values = ()
+        text = f"Context:\n{context}\n\nEvaluate proposition: {proposition}"
+        query = ""
+    else:
+        raise ValueError(f"unknown Verdict question type: {kind}")
+    labels += (ABSTAIN_DESCRIPTION,)
+    ids += (ABSTAIN_ID,)
+    prefix = "".join(f"<<LABEL>>{label}" for label in labels)
+    return Request(kind, f"{prefix}<<SEP>>{text}", labels, ids, values)
+
+
+def temperature(calibrator: dict[str, Any], count: int) -> float:
+    per_k = calibrator.get("per_k", {})
+    value = float(per_k.get(str(count), calibrator["temperature"]))
+    if not math.isfinite(value) or value <= 0:
+        raise ValueError("invalid trained temperature")
+    return value
+
+
+def decode(logits: np.ndarray, request: Request, calibrator: dict[str, Any]) -> dict[str, Any]:
+    """Apply the released per-K calibration and preserve native abstention."""
+    k = len(request.ids)
+    z = np.asarray(logits, dtype=np.float64).reshape(-1)[:k]
+    if len(z) != k or not np.isfinite(z).all():
+        raise ValueError("invalid Verdict logits")
+    z = z / temperature(calibrator, k)
+    p = np.exp(z - z.max())
+    p /= p.sum()
+    distribution = dict(zip(request.ids, map(float, p)))
+    selected_id = request.ids[int(p.argmax())]
+    abstained = selected_id == ABSTAIN_ID
+    result: dict[str, Any] = {
+        "type": request.kind,
+        "selected_id": selected_id,
+        "probabilities": distribution,
+        "is_abstention": abstained,
+        "p_abstain": distribution[ABSTAIN_ID],
+        "calibration_status": "calibrated_for_scope",
+    }
+    if request.kind == "choice":
+        result["choice"] = selected_id
+    elif request.kind == "noul":
+        substantive = distribution["true"] + distribution["false"]
+        result["noul"] = distribution["true"] / substantive if substantive and not abstained else None
+    else:
+        substantive = sum(distribution[key] for key in request.ids[:-1])
+        result["score"] = (
+            sum(value * distribution[key] / substantive for key, value in zip(request.ids[:-1], request.values))
+            if substantive and not abstained
+            else None
+        )
+    return result
+
+
+def load_calibrator(source: Path) -> dict[str, Any]:
+    return json.loads((source / "calibrator.json").read_text())

preprocessing.py

@@ -0,0 +1,24 @@
+"""Tensorize Verdict's native rendered text for one fixed Core ML bucket."""
+
+from __future__ import annotations
+
+import numpy as np
+
+
+def prepare(tokenizer, model, rendered: str, length: int, max_candidates: int) -> dict[str, np.ndarray]:
+    full = tokenizer(rendered, truncation=False)
+    if len(full["input_ids"]) > length:
+        raise ValueError(f"Verdict prompt needs {len(full['input_ids'])} tokens; L{length} has no room")
+    encoded = tokenizer(rendered, truncation=False, padding="max_length", max_length=length, return_tensors="np")
+    ids = encoded["input_ids"].astype(np.int32)
+    positions = np.flatnonzero(ids[0] == model.config.class_token_index)
+    if len(positions) > max_candidates:
+        raise ValueError("candidate markers exceed exported head capacity")
+    markers = np.zeros((1, max_candidates, length), dtype=np.float32)
+    for row, position in enumerate(positions):
+        markers[0, row, position] = 1.0
+    return {
+        "input_ids": ids,
+        "attention_mask": encoded["attention_mask"].astype(np.int32),
+        "class_marker_map": markers,
+    }

profile-L128.json

@@ -0,0 +1,91 @@
+{
+  "hardware": {
+    "device": "arm64",
+    "chip": "Apple M5 Pro",
+    "ram": "24GB",
+    "os_version": "macOS 27.0",
+    "timestamp": "2026-09-22T20:35:07.289343+00:00"
+  },
+  "models": [
+    {
+      "model_path": "/var/folders/hg/l0263vy543b3r7rfwgxkt_680000gn/T/verdict_fp16_L128_candidates25.mlmodelc",
+      "model_name": "verdict_fp16_L128_candidates25",
+      "metadata": {
+        "description": "Verdict decision logits; use abstention and per-K calibration in host",
+        "author": "Heman10x-NGU (Verdict weights); Fluid Inference (Core ML conversion)",
+        "license": "Apache-2.0"
+      },
+      "cold_compile_ms": 4181.106,
+      "results": [
+        {
+          "compute_units": "all",
+          "summary": {
+            "cpu_percent": 31.73,
+            "gpu_percent": 0.0,
+            "ane_percent": 68.27
+          },
+          "latency": {
+            "compile_ms": 28.336,
+            "median_ms": 3.915,
+            "mean_ms": 4.016,
+            "min_ms": 3.701,
+            "max_ms": 5.103,
+            "std_ms": 0.331,
+            "iterations": 20
+          }
+        },
+        {
+          "compute_units": "cpu_only",
+          "summary": {
+            "cpu_percent": 100.0,
+            "gpu_percent": 0.0,
+            "ane_percent": 0.0
+          },
+          "latency": {
+            "compile_ms": 0.414,
+            "median_ms": 11.721,
+            "mean_ms": 11.715,
+            "min_ms": 11.541,
+            "max_ms": 11.985,
+            "std_ms": 0.104,
+            "iterations": 20
+          }
+        },
+        {
+          "compute_units": "cpu_and_gpu",
+          "summary": {
+            "cpu_percent": 0.0,
+            "gpu_percent": 100.0,
+            "ane_percent": 0.0
+          },
+          "latency": {
+            "compile_ms": 1.007,
+            "median_ms": 3.627,
+            "mean_ms": 3.651,
+            "min_ms": 3.502,
+            "max_ms": 3.896,
+            "std_ms": 0.116,
+            "iterations": 20
+          }
+        },
+        {
+          "compute_units": "cpu_and_neural_engine",
+          "summary": {
+            "cpu_percent": 31.73,
+            "gpu_percent": 0.0,
+            "ane_percent": 68.27
+          },
+          "latency": {
+            "compile_ms": 23.874,
+            "median_ms": 3.71,
+            "mean_ms": 3.739,
+            "min_ms": 3.574,
+            "max_ms": 4.226,
+            "std_ms": 0.153,
+            "iterations": 20
+          }
+        }
+      ]
+    }
+  ]
+}

tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}

verdict_fp16_L128_candidates25.conversion.json

@@ -0,0 +1,15 @@
+{
+  "package": "verdict_fp16_L128_candidates25.mlpackage",
+  "source_repo": "heman10x/rlcd-modernbert-151m",
+  "source_revision": "8af2496eb63c7fa66d7d234e1f62629380030eb4",
+  "source_weight_sha256": "d252823994d47a7933217fc86449493299643af6a0c0d83d6bd5a7666d3253ef",
+  "parameters": 151378177,
+  "wrapper_max_logit_error": 1.9073486328125e-06,
+  "conversion_seconds": 5.913961540965829,
+  "package_bytes": 303210832,
+  "package_files_sha256": {
+    "Manifest.json": "fc5d31df174a9f450bfb9ba406307c379c30fabba5cad38fcaf8717b806c69a8",
+    "Data/com.apple.CoreML/model.mlmodel": "6cc079930a63c1069a7a2c259f4c086879981021047767f2bb4d68b6d1f02b00",
+    "Data/com.apple.CoreML/weights/weight.bin": "a982e14147982df9abb99a3637075a0193835c6b99edc396a52c1a43c2a4854a"
+  }
+}

verdict_fp16_L128_candidates25.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6cc079930a63c1069a7a2c259f4c086879981021047767f2bb4d68b6d1f02b00
+size 306983

verdict_fp16_L128_candidates25.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a982e14147982df9abb99a3637075a0193835c6b99edc396a52c1a43c2a4854a
+size 302903232

verdict_fp16_L128_candidates25.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "82619DAC-DE6A-49C3-BFE9-4510CA1EB9B1": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        },
+        "D522AC1C-3AF7-4ADE-936E-9D7D8A090D98": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        }
+    },
+    "rootModelIdentifier": "D522AC1C-3AF7-4ADE-936E-9D7D8A090D98"
+}

verification-L128.json

@@ -0,0 +1,120 @@
+{
+  "source_repo": "heman10x/rlcd-modernbert-151m",
+  "source_revision": "8af2496eb63c7fa66d7d234e1f62629380030eb4",
+  "package": "/Users/hanweng/Documents/mobius-verdict-laya/models/computer-use/verdict/coreml/build/verdict_fp16_L128_candidates25.mlpackage",
+  "chip": "arm",
+  "macos": "27.0",
+  "max_calibrated_probability_error": 0.003407791923494219,
+  "selected_id_agreements": 3,
+  "questions": 3,
+  "rows": [
+    {
+      "name": "lost card",
+      "type": "choice",
+      "tokens": 53,
+      "candidates_including_abstention": 3,
+      "native": {
+        "type": "choice",
+        "selected_id": "card_lost",
+        "probabilities": {
+          "card_lost": 0.627995456676331,
+          "pin_reset": 0.12699368118294335,
+          "__insufficient_evidence__": 0.2450108621407256
+        },
+        "is_abstention": false,
+        "p_abstain": 0.2450108621407256,
+        "calibration_status": "calibrated_for_scope",
+        "choice": "card_lost"
+      },
+      "coreml": {
+        "type": "choice",
+        "selected_id": "card_lost",
+        "probabilities": {
+          "card_lost": 0.6283802556009165,
+          "pin_reset": 0.1261597182435837,
+          "__insufficient_evidence__": 0.24546002615549983
+        },
+        "is_abstention": false,
+        "p_abstain": 0.24546002615549983,
+        "calibration_status": "calibrated_for_scope",
+        "choice": "card_lost"
+      },
+      "max_calibrated_probability_error": 0.0008339629393596393,
+      "selected_id_agrees": true,
+      "median_model_call_ms": 3.475604491541162
+    },
+    {
+      "name": "binary inquiry",
+      "type": "noul",
+      "tokens": 50,
+      "candidates_including_abstention": 3,
+      "native": {
+        "type": "noul",
+        "selected_id": "true",
+        "probabilities": {
+          "true": 0.46928461522757403,
+          "false": 0.24040802067067693,
+          "__insufficient_evidence__": 0.29030736410174907
+        },
+        "is_abstention": false,
+        "p_abstain": 0.29030736410174907,
+        "calibration_status": "calibrated_for_scope",
+        "noul": 0.6612505068952915
+      },
+      "coreml": {
+        "type": "noul",
+        "selected_id": "true",
+        "probabilities": {
+          "true": 0.4658768233040798,
+          "false": 0.2409705632084786,
+          "__insufficient_evidence__": 0.29315261348744165
+        },
+        "is_abstention": false,
+        "p_abstain": 0.29315261348744165,
+        "calibration_status": "calibrated_for_scope",
+        "noul": 0.6590911025400003
+      },
+      "max_calibrated_probability_error": 0.003407791923494219,
+      "selected_id_agrees": true,
+      "median_model_call_ms": 3.5811454872600734
+    },
+    {
+      "name": "urgency score",
+      "type": "score",
+      "tokens": 55,
+      "candidates_including_abstention": 4,
+      "native": {
+        "type": "score",
+        "selected_id": "__insufficient_evidence__",
+        "probabilities": {
+          "low": 0.1979320574725608,
+          "medium": 0.289823573347529,
+          "high": 0.18632966270420953,
+          "__insufficient_evidence__": 0.3259147064757007
+        },
+        "is_abstention": true,
+        "p_abstain": 0.3259147064757007,
+        "calibration_status": "calibrated_for_scope",
+        "score": null
+      },
+      "coreml": {
+        "type": "score",
+        "selected_id": "__insufficient_evidence__",
+        "probabilities": {
+          "low": 0.19782407464261623,
+          "medium": 0.2888377991986832,
+          "high": 0.18659805748489713,
+          "__insufficient_evidence__": 0.32674006867380345
+        },
+        "is_abstention": true,
+        "p_abstain": 0.32674006867380345,
+        "calibration_status": "calibrated_for_scope",
+        "score": null
+      },
+      "max_calibrated_probability_error": 0.0009857741488458194,
+      "selected_id_agrees": true,
+      "median_model_call_ms": 3.4559165069367737
+    }
+  ],
+  "passed": true
+}