soft-decider-421m

An RLCD fine-tune of convaiinnovations/laya (ModernBERT-large, 421M) on the LocalLLaMA/typed-decisions train split — a non-generative System-1 decision model: give it a state (JSON/text) plus typed choice / score / noul questions, get calibrated probability distributions back in one forward pass (~50 ms, GPU).

What makes this checkpoint different from the published laya-typed-decisions is not raw accuracy — it is calibration discipline. The upstream checkpoint ships temperatures fitted on data it had already trained on (NandhaKishorM/laya#186), so its confidence numbers are honestly marked "uncalibrated" by its own authors. Here the calibration slice (400 decisions) is held out of training before any batch is formed, and both per-type and per-option-count bucket temperatures are fitted on it.

Measured — official test split (400 cases / 2,000 decisions)

Our harness reproduces laya-typed-decisions at 0.7664 against its published 0.766, so the ruler is calibrated before it measures us.

model	acc	soft acc	Brier	ECE ↓	score MAE ↓	flip ↓	auto-decidable@5%	p50 ms
soft-decider-421m	0.774	0.551	0.192	0.141	0.221	0.077	0.374	50
laya-typed-decisions (this harness)	0.766	0.500	0.213	0.214	0.243	0.065	0.378	50
TypeSafe Jev 1.13.0 (published)	0.727	0.580	0.148	0.144	0.391	–	–	710
teacher self-agreement ceiling	0.735	–	–	–	–	–	–	–

flip = share of choice decisions whose argmax changes when the option list is reshuffled. auto-decidable@5% = share of traffic a confidence gate can automate at ≤5% running error — the number that actually decides how much of an agent loop you can take off the LLM bill.

Training recipe (fully reproducible, scripts/)

Single RTX 3090, bf16, full fine-tune (not LoRA), ~17 min: RLCD objective — zero-mean Gaussian exploration on logits, reward = strictly proper scoring rule (log + spherical, w_sph=1.5, +RPS for ordinal), REINFORCE with group-mean baseline (G=8), plus soft cross-entropy against the benchmark's teacher distributions. 6 epochs, encoder LR 1.75e-5 / head LR 7e-5, cosine, held-out calibration slice as above.

Honest limits — read before deploying

• Specialist. Trained on and for the benchmark's four synthetic workflows. Outside them, expect base-checkpoint behaviour.
• Out-of-domain failures are real and confident. On 10 hand-written agent-loop scenarios (context compaction value, cost-rule batching, loop-breaking), this model and the published laya checkpoint were wrong on ~5, sometimes with high confidence — same-lineage blind spots, so ensembling the two does not fix it. Gate on confidence, route low-confidence decisions to an LLM or human tier, and consider a domain fine-tune on your own traces (scripts/train_single.py --items ...).
• Never use it as the judge for rights / legal / money release gates — those belong in deterministic rule code; a distribution is not a compliance decision.
• English states only. ~768 effective state tokens at the default 1024/256 budget; keep choice under ~20 options or raise head_max_len.
• Option-order flips are mitigated, not eliminated (7.7%): average over 2–3 orderings on hard calls.
• action.act_probability is inherited from upstream and carries no signal (#185). Use confidence.

Use

pip install laya

from huggingface_hub import snapshot_download
import laya

agent = laya.Agent(snapshot_download("winwinwinbb/soft-decider-421m"), device="cuda")
res = agent.predict(
    {"tool": "web_search", "result_summary": "docs page already used in step 5, step 9 of 20",
     "task_goal": "flag the render command"},
    {"keep": {"type": "noul", "instructions": "Is this tool result still needed for the task?"}},
)
# res["answers"]["keep"]["noul"] -> probability the result is still needed

A Jev-compatible server (same POST /v1/systemone shape as the TypeSafe API) is included at scripts/serve_soft.py; official typesafe-sdk clients work against it by setting the base URL.

Provenance

Built from convaiinnovations/laya (Apache-2.0, backbone answerdotai/ModernBERT-large) on the LocalLLaMA/typed-decisions train split (Apache-2.0). Raw per-model metrics: benchmark_report.json. Full training/eval scripts under scripts/ — same harness produced every number on this card.