doofz/systemone-rlcd

System One — calibrated decisions in one forward pass

System One is HAL-X's non-autoregressive decision engine. You give it a state (a message, an email, a ticket, a JSON record or a conversation) and a set of typed questions. It returns a typed answer to every question, each with a calibrated probability, in a single encoder forward pass: about 9 ms on one RTX 4090. It never generates text, so there is nothing to parse and nothing to hallucinate. The answer space is defined at request time, so a new schema needs no retraining.

This release adds System One AZ, a checkpoint tuned for Azerbaijani. It comes with a language router that sends every request to the right checkpoint in about 20 µs.

Azerbaijani accuracy	0.88 on MASSIVE-scenario (base multilingual: 0.40) · 0.94 on yes/no intent (0.65) · 0.72 zero-shot on SIB-200 (0.67)
Calibration	mean ECE on Azerbaijani 0.054 (base: 0.215, English checkpoint: 0.285)
Latency	8.9 ms for 1 question · 17.3 ms for 50 questions in one call (RTX 4090, p50)
Footprint	322M parameters, 0.65 GB of VRAM in bf16. All four checkpoints fit together in about 3.2 GB.
License	Apache 2.0, open weights, self-hosted

---

Checkpoints

The repository bundles four checkpoints. The router picks one per request, and only the subfolder you ask for is downloaded.

Checkpoint	Path	Backbone	Params	Context	Best at
System One AZ (new, HAL-X)	azerbaijani/	mmBERT-base	322M	1024	Azerbaijani (Latin; ASCII-typed text is also handled), intent routing, sentiment. Strong English retention.
System One EN	repo root	ModernBERT-large	421M	512	English guardrails, email triage
System One Multilingual	multilingual/	mmBERT-base	322M	1024	100+ other languages
System One Typed-Decisions	typed-decisions/	ModernBERT-large	421M	1024	four typed-decision business workflows

---

Quickstart

# the `systemone` package ships in this repo; fetch only the code, weights download lazily on first use
hf download doofz/systemone-rlcd --include "systemone/*" "pyproject.toml" "setup.py" "README.md" --local-dir systemone-src
pip install "./systemone-src[server]"

from systemone import Router

router = Router(preload=True, device="cuda")      # all checkpoints resident; routing costs ~20 µs

state = {"body": "Salam, mart ayı üçün hesabımızdan iki dəfə pul çıxılıb. "
                 "Zəhmət olmasa artıq ödənişi bu gün geri qaytarın."}

questions = {
    "department": {"type": "choice", "instructions": "Which department should handle this request?",
                   "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",
                                "sales": "pricing, new contracts", "other": "everything else"}},
    "urgency":    {"type": "score", "instructions": "How urgent is this request?",
                   "criteria": ["not urgent", "soon", "critical"]},
    "refund":     {"type": "noul",  "instructions": "Does the user explicitly request a refund?"},
}

res = router.predict(state, questions)
res["answers"]["department"]    # {'choice': 'billing', 'probabilities': {'billing': 0.979, ...}, 'confidence': ...}
res["routing"]                  # {'model': 'azerbaijani', 'reason': 'Azerbaijani (Latin script); ...'}

Questions can be written in English or in Azerbaijani, because the AZ checkpoint was trained on both. The three primitives are:

type	answer	criteria
choice	one label + a distribution over labels	{label: description} or [labels]
score	expected level $\mathbb{E}[s]$ + a distribution over levels	ordered list of level descriptions
noul	$P(\text{true})$	none needed

---

REST API

A production server ships in the package (FastAPI, bearer-token auth, every checkpoint preloaded):

SYSTEMONE_API_KEY=... CUDA_VISIBLE_DEVICES=0 python -m systemone.server --port 8095

Method	Endpoint	Purpose
GET	/health	liveness and the loaded checkpoints
GET	/v1/models	checkpoints available for routing
GET	/v1/presets	ready-made schemas: triage, email, guard, moderation
POST	/v1/route	which checkpoint a state would use (no inference)
POST	/v1/decide	{state, questions, model?, lang?} → answers + routing + latency
POST	/v1/decide/batch	{items: [{state, questions}]} → grouped by checkpoint, one batched pass each

curl -s localhost:8095/v1/decide -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{
  "state": "cox pis proqramdir, islemir",
  "questions": {"sentiment": {"type": "choice", "instructions": "Sentiment of this review?",
                              "criteria": ["positive", "negative"]}}}'
# -> {"answers": {"sentiment": {"choice": "negative", "probabilities": {"negative": 0.9997, ...}}},
#     "routing": {"model": "azerbaijani", ...}, "latency_ms": 10.4}

Why not vLLM? vLLM is built for autoregressive decoding (paged KV cache, continuous batching of generated tokens). System One generates no tokens. It is one bidirectional encoder pass, followed by a custom head that reads a score from each option's [MASK] position, and vLLM's pooling runner has no model class for that head. A single PyTorch process already answers in about 9 ms with no KV cache to manage. The throughput path is /v1/decide/batch, which packs requests into length-sorted batches.

---

How it works

Sequence construction

Every question becomes one sequence, and all the questions of a request are batched into one forward pass:

$$\underset{\text{instruction}}{\underbrace{\mathtt{\text{[CLS]}}⟨t⟩\text{question: }q\mathtt{\text{[SEP]}}}}\underset{\text{options (head budget }\le 256\text{ tok)}}{\underbrace{\mathtt{\text{[MASK]}}{o}_1\mathtt{\text{[MASK]}}{o}_2\cdots \mathtt{\text{[MASK]}}{o}_k\mathtt{\text{[SEP]}}}}\underset{\text{state}}{\underbrace{x\mathtt{\text{[SEP]}}}}\backslash underbrace\{\backslash texttt\{[CLS]\}\backslash ;\backslash langle\; t\backslash rangle\backslash ;\backslash text\{question:\; \}\; q\backslash ;\backslash texttt\{[SEP]\}\}\_\{\backslash text\{instruction\}\}\backslash ;\; \backslash underbrace\{\backslash texttt\{[MASK]\}\backslash ,o\_1\backslash ;\backslash texttt\{[MASK]\}\backslash ,o\_2\backslash ;\backslash cdots\backslash ;\backslash texttt\{[MASK]\}\backslash ,o\_k\backslash ;\backslash texttt\{[SEP]\}\}\_\{\backslash text\{options\; (head\; budget\; \}\; \backslash le\; 256\backslash text\{\; tok)\}\}\backslash ;\; \backslash underbrace\{x\backslash ;\backslash texttt\{[SEP]\}\}\_\{\backslash text\{state\}\}$$

The encoder output $H$ gets a type embedding $e_t$ added, then passes through two more transformer layers (the decision head). Each option $i$ is scored at the hidden state of its own marker $m_i$:

$$z_i=w_2^{\mathrm{\top }}\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}\text{\hspace{0.17em}⁣}(W_1\mathrm{L}\mathrm{N}({\tilde{h}}_{m_i})),\tilde{H}=\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}(H+e_t),p_i=\frac{\exp (z_i\mathrm{/}{T}_b)}{\sum _{j=1}^k\exp (z_j\mathrm{/}{T}_b)}z\_i\; =\; w\_2^\{\backslash top\}\backslash ,\backslash mathrm\{GELU\}\backslash !\backslash left(W\_1\backslash ,\backslash mathrm\{LN\}(\backslash tilde\; h\_\{m\_i\})\backslash right),\backslash qquad\; \backslash tilde\; H\; =\; \backslash mathrm\{Head\}(H\; +\; e\_t),\backslash qquad\; p\_i\; =\; \backslash frac\{\backslash exp(z\_i\; /\; T\_b)\}\{\backslash sum\_\{j=1\}^\{k\}\backslash exp(z\_j\; /\; T\_b)\}$$

Here $T_b$ is a temperature fitted per bucket of (question type, option count). Because the options are part of the input, the label set can change on every request.

RLCD objective: strictly proper scoring

The policy reports a distribution $q$ over the options and is rewarded by a strictly proper scoring rule, so reporting honest probabilities is the only way to maximise the expected reward:

$$R(q,y)=\underset{\text{log score}}{\underbrace{\log q_y}}+{\lambda }_{\text{sph}}\underset{\text{spherical}}{\underbrace{\frac{q_y}{\parallel q{\parallel }_2}}}-{\lambda }_{\text{rps}}\mathbb{1}[t=\text{score}]\underset{\text{ranked probability score}}{\underbrace{\frac{1}{k-1}\sum _{j=1}^k({\sum }_{i\le j}{q}_i-{\sum }_{i\le j}{y}_i{)}^2}}R(q,\; y)\; \backslash ;=\backslash ;\; \backslash underbrace\{\backslash log\; q\_y\}\_\{\backslash text\{log\; score\}\}\; \backslash ;+\backslash ;\; \backslash lambda\_\{\backslash text\{sph\}\}\backslash ,\backslash underbrace\{\backslash frac\{q\_y\}\{\backslash lVert\; q\backslash rVert\_2\}\}\_\{\backslash text\{spherical\}\}\; \backslash ;-\backslash ;\; \backslash lambda\_\{\backslash text\{rps\}\}\backslash ,\backslash mathbb\{1\}[t=\backslash text\{score\}]\backslash ;\backslash underbrace\{\backslash frac\{1\}\{k-1\}\backslash sum\_\{j=1\}^\{k\}\backslash Big(\backslash textstyle\backslash sum\_\{i\backslash le\; j\}\; q\_i\; -\; \backslash sum\_\{i\; \backslash le\; j\}\; y\_i\backslash Big)^\{2\}\}\_\{\backslash text\{ranked\; probability\; score\}\}$$

with $\lambda_{\text{sph}} = 0.5$ and $\lambda_{\text{rps}} = 1$. The upstream base checkpoints were trained with REINFORCE: Gaussian exploration noise on the logits, a group-mean baseline and TD($\lambda$) for multi-turn states. System One AZ maximises the same reward by direct gradient ascent against one-hot targets. That is the deterministic-gradient form of the objective and has the same maximiser, $q^{\star} = p(y \mid x)$.

Calibration and confidence

Temperatures are fitted per bucket on held-out Azerbaijani validation data by minimising the NLL, $T_b^{\star}=\arg\min_{T}\sum_{n}-\log \mathrm{softmax}(z^{(n)}/T)_{y_n}$, and clamped to $[0.5, 5]$. Calibration is reported as

$$\mathrm{E}\mathrm{C}\mathrm{E}=\sum _{b=1}^B\frac{\mathrm{\mid }{S}_b\mathrm{\mid }}{N}\mid \mathrm{a}\mathrm{c}\mathrm{c}(S_b)-\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f}(S_b)\mid ,\text{confidence}=1-\frac{H(p)}{\log k}\backslash mathrm\{ECE\}\; =\; \backslash sum\_\{b=1\}^\{B\}\backslash frac\{|S\_b|\}\{N\}\backslash ,\backslash Big|\backslash ,\backslash mathrm\{acc\}(S\_b)\; -\; \backslash mathrm\{conf\}(S\_b)\backslash ,\backslash Big|,\; \backslash qquad\; \backslash text\{confidence\}\; =\; 1\; -\; \backslash frac\{H(p)\}\{\backslash log\; k\}$$

---

Benchmarks

All numbers below were measured by HAL-X on one RTX 4090 with bf16 weights. Every checkpoint answered byte-identical questions in the same run. Questions are always written in English (the developer's schema), and the state is in the target language.

Azerbaijani task	n	EN	Multilingual	AZ (HAL-X)	Router	Split status for AZ
MASSIVE scenario, 18-way	1000	0.135	0.404	0.882	0.875	in-domain (trained on MASSIVE train)
MASSIVE intent, 20-way	1000	0.171	0.363	0.894	0.887	in-domain
Intent yes/no (noul)	1000	0.529	0.652	0.945	0.944	in-domain
App-review sentiment	1000	0.586	0.735	0.856	0.857	in-domain (review train split)
SIB-200 topic, 7-way	204	0.324	0.672	0.716	0.716	zero-shot, task never seen in training

The Router column is what Router().predict() returns on the same inputs. On Azerbaijani it matches the AZ checkpoint because 97–100% of the states are routed to it.

English task	EN	Multilingual	AZ (HAL-X)
MASSIVE scenario	0.542	0.674	0.895
MASSIVE intent@20	0.666	0.619	0.903
Intent yes/no	0.715	0.667	0.951
SIB-200 topic (zero-shot)	0.750	0.784	0.794

The AZ checkpoint was trained with 4,000 English MASSIVE replay items, so its MASSIVE-en scores are in-domain. On the held-out SIB-200 task it still edges out both upstream checkpoints, so English was not traded away for Azerbaijani.

Calibration

The base multilingual checkpoint ships with all temperatures at 1.0 and is strongly over-confident on Azerbaijani: at 85% reported confidence it is right 46% of the time. After tuning and bucket-wise temperature fitting, the AZ checkpoint tracks the diagonal. Its ECE is 0.040 over 7,948 pooled Azerbaijani decisions, against 0.223 for the base. A confidence threshold is therefore meaningful: route low-confidence cases to a human or an LLM, and act automatically on the rest.

Latency

questions per call	EN (421M)	Multilingual (322M)	AZ (322M)
1	11.2 ms	9.3 ms	8.9 ms
5	11.8 ms	9.7 ms	9.4 ms
10	12.6 ms	10.3 ms	10.1 ms
50	30.0 ms	17.1 ms	17.3 ms

Language detection in the router takes ~20 µs per request (pure Python). Peak VRAM per checkpoint is about 1.7–1.8 GB including activations.

Versus TypeSafe Jev

TypeSafe Jev is a closed, API-only decision model with the same interface idea: typed questions in, calibrated answers out. We have no Jev API access, so the Jev figures below are third-party published numbers reported by the upstream project and were not measured by HAL-X. The "System One" column covers the upstream EN and typed-decisions checkpoints, not the AZ checkpoint.

	TypeSafe Jev 1.13.0	System One
typed-decisions accuracy (2,000 decisions)	0.727	0.766 (typed-decisions checkpoint)
AG News / DAIR Emotion	0.910 / 0.480	0.950 / 0.595
Banking77 (>70 labels in one question)	0.870	0.425
Soft accuracy vs teacher distributions	0.580	0.471
p50 latency, 1 question	236–276 ms (network API)	8.9–11.2 ms (local 4090, measured here)
Azerbaijani	no published benchmark	measured above
Weights / deployment	closed API	open, on-premise

Jev still leads on very high-cardinality label sets and on matching soft teacher distributions. Latency is not an apples-to-apples comparison: Jev figures include the network round trip.

---

Router

state ──► script + language detection (~20 µs)
            ├─ Azerbaijani (ə, or ı/ş/ğ without Turkish function words, or az stop-words incl. ASCII spellings) ─► azerbaijani/
            ├─ confident English (English function words)                                                  ─► root (EN)
            ├─ other scripts / identified other languages (ru, tr, de, fr, es, hi, zh …)                   ─► multilingual/
            └─ short / unidentified Latin text  (az_first profile, default)                                ─► azerbaijani/

Routing is always overridable: router.predict(state, q, model="english") or lang="az". Set Router(az_first=False) to restore the upstream behaviour, where unidentified Latin text goes to the English checkpoint.

---

What to use it for

System One is a System 1 component. It makes the cheap, fast, high-volume decisions and leaves reasoning to a System 2 model (an LLM) only where its own confidence says it is needed.

• LLM gateway and model routing: pick the right model, tool or agent for each request in about 10 ms, and escalate to a large model only when confidence is low. This is usually the biggest cost saver.
• Guardrails and moderation: jailbreak, PII, toxicity and policy checks on every prompt and every response, in Azerbaijani, English and 100+ other languages, before the LLM sees the text.
• Support and email triage: department, urgency, churn risk and refund intent for Azerbaijani banking, telecom and e-commerce inboxes, all in one pass.
• Voice assistants and call centres: intent and slot-domain classification of ASR transcripts (the model was tuned on MASSIVE, which is spoken-assistant data).
• App-store and social listening: Azerbaijani review sentiment and complaint detection at thousands of items per second.
• Agent observability: score agent traces for risk, needs-review and outcome with calibrated probabilities you can threshold.
• Human-in-the-loop automation: because probabilities are calibrated, "act automatically when $p \ge 0.9$" actually means about 90% precision.

---

Honest limits

• Azerbaijani gains are mostly in-domain. MASSIVE and app-review scores use disjoint test splits of datasets whose train splits were used for tuning. The zero-shot signal is SIB-200 (0.672 → 0.716). Expect smaller gains on unseen Azerbaijani schemas than on the in-domain rows.
• Business-triage noul questions in Azerbaijani are not yet tuned. On an Azerbaijani billing email, department and refund come out right, but "does the user threaten to cancel?" is under-detected (P = 0.10 on an explicit threat). Fine-tune on your own labelled tickets before relying on churn-type questions.
• Ordinal score questions were not part of the AZ tuning. Their temperature bucket is unfitted (T = 1.0).
• High-cardinality choice. With more than 20–30 options, raise agent.cfg["head_max_len"] or split the options into a coarse-to-fine hierarchy.
• Very short text is ambiguous. "Super" or "ok" carries no language evidence; the az_first profile sends it to the AZ checkpoint.

---

Reproduce

Everything that produced these numbers is in this repo:

file	purpose
systemone/	the Python package: Agent, Router, language detection, REST server
bench/tasks.py, bench/run_bench.py	benchmark suite (MASSIVE en/az, SIB-200 en/az, AZ app reviews)
bench/plots.py	the charts in this card
train/train_az.py	Azerbaijani fine-tune: 25k items, 3 epochs, 3.9 min on one RTX 4090, embeddings frozen, 8-bit AdamW
results/*.json	raw benchmark outputs

Data: MASSIVE (az-AZ, en-US), SIB-200 (azj_Latn, eng_Latn), Azerbaijani app reviews (1–2★ vs 5★, balanced).

---

License and attribution

Apache 2.0. System One is developed by HAL-X. The EN, Multilingual and Typed-Decisions checkpoints and the core runtime are derived from Laya by Convai Innovations (Apache 2.0). The Azerbaijani checkpoint, the Azerbaijani-aware router, the batched runtime, the REST server and all benchmarks in this card are HAL-X work. See NOTICE.

@misc{halx2026systemone,
  title  = {System One: Calibrated Non-Autoregressive Decisions for Azerbaijani and 100+ Languages},
  author = {HAL-X},
  year   = {2026},
  url    = {https://huggingface.co/doofz/systemone-rlcd}
}