LemonDecider 4B

Model ID: lemondecider-4b-20261015-v2 Base model: Qwen/Qwen3.5-4B

LemonDecider 4B is a typed decision engine: it reads a state and a question with a list of options, runs one forward pass, and returns a probability distribution over the options (choice / noul questions) or an ordered rating (score questions). No text is generated and no input is ever truncated.

Model description

Property	Value
Architecture	Qwen3.5-4B-class hybrid backbone (linear-attention + periodic full-attention) + a bias-free 255-way readout (LemonDeciderReadout, FP32 precision island), loaded through modeling_lemondecider.LemonDeciderForDecisionClassification (trust_remote_code)
model_type / protocol	lemondecider_decision / lemondecider-v1
Attention	noncausal_full_attention — all-attend prompt for the periodic full-attention layers, causal recurrence for the Gated DeltaNet layers, SDPA only
Pooling / temperature	last-token pooling, temperature 1.0 (uncalibrated)
Max input length	262,144 tokens (256K context) — the backbone's full position window; requests over the limit are rejected, never cut
Precision	backbone BF16, readout FP32 (readout GEMM executed in FP32 under autocast)
Serialized tensors	724 (all vision tensors included; vision stem executed as a matrix-product patch projection)
Runtime	in-package Python runtime (modeling_lemondecider.py, configuration_lemondecider.py, contract.py, decision_utils.py, model.py), validated with transformers 5.17, flash-linear-attention 0.5.2, PyTorch SDPA

Training

LemonDecider 4B was fine-tuned (SFT) from the common Qwen3.5-4B initialization on more than 700,000 public decision/QA rows collected by the project (about 805k train rows from ~818k collected rows incl. dev/test splits). Every source is public and permissive; no benchmark or evaluation data was used for training. The released weights combine two independent SFT runs over this corpus (weight-level combination, no training after that).

Question formats: choice (up to 255 options), noul (no/yes) and score (ordinal ratings); some multimodal rows are image questions over locally registered project-generated charts, and roughly 600k train rows carry teacher-style soft label distributions.

Data categories:

• service decision arcs — user-intent routing and case triage (refund, logistics, account actions), policy and adequacy judgment;
• language judgment — sentiment, sarcasm, NLI/factuality and paraphrase-style checks;
• knowledge and reasoning QA — multiple-choice knowledge, multi-hop reasoning, arithmetic and math word problems;
• retrieval and selection — document and tabular retrieval, product/route selection;
• multimodal reading — chart/graph and small-graphic question answering;
• calibration-style rows — probability and ordinal confidence judgments.

Training parameters (from the training receipts):

• distributed FSDP on 64 GPUs (4 nodes x 16 GPUs);
• effective batch 256 rows per step, learning rate 2e-06;
• two independent runs: 2,446 steps (sequence cap 8,192) and 700 steps (sequence cap 2,048);
• objective valid_option_soft_cross_entropy over valid options only (noul_tie_policy = true_on_equal_probability, choice_permutation = global-reference, vision tower trainable);
• precision policy fp32-master-bf16; export dtype BF16, readout FP32, temperature 1.0.

Package artifacts

Weight files

File	Size	SHA-256
model-00001-of-00002.safetensors	4,973,336,472	27b985a94ccf63aace88b2ff67c1ea0871ce745a49b22b9cda43ea4d32297d8a
model-00002-of-00002.safetensors	4,107,897,400	81f61ce9daead4be7e3d950cf027f63f16f757a176407f020406190dc2068bc0
tokenizer.json	19,989,492	6f32ce20dc35f57a7f9ad1eac03525bd7d30f9df8cea6507e958279cc3657706
model.safetensors.index.json	68,426	bda09c55c0152d7a6fb2db6898aea70e728dc528df6f9c026cefb0bd5d1d673b

Evaluation results

Both evaluations ran on 2026-10-15 on self-hosted compute (64 GPUs) through the model's own HTTP /v1/systemone wire contract. No result here is an official leaderboard or board submission; the harnesses enforce no-truncation, no option filtering, one fixed rendering, and coverage-adjusted scoring. Latency is per-request server wall time on internal accelerators and is not the sub-1-second latency-admission measurement of any external board.

Evidence directories keep the run names under which they were created during the v1 execution (lemondeeder-eval-20261015/); no shipped artifact in the model packages carries that legacy spelling.

1) Decision Index, edition 0.3 — FULL public suite (complete run)

⚙️ 140,620 requests issued across 64 shards (frozen suite-0.3 staging, official scorer, coverage-adjusted, chance-corrected) → 140,178 scored requests, complete: true.

Headline	Value
Decision Index (public 0.3)	50.31
Raw index	62.47
Median / mean / p95 request latency	50.4 ms / 205.6 ms / 756.2 ms

The public index averages the native, chance-corrected, coverage-adjusted score of 37 benchmarks in five areas with fixed weights (knowledge 25.8%, language 25.8%, retrieval 20.0%, tools 18.3%, arts 10%); inside an area, ★ gold benchmarks weigh 1.2 and the rest 1.0. Unanswered counts as wrong.

Per-area scores

Area	Raw	Skill (chance-corrected)	Coverage	Benchmarks
Knowledge & Reasoning	0.4895	0.3395	0.9998	11
Language Understanding	0.6962	0.5751	1.0000	9
Retrieval & Classification	0.6661	0.5494	1.0000	6
Tools & Automation	0.7108	0.6741	0.9986	5
Arts & Human Taste	0.5487	0.3340	0.9978	6

Per-benchmark native scores (★ = counted in the public index)

#	Benchmark	Metric	Requests	Score	Skill	Cov.
1	BFCL ★	case exact accuracy	1,694	0.9416	0.9211	1.000
2	ToolRet ★	nDCG@10	685	0.6540	0.5982	0.997
3	API-Bank ★	accuracy	508	0.8320	0.8254	0.996
4	BANKING77 ★	macro-F1	3,080	0.7418	0.7385	1.000
5	CLINC150+OOS ★	macro-F1	5,500	0.8634	0.8626	1.000
6	RouterBench	selected quality	10,000	0.7746	0.4715	1.000
9	Home appliance simulator ★	case exact accuracy	88	0.4091	0.4091	1.000
10	SGD/SGD-X	macro-F1	2,500	0.4360	0.0616	1.000
11	ContractNLI ★	macro-F1	123	0.7389	0.6224	1.000
12	ANLI ★	macro-F1	3,200	0.6729	0.5100	1.000
20	BPoMP ★	accuracy	5,000	0.8718	0.7525	1.000
21	Humicroedit ★	accuracy	2,628	0.6111	0.2222	1.000
22	POP909-CL ★	accuracy	2,000	0.1890	0.1701	0.987
23	cfcolor ★	accuracy	5,000	0.6088	0.1827	1.000
24	MMLU	accuracy	14,033	0.7914	0.7219	1.000
25	GPQA Diamond ★	accuracy	196	0.4286	0.2381	1.000
26	ARC-Easy	accuracy	2,376	0.9836	0.9781	1.000
27	ARC-Challenge	accuracy	1,172	0.9548	0.9397	1.000
28	WinoGrande ★	accuracy	1,267	0.8019	0.6038	1.000
29	HellaSwag ★	accuracy	10,042	0.8520	0.8027	1.000
30	GSM8K ★	accuracy	2,638	0.5716	0.4866	1.000
31	ChessBench ★	accuracy	5,000	0.1384	0.0616	1.000
32	MuSR ★	accuracy	752	0.5731	0.3213	1.000
33	SATA-Bench ★	case exact accuracy	1,650	0.1539	0.1427	1.000
34	SimpleBench	accuracy	10	0.1000	0.0000	1.000
36	BRIGHT ★	nDCG@10	220	0.4444	0.3714	1.000
37	Amazon ESCI ★	macro-F1	5,000	0.4827	0.3511	1.000
38	ACOS ★	per-review F1	1,565	0.1970	0.1713	1.000
39	FinEntity ★	macro-F1	979	0.9381	0.9090	1.000
40	iSarcasmEval ★	Sarcasm F1 (track A, EN)	4,600	0.5552	0.4278	1.000
41	VAST ★	macro-F1	3,006	0.7251	0.5876	1.000
42	NLI4CT ★	macro-F1	5,500	0.7997	0.6103	1.000
43	CRUXEval ★	accuracy	570	0.6035	0.3709	1.000
44	CLadder ★	accuracy	5,000	0.6846	0.3692	1.000
45	HLE ★	accuracy	501	0.1140	0.0000	0.998
50	Habermas Machine ★	accuracy	1,676	0.4063	0.1384	1.000
56	PhishNChips ★	phishing-decision accuracy	2,000	0.7315	0.4630	1.000
57	MMLU-Pro ★	accuracy	12,032	0.5780	0.5254	1.000
58	BBH fixed-option tasks ★	accuracy	5,507	0.7018	0.5678	1.000
59	RAGTruth ★	F1 on hallucinated class	2,700	0.7601	0.5026	1.000
61	HoVer ★	claim-verification accuracy	4,000	0.7225	0.4450	1.000
62	When2Call MCQ ★	accuracy	3,652	0.6528	0.5371	1.000
64	New Yorker caption matching ★	accuracy	528	0.6307	0.5384	1.000

Evidence: lemondeeder-eval-20261015/di/runs/20261015-lemondecider-4b-di-full-0.3/ (64 shard dirs + merged/results.jsonl + official scores.json, index.json, benchmark-summary.json).

2) JevBench public-231 (typed-decision public sanity suite)

231/231 items attempted and scorable, complete, native probability output, dataset hash-pinned (dc3995d8…de51, jevbench-v1 harness, typesafe adapter).

Metric	Value
Accuracy	83.12% (192/231)
Macro accuracy	84.55%
Brier (mean)	0.2392
Ordinal MAE (score items)	0.1207
Paraphrase consistency	36/36 pairs agree (100%)
Coverage / operational success / schema validity	1.0 / 1.0 / 1.0 (strict 1.0)
Latency p50 / p95	47.1 ms / 340.3 ms

Per-family results

Family	n	Accuracy	Brier
adequacy	12	1.000	0.051
adversarial	6	1.000	0.002
ambiguous	7	0.571	0.661
extraction	24	1.000	0.000
fact	12	1.000	0.000
intent	24	1.000	0.001
judge_hard	17	0.647	0.499
long_policy	19	0.579	0.585
multi_hop	18	0.722	0.481
ordinal	12	1.000	0.000
policy	12	1.000	0.007
probability	10	0.800	0.338
routing	12	1.000	0.013
routing_hard	5	1.000	0.000
temporal_numeric	15	0.067	1.006
tool_selection	12	1.000	0.000
tradeoff	6	0.833	0.450
trap	8	1.000	0.038

Known weak spots on this suite: long-policy judgement, ambiguous states, hard-judge calls, and especially temporal/numeric reasoning (6.7%), which the model inherits from its parents' training data.

Evidence: lemondeeder-eval-20261015/jevbench/20261015-lemondecider-4b-jevbench-public231/ (results.jsonl, ledger.jsonl, public-summary.json, 5-item smoke gate passed).

Usage

Dependencies: transformers==5.17.0 (needs the Qwen3.5 hybrid classes), flash-linear-attention==0.5.2, pillow, torch with SDPA. This package honours a single runtime env var: LEMONDECIDER_CUDNN_SDPA=1 (keep the cuDNN SDPA backend on CUDA; defaults are kept on CPU/ROCm).

import sys, torch
from transformers import AutoModel

ckpt = "lemondecider-4b-20261015-v2"      # local package directory
sys.path.insert(0, ckpt)                  # make the in-package runtime importable
from decision_utils import load_processor, predict_batch

model = AutoModel.from_pretrained(
    ckpt, trust_remote_code=True, local_files_only=True,
    dtype=torch.bfloat16, device_map="cuda:0", attn_implementation="sdpa",
).eval()
processor = load_processor(ckpt)

rows = [{
    "state": "The user asks how to return a damaged order placed last week.",
    "question": {"type": "choice",
                 "instructions": "Pick the best next action",
                 "criteria": {"refund": "Process a refund",
                              "replace": "Send a replacement",
                              "escalate": "Escalate to a human"}},
}]
answers = predict_batch(model, processor, rows)
print(answers["answers"][0])   # choice, probabilities over the 3 options, confidence

CLI from inside the package: python3 model.py --checkpoint . --device cuda:0 --input rows.json.

Batching: each predict_batch call runs ONE forward pass over all the rows you pass (inputs are left-padded to the longest prompt); the HTTP serving bridge chunks its requests at 8 questions per forward pass. Requests over the 262,144-token max_input_length limit are rejected with the limit named (HTTP 422 in server wrappers) — nothing is truncated.

Intended use & limitations

• Intended: typed decision endpoints — routing, intent classification, policy/ adequacy checks, structured QA over explicit option lists, pipeline control.
• The model emits option probabilities only; it is not a generative assistant.
• Ratings (score items) are supported through the same readout; they inherit the parents' weak arithmetic/timing behavior (see the temporal_numeric row above).
• Trained on public data sources assembled by the project; no evaluation rows were used for training (corpus admission contracts enforced this).
• Uncalibrated: temperature 1.0, no post-hoc calibration fitted. The 6 low-confidence bins in the JevBench calibration table show mild over-confidence in the 0.6–0.8 confidence range.

License

• LICENSE — Apache-2.0 text as preserved from the parent checkpoint lineage (base: Qwen/Qwen3.5-4B, inherit the base model's terms).
• LICENSE.code — MIT for runtime/server code components adapted from upstream.
• No benchmark score is claimed by this package beyond the reproducible evidence files listed above; the Decision Index owner reserves 80% of its Full score for private tests, so the 50.31 public index is not a board submission.

Acknowledgements

Built by the Lemon project. Runtime and server components build on Apache-2.0 and MIT upstream code (see LICENSE files). Base backbone by the Qwen team.