card: the decider family table, contents, changelog
Browse files
README.md
CHANGED
|
@@ -15,6 +15,23 @@ calibrated probability over the options at a single answer slot. No generation.
|
|
| 15 |
frame costs 64 visual tokens. Text-only questions work too, with decider-2b v5's behaviour,
|
| 16 |
including its abstention handling.
|
| 17 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
## Usage
|
| 19 |
|
| 20 |
```python
|
|
@@ -35,9 +52,11 @@ MiniGrid and Super Mario Bros labelled by scripted policies (rare actions oversa
|
|
| 35 |
DAgger frames from an earlier model's own play); multiple-choice image tasks from The Cauldron
|
| 36 |
(A-OKVQA, AI2D, ScienceQA, IconQA, TQA, Raven, Hateful Memes); a replay of the text mixture.
|
| 37 |
Then PPO from pixels on Breakout and Pong (the softmax over action options is the policy).
|
| 38 |
-
Code: https://github.com/Mapika/decider
|
|
|
|
|
|
|
| 39 |
|
| 40 |
-
|
| 41 |
|
| 42 |
| task | accuracy | ECE |
|
| 43 |
|---|---|---|
|
|
@@ -51,4 +70,22 @@ teacher scores 22), Pong 3 (teacher 8), CliffWalking -13 (optimal), MiniGrid Emp
|
|
| 51 |
(teacher level); held-out Freeway 0, FrozenLake 0, the harder grid worlds 0 (their scripted
|
| 52 |
teachers also score 0), Mario 1-1 315 px. The previous vision release (v4-based) scored
|
| 53 |
Breakout 16, Pong 8, Freeway 8, BabyAI-GoTo 0.30; this one trades Pong and Freeway for
|
| 54 |
-
Breakout and for the corrected abstention behaviour.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
frame costs 64 visual tokens. Text-only questions work too, with decider-2b v5's behaviour,
|
| 16 |
including its abstention handling.
|
| 17 |
|
| 18 |
+
**Contents:** [The decider family](#the-decider-family) 路 [Usage](#usage) 路 [Training](#training) 路 [Results](#results) 路 [Limitations](#limitations) 路 [Changelog](#changelog)
|
| 19 |
+
|
| 20 |
+
## The decider family
|
| 21 |
+
|
| 22 |
+
All five repositories share one interface (`decider.infer.Decider`, `POST /v1/systemone` in TypeSafe's format) and one
|
| 23 |
+
readout: the letter logits at an answer slot, softmaxed over the options. Pick by size and input.
|
| 24 |
+
|
| 25 |
+
| model | base | weights | use it for | numbers |
|
| 26 |
+
|---|---|---|---|---|
|
| 27 |
+
| [decider-2b](https://huggingface.co/Mapika/decider-2b) v10 | Qwen3.5-2B-Base | 3.5 GB bf16 | the default: routing, classification, judgments, browser agents; 4 ms per request with CUDA graphs on one GPU | regression set 0.805 in-task / 0.755 held-out; live browser 93%; Bespoke suite 0.704 |
|
| 28 |
+
| [decider-35b-a3b](https://huggingface.co/Mapika/decider-35b-a3b) v1 | Qwen3.5-35B-A3B-Base (3B active) | 65 GB bf16 | when accuracy is worth 3 to 4 times the cost per decision: knowledge and multi-step questions, long policies | 0.855 / 0.810, above the 2B on 93 of 95 tasks; JevBench hard 0.676; Bespoke 0.774; no RL stage |
|
| 29 |
+
| [decider-35b-a3b-nvfp4](https://huggingface.co/Mapika/decider-35b-a3b-nvfp4) | the 35B in NVFP4 | 19.6 GB | the 35B on Blackwell through vLLM or TensorRT-LLM | 1.0 to 1.5 points under bf16 on the measured fixtures |
|
| 30 |
+
| [decider-0.8b](https://huggingface.co/Mapika/decider-0.8b) | Qwen3.5-0.8B-Base | 1.4 GB bf16 | the smallest: routing, yes/no and short-state lookups within 1 to 4 points of the 2B, 1.5x faster | 0.776 / 0.707 on the single-run protocol (2B: 0.809 / 0.739) |
|
| 31 |
+
| [decider-2b-vision](https://huggingface.co/Mapika/decider-2b-vision) | Qwen3.5-2B vision-language, v5 text weights | 4.1 GB bf16 | decisions from an image plus a question; game frames | Visual7W 0.89; Breakout 41 from pixels |
|
| 32 |
+
|
| 33 |
+
Code, data registry, training scripts, the changelog and the per-version history: https://github.com/Mapika/decider.
|
| 34 |
+
|
| 35 |
## Usage
|
| 36 |
|
| 37 |
```python
|
|
|
|
| 52 |
DAgger frames from an earlier model's own play); multiple-choice image tasks from The Cauldron
|
| 53 |
(A-OKVQA, AI2D, ScienceQA, IconQA, TQA, Raven, Hateful Memes); a replay of the text mixture.
|
| 54 |
Then PPO from pixels on Breakout and Pong (the softmax over action options is the policy).
|
| 55 |
+
Code: https://github.com/Mapika/decider (`decider/vision/`).
|
| 56 |
+
|
| 57 |
+
## Results
|
| 58 |
|
| 59 |
+
300 items per task.
|
| 60 |
|
| 61 |
| task | accuracy | ECE |
|
| 62 |
|---|---|---|
|
|
|
|
| 70 |
(teacher level); held-out Freeway 0, FrozenLake 0, the harder grid worlds 0 (their scripted
|
| 71 |
teachers also score 0), Mario 1-1 315 px. The previous vision release (v4-based) scored
|
| 72 |
Breakout 16, Pong 8, Freeway 8, BabyAI-GoTo 0.30; this one trades Pong and Freeway for
|
| 73 |
+
Breakout and for the corrected abstention behaviour.
|
| 74 |
+
|
| 75 |
+
## Limitations
|
| 76 |
+
|
| 77 |
+
Not a chat model and not a captioner: it answers lettered options at one slot. The text weights inside are decider-2b **v5**,
|
| 78 |
+
so the text-only behaviour is that of v5 (its abstention handling, none of the v6 to v10 input shapes, calibration or browser
|
| 79 |
+
results); a retrain on the current text weights has not been released. Game play from pixels is measured on the Atari, MiniGrid
|
| 80 |
+
and Mario frames it was trained on plus a few held-out games, three episodes each. English only; calibration is measured on the
|
| 81 |
+
listed datasets, not on your images.
|
| 82 |
+
|
| 83 |
+
## Changelog
|
| 84 |
+
|
| 85 |
+
| version | what changed |
|
| 86 |
+
|---|---|
|
| 87 |
+
| **current weights** | v5 text weights transplanted into the Qwen3.5-2B vision-language model, one epoch on game frames, The Cauldron multiple-choice tasks and a text replay, then PPO from pixels on Breakout and Pong |
|
| 88 |
+
| previous release | v4-based: Breakout 16, Pong 8, Freeway 8, BabyAI-GoTo 0.30 |
|
| 89 |
+
|
| 90 |
+
Every decider release is listed in [docs/CHANGELOG.md](https://github.com/Mapika/decider/blob/main/docs/CHANGELOG.md) of the
|
| 91 |
+
GitHub repository.
|