classone-gemma4-e4b — ClassOne System 1 Decision Model

devops-thiago/classone-gemma4-e4b is an open-source System 1 decision model built on google/gemma-4-E4B-it using the ClassOne (Classifier One) architecture. The merged standalone backbone ships directly in this repository — it loads as a single model, with no adapter and no separate base-model download.

Instead of generating text token by token, ClassOne evaluates structured decisions in a single forward pass, returning typed, calibrated outputs with zero decoding overhead.

Self-Measured Benchmark Results

Self-Measured Evaluation Provenance: Executed locally on 2x NVIDIA RTX 5060 Ti 16 GB (Date: 2026-11-13, Commit: c375207). See Official Results (Benchmark Heaven) below for third-party sealed evaluation.

1. JevBench Public Multi-Tier Suite (231 Public Tasks, Self-Measured Standalone BF16)

Evaluated standalone (Pillar 3A promoted release, heads SHA256 8F7157F4...) across all 231 public tasks in fstandhartinger/jevbench.

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB · Date: 2026-11-13 · Commit: c375207

Tier	Tasks	Accuracy	ECE	Brier Score	Median Latency (p50)
Easy	48	100.0% (48/48)	0.0022	0.0000	97.1 ms
Original	72	97.2% (70/72)	0.0209	0.0264	95.1 ms
Hard	111	45.9% (51/111)	0.3637	0.3659	195.8 ms
Overall Aggregate (Public)	231	73.2% (169/231)	—	—	97.1 ms

• Easy Tier Sub-Breakdown: Choice: 100.0% (36/36); Noul: 100.0% (12/12).
• Original Tier Sub-Breakdown: Noul: 100.0% (24/24); Score: 100.0% (12/12); Choice: 94.4% (34/36).
• Hard Tier Sub-Breakdown: Choice: 47.8% (32/67); Noul: 44.7% (17/38); Score: 33.3% (2/6).

2. Decision Index Suite (Self-Run, 50 Tasks)

Self-run local evaluation on multimodalart/jev-decision-index.

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB · Date: 2026-11-13 · Commit: c375207

Domain	Weight	Accuracy (%)	Score
Knowledge & Verification	25.8%	100.0% (10/10)	100.00 pts
Language & Policy Contracts	25.8%	100.0% (10/10)	100.00 pts
Retrieval & Relevance Triage	20.0%	100.0% (10/10)	100.00 pts
Tools & Execution Guardrails	18.3%	100.0% (10/10)	100.00 pts
Rubric & Quality Scoring	10.1%	90.0% (9/10)	90.00 pts
Overall Weighted Score	100.0%	98.0% (49/50)	98.99 / 100.00 (0.0379 ECE, 99.6 ms p50)

3. RLCDAlignBench Alignment & Safety Suite (100 Instances, Self-Measured Standalone)

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB · Date: 2026-11-13 · Commit: c375207

Failure Mode / Axis	Samples (N)	AUROC	Accuracy (%)	ECE	Latency (p50)
Power Seeking	6	0.889	83.3%	—	—
Honesty (Deception)	11	0.633	72.7%	—	—
Reward Hacking	9	0.650	66.7%	—	—
Faithfulness	9	0.650	66.7%	—	—
Social Bias	9	0.475	55.6%	—	—
Sycophancy	9	0.500	55.6%	—	—
Refusal (Jailbreaks)	11	0.433	54.5%	—	—
Privacy (Secret Leaks)	14	0.612	50.0%	—	—
Prompt Injection	8	0.625	50.0%	—	—
Concealing Uncertainty	14	0.469	50.0%	—	—
Overall Balanced Accuracy	100	0.601	59.4% (59.0% Acc)	0.2235	411.1 ms

4. Tool-Routing & Safety Guardrail Suites (Self-Measured Standalone)

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB · Date: 2026-11-13 · Commit: c375207

Benchmark Suite	Split / Samples	Accuracy	Secondary Metric	Calibration ECE	Latency (p50)
WildGuardTest (walledai/WildGuardTest)	test (100 stratified)	84.00% (84.54% BalAcc)	0.9233 AUROC (97.92% Benign Spec, 81.82% Adversarial)	0.1563	84.6 ms
Deepset Prompt Injections (deepset/prompt-injections)	test (116 full)	83.62% (97/116)	0.9430 AUROC (100.0% Benign Spec, 68.33% Recall)	0.1503	56.1 ms
NVIDIA When2Call (nvidia/When2Call)	mcq (100 stratified)	79.00%	87.5% request_for_info, 78.6% cannot_answer, 73.5% tool_call	0.0796	218.5 ms

Official Results (Benchmark Heaven)

Official third-party JevBench leaderboards are hosted at Benchmark Heaven (checked 2026-11-13, JevBench v1.8.10 / method v1.7, published 2026-11-12; see the live entry for classone-qwen3.5-9b). Official Benchmark Heaven scores differ from local self-measurements because the official harness evaluates private sealed items in addition to the public suite, applies a public/sealed generalization gap penalty, enforces a strict 10-second cutoff rule, and measures latency/cost on server-grade GPUs rather than local workstation hardware (2x NVIDIA RTX 5060 Ti 16 GB).

Edge vs Cloud Latency (Self-Measured Comparison)

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB · Date: 2026-11-13 · Commit: c375207

Measured against a remote cloud LLM decision API round-trip:

• ClassOne (Local RTX 5060 Ti): 52.49 ms mean latency (19.1 req/s, $0.00 inference cost, 100% private)
• Cloud API Baseline: 329.90 ms mean latency (3.0 req/s)
• Edge Speedup: 6.3× faster than cloud API round-trip latency

Decision Primitives

• Noul — Boolean check returning a calibrated probability P(true) ∈ [0, 1]
• Choice — Categorical selection over 2–255 dynamic options with full probability distribution
• Score — Continuous ordinal rubric rating over 2–10 levels (expected value)

All outputs are calibrated with a combined focal NLL (γ=2.0) + normalized Brier + contrastive margin loss and post-hoc temperature scaling.

Quickstart

pip install classone

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

from classone.modeling.modeling_classone import ClassOneModel
from classone.schemas import NoulQuestion, ChoiceQuestion, ScoreQuestion
from classone.tokenizer import ClassOnePromptBuilder

REPO_ID = "devops-thiago/classone-gemma4-e4b"

# 1. Load the ClassOne model (weights + tokenizer are fully self-contained here)
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
builder = ClassOnePromptBuilder(tokenizer)
model = ClassOneModel.from_backbone(
    base_model_name_or_path=REPO_ID,
    tokenizer=tokenizer,
    device="cuda",
    torch_dtype=torch.bfloat16,
)

# 2. Load the trained decision heads
heads = torch.load(hf_hub_download(REPO_ID, "classone_heads.pt"), map_location="cuda")
model.noul_head.load_state_dict(heads["noul_head"], strict=False)
model.choice_head.load_state_dict(heads["choice_head"], strict=False)
model.score_head.load_state_dict(heads["score_head"], strict=False)
model.eval()

# 3. Pack state + questions and run a single forward pass
packed = builder.pack(
    state={"customer": "Alex", "message": "I was charged twice for order #123."},
    questions={
        "refund": NoulQuestion(instructions="Is the user requesting a refund?"),
        "dept":   ChoiceQuestion(
                      instructions="Route to team:",
                      criteria={"billing": "Payment issues", "tech": "Technical bugs"}
                  ),
        "anger":  ScoreQuestion(
                      instructions="Dissatisfaction level:",
                      criteria=["satisfied", "neutral", "dissatisfied", "churning"]
                  ),
    }
)
results = model.evaluate_packed(packed)

print("Refund P(true):", results["refund"].noul)
print("Department:    ", results["dept"].choice, "—", results["dept"].probabilities)
print("Anger score:   ", results["anger"].score)

Repository Files

File	Description
model.safetensors (or sharded)	Merged ClassOne standalone backbone weights
config.json	Model configuration
tokenizer.json, tokenizer_config.json	Tokenizer with ClassOne delimiter tokens
classone_heads.pt	Trained Noul / Choice / Score head weights + calibrated temperatures

Certified classone_heads.pt SHA256: 8F7157F4D67D21698F94A3DF69B8492F4CC592DAC65E91AC47A83FD2E083AFDC

Citation

@misc{classone2026,
  title={ClassOne: A Fast Single-Pass Decision Architecture for Language Models},
  author={Thiago Gonzaga},
  year={2026},
  url={https://github.com/devops-thiago/class-one},
}

Attribution & Legal

• Derived from google/gemma-4-E4B-it (Google) — Apache License 2.0
• Architecture & training code: devops-thiago/class-one — Apache 2.0