Nanbeige4.2-3B-FP8-Dynamic

FP8 W8A8 (dynamic per-token activations) quantization of Nanbeige/Nanbeige4.2-3B — the looped transformer (22 layers × num_loops: 2). 4.9 GB (from 7.9), +60% batch-1 decode, no measurable quality loss on any gate I ran. This is my recommended serving artifact for this model.

Why FP8 and not 4-bit for this architecture

The looped architecture executes every layer twice per token, which compounds quantization error twice per token. In my gate battery (below), uniform NVFP4 lost 8 points of GSM8K strict; FP8-dynamic lost nothing — its runtime per-token activation scales adapt to each loop pass's distinct activation distribution. As far as I can tell these are the first published FP4/FP8 quantization results for a looped LLM (the only prior looped-LLM PTQ study, LoopQ arXiv:2605.16343, tested INT formats only).

Quality evals (RTX 5090, vLLM v0.25.1, 64k ctx, fp8 KV)

Public harnesses first (lm-eval, reproducible), then my closed harnesses (relative signals within this family). Speed has its own section below.

bench	mode	bf16	FP8-Dynamic
GSM8K strict (n=100)	thinking	not measured	89
GSM8K flexible	thinking	not measured	93
GPQA-diamond (flexible-extract, n=198, 32k budget)	thinking	not measured	81.3
IFEval prompt-strict (n=250)	non-thinking	not measured	76.4
IFEval inst-strict	non-thinking	not measured	83.5
MMLU-Pro (25/category)	non-thinking	not measured	62.6
BBH CoT few-shot	non-thinking	not measured	64.9
MultiHop-RAG (gold evidence, n=248)	non-thinking	not measured	74.2
Blind-judge summarization (closed, 48 articles, same-judge pair)	non-thinking	4.61	4.55 (noise-level Δ)
Judged faithfulness (closed)	non-thinking	5.0	4.95-5.0
Judged fabrication / context leaks (closed)	non-thinking	0% / 0%	0-2% / 0%
Dictation-rewrite taxonomy (closed, 20 cases)	non-thinking	18/20	18/20

Judge protocol: blind single-judge, per-article anonymized+shuffled candidates, strict rubric (faithfulness/coverage/conciseness/coherence/concern-validity).

Speed (RTX 5090, batch-1, vLLM v0.25.1, 64k ctx, fp8 KV)

Decode tok/s, single stream, per-workload best speculative config:

workload	spec decode	bf16	FP8-Dynamic
freeform / chat / agent	off	96	154
summarize / RAG (2k+ ctx prompts)	ngram, 8 tok	136	206

These are two workload anchors, not an ISL sweep; decode speed shifts with context length, batch size, and backend. Notes from the serving campaign:

• The looped arch reads every weight twice per token, so weight-compression speedups roughly double vs a normal 3B (+60% batch-1 here).
• ngram speculation is workload-dependent: +40-60% on copy-heavy work (summarization/RAG), −15-20% on freeform. The rows above use each workload's best config.
• Attention backend is context-dependent: FlashInfer (default) wins at 2k+ ctx; VLLM_ATTENTION_BACKEND=TRITON_ATTN lifted spec-off short-form decode a further +13% in my follow-up runs.

Performance under load (GuideLLM, real articles, eagle3 head)

Concurrency sweep of the production config (FP8 + EAGLE-3 draft, TRITON_ATTN, k=3) on real article prompts. Aggregate throughput scales with concurrency; inter-token latency stays essentially flat under load.

concurrent streams	aggregate output tok/s	TTFT p50 (ms)	ITL p50 (ms)
4	501	166	6.2
8	878	70	6.7
16	1017	242	7.6

Measured with GuideLLM on an RTX 5090, vLLM v0.25.1, real multi-kB article prompts, EAGLE-3 v2 (thinking-aware) head. The server saturates to ~1000 tok/s aggregate at concurrency 16 with per-token latency near-flat (6.2 to 7.6 ms), so it batches well for multi-user / parallel-agent workloads. TTFT p50 is queueing-dominated at these request counts, not a decode signal. Spec acceptance held over the run (154k accepted / 354k drafted, 0.44).

Serving (vLLM)

The architecture is not yet in upstream vLLM (PR #49433 open). Until it merges, install the bundled out-of-tree plugin (vendored from that PR), then serve normally — the checkpoint format is auto-detected:

pip install --no-deps ./vllm_plugin
VLLM_ATTENTION_BACKEND=TRITON_ATTN \
vllm serve <this-repo> --trust-remote-code \
  --max-model-len 65536 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"eagle3","model":"NullSense/Nanbeige4.2-3B-EAGLE3","num_speculative_tokens":3}'

Since 2026-07-23 the recommended speculation is my EAGLE-3 draft head, thinking-aware since the 2026-07-24 retrain: thinking-mode acceptance 0.41 (0.33 before), and +12-41% single-stream decode over the prior head depending on context, in one config (TRITON_ATTN required with it; FlashInfer + spec drops CUDA graphs to piecewise on SM120). One measured exception: LONG-document summarization (multi-k-char inputs) still favors the ngram config ({"method":"ngram","num_speculative_tokens":8, "prompt_lookup_max":4,"prompt_lookup_min":2}): 231 vs 160 output tok/s on 16 real articles; verbatim copy-spans reward lookup drafting. Pick per dominant workload.

• Thinking ON by default; pass chat_template_kwargs: {"enable_thinking": false} (+ preserve_thinking: false for chat) to disable. Sampling defaults (T=0.6/top_p=0.95/top_k=20) ship in generation_config.json; use T=1.0 for agentic.
• ngram speculation: +40-60% on summarization/RAG, −15-20% on freeform — drop the --speculative-config for freeform-heavy serving.
• --kv-cache-dtype nvfp4 is SM100-only; use fp8 on consumer Blackwell.

Download just this artifact:

hf download NullSense/Nanbeige4.2-3B-FP8-Dynamic --local-dir Nanbeige4.2-3B-FP8-Dynamic

Creation

llmcompressor QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]), data-free oneshot(). transformers ≥5 users: the bundled modeling_nanbeige.py includes two one-line compat patches vs the original repo (rope_scaling "type" key, _tied_weights_keys list→dict) — required for any transformers-5 loading (llmcompressor, finetuning); vLLM serving does not use them.

Limitations

• vLLM-only until PR #49433 merges (transformers inference works but is unoptimized for the looped arch).
• English+Chinese model; all my evals are English-only.
• Evals are n=100 (GSM8K) / n≈44 (judged summarization) — directional, not leaderboard-grade.

Which artifact should I choose?

artifact	size	tok/s (freeform / summ)	pick when
FP8-Dynamic (this repo)	4.9 GB	154 / 206	default: no measured quality loss on any gate. Recommended.
NVFP4-FP8-LoopShield	4.5 GB	159 / 202	smallest artifact that keeps reasoning at FP8 parity; best JSON reliability. Recommended for tight VRAM.
NVFP4A16	3.6 GB	175 / 220	fastest; summarization/extraction only (reasoning drops 8 GSM8K points). Not for math/agentic.
EAGLE3 draft	+1.5 GB	+12-41% decode	add-on speculator for any of the above; thinking-aware retrain (2026-07-24), thinking-mode acceptance 0.41, lossless. Serve with TRITON_ATTN.

All three serve identically (same plugin, same flags); only the checkpoint differs. Rule of thumb: this model + 64k fp8-KV context fits in ~7 GB VRAM at FP8, ~6 GB at NVFP4A16; any 8 GB card runs the full family.

Full quality matrix vs the bf16 original

metric	bf16 original	FP8 (this repo)
rewrite taxonomy (/20, non-think)	18	18
blind-judge summarization (same judging pass)	4.61	4.55 (noise-level delta)
JSON parse rate (/48)	44	42
GSM8K, IFEval, MMLU-Pro, BBH, MultiHop-RAG	not measured on bf16	measured (tables above)

Where bf16 was measured, FP8 shows no regression; that is why FP8 serves as the reference for the rest of the family instead of re-running every gate on bf16. The vendor's published numbers are a different suite entirely (agentic/frontier benches, thinking mode at a 131k token budget) and don't map onto these rows — see Benchmark provenance below.

Links & provenance

• Base model: Nanbeige/Nanbeige4.2-3B
• Family: FP8-Dynamic · NVFP4-FP8-LoopShield · NVFP4A16 · EAGLE3 draft
• Arch serving support: vLLM PR #49433 (until merged, the bundled vllm_plugin/ registers it out-of-tree)
• Upstream tooling reports I filed from this work: llm-compressor#2952 (GPTQ Hessian inversion fails on all modules of this looped arch — root cause unknown; silent RTN fallback) · llm-compressor#2953 (AWQ lacks arch mappings)
• Quantized with llm-compressor 0.12.0 (compressed-tensors format)

Benchmark provenance (ordered: public-harness first, then my closed harnesses)

Public, reproducible (lm-eval-harness local-chat-completions, exact configs in each row's annotation):

1. GSM8K — grade-school math, the reasoning gate (thinking mode, n=100, max_tokens 8192)
2. GPQA-diamond — graduate-level science QA (gpqa_diamond_cot_zeroshot, thinking mode, 32k generation budget, full n=198, seed 1234, mean generation ~12.5k tokens, zero budget truncation; flexible-extract because strict-match's answer regex does not fit thinking output; served with the lossless EAGLE-3 head, which does not affect outputs)
3. IFEval — verifiable instruction following (non-thinking, n=250)
4. MMLU-Pro — 10-choice knowledge/reasoning (non-thinking, 25/category)
5. BBH — hard reasoning suite, CoT few-shot (non-thinking, 15/subtask)
6. MultiHop-RAG — multi-doc news QA; I run generator-only with gold evidence (custom harness, dataset public)

Closed/personal harnesses (not publicly reproducible — my own serving-workload gates; treat as relative signals between artifacts in THIS family, not cross-model scores):

• Blind-judge summarization — 48 stratified real articles, per-article anonymized+shuffled candidates, single LLM judge scoring faithfulness/coverage/fabrication. Tests: does the quant change long-form grounded generation quality?
• Rewrite taxonomy — 20 dictation-cleanup cases from a production ASR pipeline. Tests: instruction-constrained short-form editing.
• JSON parse rate — structured-output emission over the summarization set. Tests: format discipline under quantization.

Citation

@misc{peciukonis2026nanbeige42fp8,
  author       = {Pe{\v{c}}iukonis, Matas (NullSense)},
  title        = {Nanbeige4.2-3B quantization family: loop-aware FP8 and NVFP4 quants of a looped transformer},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic},
  note         = {First published classic-bench (GSM8K/IFEval/MMLU-Pro/BBH) numbers for this model; no measured regression vs bf16 on any gate run.}
}