vinpix/Bonsai-8B-llama.cpp

Ternary-Bonsai-8B for llama.cpp

GGUF builds of PrismML's Ternary-Bonsai-8B for llama.cpp, covering both deployment targets: a native TQ2_0 build for CPU, and a ternary-aware zero-exact Q2_K build for GPU and every k-quant runtime — at matching perplexity.

Ternary-Bonsai is an end-to-end 1.58-bit model (weights in {-1, 0, +1}, trained with ternary QAT, no higher-precision escape hatches). It punches far above its size class, but on llama.cpp it meets a practical wall: the tight ternary format has no GPU kernel. This repository provides both halves of the solution, plus the analysis behind them.

Files

File	Format	BPW	Size	Backends	PPL (wikitext-2, 20×512)
Bonsai-8B-TQ2_0.gguf	native ternary + Q3_K head (imatrix)	2.14	2.41 GB	CPU only	12.261
Bonsai-8B-Q2_KT.gguf	ternary-aware zero-exact Q2_K	2.911	2.99 GB	CPU · CUDA · Metal · Vulkan · ROCm · SYCL	12.299

Both decode to the same learned weights. They differ in storage container — which determines which backends can run them — and in how the non-ternary tensors (output head, token embedding) are quantized.

TL;DR — which one? CPU box → TQ2_0 (fewest bytes streamed, smallest RAM, fastest generation on CPU). GPU offload, or any runtime that only speaks k-quants → Q2_KT.

---

Why two files

TQ2_0 — the native CPU format

TQ2_0 is the tightest faithful ternary layout in ggml (~2.06 BPW for the body). It is the right choice on CPU and it is what PrismML ships. Its one hard limit:

TQ2_0 has no GPU kernel. In current llama.cpp it implements a vec_dot for CPU only — no CUDA, Metal, or Vulkan path. -ngl silently keeps the ternary tensors on the host.

The build here differs from a stock TQ2_0 in one respect: the output projection is quantized to Q3_K under an importance matrix (see Output head below), which trims the head, lowers perplexity slightly, and improves generation throughput — the body remains native ternary.

Q2_KT — the GPU-portable format

Q2_K is one of the most broadly supported quant types in the ecosystem — hand-written kernels for CUDA, Metal, Vulkan, ROCm, SYCL. Re-hosting the ternary weights inside a Q2_K container makes Bonsai portable everywhere Q2_K runs.

The catch: you cannot just run llama-quantize ... Q2_K on a ternary model. Doing so is actively harmful — for a reason worth explaining.

---

The trap: naively quantizing a ternary model to Q2_K

Q2_K dequantizes each weight as

w = d * (scale & 0xF) * q  -  dmin * (scale >> 4),   q ∈ {0, 1, 2, 3}

The stock Q2_K quantizer (make_qkx2_quants) is an L2-optimal grid fitter for continuous FP16 distributions. Handed a ternary tensor whose values are exactly {-s, 0, +s}, it fits a 4-level grid over 3-level data and places the levels so the ternary zero lands between grid points — at q ≈ 1.5, rounding to an offset of ±s/3.

Measured directly on Bonsai-8B (against the ternary weights, read in ggml's canonical qs order):

Encoding	Non-zero weights (sign + magnitude)	Zero reconstruction
Stock Q2_K (--pure)	preserved (100 % signs, magnitude err ~1e-5)	-0.0095 — shifted to ~`±s/3`
Q2_KT (this repo)	preserved (100 % signs)	0.000000 — exact

The stock quantizer reproduces the non-zero ternary levels faithfully; its one failure on ternary data is the zero. Because a ternary model is ~1/3 zeros, sending every zero to ±s/3 injects error across a third of all weights — enough that a --pure Q2_K degrades badly (it loops single tokens). The stock pipeline masks this by promoting sensitive tensors (attn_output, ffn_down, attn_v) to Q3_K/Q4_K, inflating the file to ~3.18 BPW and only partially repairing the damage. That promotion is a workaround for the zero-error, not a fix.

There are two principled ways to fix the body instead of masking it — described next.

---

Two ways to quantize the ternary body correctly

1. Zero-exact packing (this repo's Q2_KT)

The ternary structure admits an exact Q2_K encoding with no residual. For a group of magnitude s:

scale nibble  sc = 15
min   nibble  m  = 15          (min-nibble == scale-nibble)
d = dmin = s / 15
q = t + 1                      t ∈ {-1, 0, +1}  →  q ∈ {0, 1, 2}

Substituting into the dequant:

w = d·15·q − dmin·15 = d·15·(q − 1) = s·(q − 1) = s·t     ← exact ternary

The −1 bias falls out of the min term for free; the ternary zero (t = 0 → q = 1) reconstructs to exactly 0; every non-zero to exactly ±s. The only error is the FP16 storage of the single per-group scale — which TQ2_0 also carries. The result is a pure Q2_K body (no tensor promotion, 2.91 BPW) at perplexity on par with the native TQ2_0 build.

The qs nibbles are packed with the exact ggml Q2_K interleave (32-byte groups, stride-32 crumbs across 128-element spans), so the file is bit-compatible with the reference dequantize_row_q2_K and the SIMD vec_dot_q2_K_q8_K in the forward pass.

Layout note. TQ2_0 and Q2_K qs streams share the same stride-32 / 128-span interleave. Matching ggml's canonical order is the whole game: an encoding that is self-consistent with a hand-rolled unpacker but disagrees with the canonical order will pass every fidelity check against itself and still emit garbage in the forward pass, because dequantize_row and vec_dot both assume the canonical order. This build matches it.

2. Importance-matrix quantization (the standard route)

A second, well-established option is to run the stock Q2_K quantizer under an importance matrix (llama-quantize --imatrix), which weights each channel by its contribution to the output and largely compensates for the zero-error.

How the two compare. Measured at matched footprint (pure Q2_K body, Q4_0 head), the two are within noise of each other on perplexity — the imatrix route and the zero-exact route land at essentially the same quality. They differ in their trade-offs:

• Zero-exact is calibration-free and deterministic: it is a closed-form property of ternary data, needs no calibration corpus, and cannot overfit to one. On GPU it is also faster, since the pure Q2_K body avoids the promoted Q3_K/Q4_K tensors an imatrix pipeline typically emits.
• imatrix is the pragmatic default when you already run a calibration pass, and it composes naturally with head/embedding choices in a single llama-quantize invocation.

This repo ships the zero-exact Q2_KT for the reasons above; both are legitimate, and the choice is a trade-off, not a hierarchy.

---

The output head: a free win via imatrix

The output projection (output.weight) is streamed once per generated token and, in a ternary model, accounts for a disproportionate share of per-token bytes (the body is 2-bit, the head is not). Two facts make it easy to optimize:

• The token embedding is a gather (only the current token's row is read), so its size does not affect generation throughput. Keep it at Q4_0; enlarging it buys nothing.
• The output head is streamed, so shrinking it directly reduces per-token bytes.

Quantizing the head to Q3_K under an importance matrix trims it, and — because the imatrix weights the head's channels by importance — lowers perplexity relative to a plain Q4_0 head rather than raising it. The CPU TQ2_0 build here uses this: smaller head, better perplexity, higher generation throughput, at no cost.

Backend caveat. This applies to CPU. On GPU, Q3_K's heavier per-element decode makes a Q3_K head slower than Q4_0 despite fewer bytes — so the Q2_KT (GPU) build keeps a Q4_0 head. The right head format depends on where the decode runs.

---

Usage

CPU (use TQ2_0)

llama-cli -m Bonsai-8B-TQ2_0.gguf -p "The capital of France is" -n 64

Smallest RAM footprint and the fastest generation of the two on CPU (it streams the fewest bytes per token). Recommended for CPU-only inference.

GPU offload (use Q2_KT)

llama-cli -m Bonsai-8B-Q2_KT.gguf -ngl 99 -p "The capital of France is" -n 64

Verified running fully offloaded on an AMD Radeon iGPU via the Vulkan RADV backend. Because Q2_K has real GPU kernels, -ngl 99 actually places the ternary body on the device — unlike TQ2_0, which would stay on the host. The same file runs on CUDA, Metal, ROCm and SYCL.

Faster prompt processing for Q2_KT: ik_llama.cpp

If you run the Q2_KT file and want quicker prompt processing (long contexts, RAG, batch), ik_llama.cpp — a performance-focused llama.cpp fork — speeds up its prefill via iqk_mul_mat kernels, which unpack the quantized weights once per batch and reuse them across the batch rows. No special file is needed: point it at Q2_KT and enable runtime repack with -rtr.

git clone https://github.com/ikawrakow/ik_llama.cpp && cd ik_llama.cpp
cmake -B build -DGGML_NATIVE=ON && cmake --build build --config Release -j
./build/bin/llama-cli   -m Bonsai-8B-Q2_KT.gguf -rtr 1 -p "The capital of France is" -n 64
./build/bin/llama-bench -m Bonsai-8B-Q2_KT.gguf -rtr 1 -p 512 -n 128   # to benchmark

This gave a solid double-digit-percent gain in Q2_KT prompt-processing in our tests, generation speed essentially unchanged. Two things to keep in mind:

• This helps Q2_KT, not TQ2_0. ik_llama.cpp does not implement the mainline TQ2_0 type, so TQ2_0 cannot be run there. On a CPU-only box, native TQ2_0 on stock llama.cpp still has the fastest prefill of any option here — it streams fewer bytes and decodes more cheaply.
• It is a separate fork — treat it as an optional accelerator, not a drop-in replacement for your stock toolchain.

Serving multiple sequences

In the bandwidth-bound regime that governs CPU generation, the weights read for one sequence serve N sequences at once. For throughput-oriented workloads (batch generation, evaluation, multi-user serving), llama-batched-bench with several parallel sequences yields a large aggregate speedup over single-stream — free, with no draft model and no speculative decoding. It does not help single-request latency; use it when you have many requests, not one.

Any GGUF loader / llama.cpp server

Q2_KT loads as an ordinary Q2_K model — no custom code, no patched runtime. Every existing loader already understands the container.

---

Quality

Perplexity on wikitext-2-raw (test split, 20 chunks of 512 tokens, identical harness):

Model	Format	BPW	PPL (lower is better)
Ternary-Bonsai-8B (native TQ2_0, stock head)	native ternary	2.06	12.311
Bonsai-8B-TQ2_0 (this repo, Q3_K head + imatrix)	native ternary	2.14	12.261
Bonsai-8B-Q2_KT (this repo, zero-exact)	ternary-aware Q2_K	2.911	12.299

Both builds are perplexity-neutral or better vs. the native ternary reference; deltas are within the ±0.5 statistical error of the measurement. The Q2_KT build pays ~0.85 BPW over native TQ2_0 for universal backend support and pays nothing for it in quality — expected, since the zero-exact encoding is lossless on the ternary levels.

For task benchmarks (MMLU-Redux, GSM8K, HumanEval+, IFEval, …), see PrismML's numbers for the base model — these files change the container, not the learned weights.

---

Technical notes & reproducibility

• Q2_KT container: 252 ternary body tensors → pure Q2_K (zero-exact); output + token_embd → Q4_0; norms → F32. No tensor promotion — the zero-exact encoding makes the Q3_K/Q4_K promotion unnecessary, which is why it is 2.91 BPW, not 3.18.
• TQ2_0 container: native ternary body; output → Q3_K (imatrix); token_embd → Q4_0; norms → F32.
• Bit-exactness: the zero-exact encoding is validated against ggml's reference dequantize_row_q2_K (100 % sign accuracy, max abs error 1e-5 from the FP16 scale) and end-to-end against the forward vec_dot (coherent generation, perplexity parity).
• On sub-2-bit formats: IQ1_BN (1.62 BPW) and IQ2_BN (2.0 BPW) were evaluated via ik_llama.cpp. Both quantize coherently — the per-group ternary scales survive — but on CPUs without dedicated low-bit instructions their heavier per-weight decode makes generation slower than TQ2_0 despite the smaller footprint: the bottleneck moves from memory bandwidth to decode compute. On commodity x86, TQ2_0 remains the generation-speed choice because it is simultaneously byte-light and decode-cheap.
• Architecture: Qwen3, 36 layers, hidden 4096, FFN 12288, GQA 32/8, full 65536-token context preserved in metadata.
• Reproduce PPL: llama-perplexity -m <file> -f wiki.test.raw --chunks 20.

A note for the llama.cpp community

The ±s/3 zero-error affects any end-to-end ternary model (Bonsai, BitNet, TriLM, …) run through the stock Q2_K quantizer. A ternary-aware branch in quantize_row_q2_K — detect a block with ≤3 symmetric levels, emit d = dmin = s/15, sc = m, q = t+1 — would let the official pipeline produce zero-exact Q2_K for the whole class, calibration-free, without the promotion workaround. Q2_KT is a concrete demonstration that the encoding is exact and backend-portable. Contributions upstream welcome.

---

Attribution & license

• Base model: Ternary-Bonsai-8B by PrismML (Babak Hassibi et al., Caltech). Ternary QAT, Qwen3 architecture, end-to-end 1.58-bit.
• These files: GGUF re-encodes for llama.cpp. TQ2_0 is the native ternary layout with an imatrix-optimized head; Q2_KT is a lossless-on-ternary Q2_K re-encode for backend portability. Learned weights unchanged.
• License: Apache 2.0, inherited from the base model.

@techreport{ternarybonsai,
  title  = {Ternary Bonsai: 1.58-bit Language Models at 8B, 4B, and 1.7B Scale},
  author = {Prism ML},
  year   = {2026},
  month  = {April},
  url    = {https://prismml.com}
}

---

Community re-encodes, not affiliated with or endorsed by PrismML. Quality figures are measured on the specific files in this repository.