fraQtl

More verified long-context users per GPU.

9 concurrent 128K-context users on one A100 — fp16 fits 2, fp8's best clean rung fits 4. 134.1 tok/s aggregate, every user's retrieval needle verified (9/9), zero fallbacks. Reproducible for ~$12.

Same GPU (A100-80GB), same model (Qwen3-4B-Instruct-2507), same context (128K/user), CUDA graphs on. That's 2.0× fp16's aggregate at 4.5× its users, and 1.14× fp8's best at 2.25× its users — per-user 14.9 tok/s, disclosed. fp8 reaches 5 users only on a degraded rung; we quote its best.

Single-user decode, honestly: ~92% of fp16 at 8K, ~97% at 32K (weights dominate there), and faster than fp16 at 128K — the win grows exactly where fp16 runs out of memory. Never "lossless"; every number retrieval-gated.

What we ship

🛠 KV-cache serving runtime (vLLM + llama.cpp)

Attention computed directly on compressed KV pages — no dequantize step. ~2.4–2.65× fp16 KV capacity, verified across three architectures (Qwen3-4B, Mistral-7B, Mistral-Nemo), retrieval-gated at every context. Pip-installable wheel for vLLM; llama.cpp membrane runtime for local serving.

🧩 Hi-Fi GGUF artifacts — fraQtl calibration

Same file size as a standard Q4_K_M; measurably closer to the full-precision teacher across code, math, chat, tool calling, and long-form text.

Measured on Qwen 3.6 35B-A3B (symmetric top-20 KLD vs the Q8 teacher, 400-record held-out slices):

~30% lower KLD at identical file size. Reproducibility drift across three independent runs: 0.00000.

→ Qwen 3.6 35B-A3B (Q4_K_M) — drop-in for llama.cpp / Ollama / LM Studio / koboldcpp / Jan

🔍 fraQtl Diagnostic — free + open (Apache 2.0)

pip install fraqtl-diagnostic — projected KV savings, inference-readiness scan, per-layer compression fingerprint. Instant, no GPU.

→ Run in your browser · PyPI · GitHub

Try it on your stack

Free 1-week before/after table on your model and workload — then a 30-day technical pilot if the numbers earn it. → fraqtl.ai · contact@fraqtl.ai

Approach

• The attention-output geometry decides where the bits go; integer formats store them — theory in arXiv:2604.11501
• Every claim carries model + context + batch, with the baseline shown at its best
• Retrieval-gated: a throughput number without a retrieval check is not a receipt
• MoE expert-weight and task-aware lanes: research status, no customer claims

Links

• 🌐 Website: fraqtl.ai
• 📄 Paper: arxiv.org/abs/2604.11501
• 📬 Contact: contact@fraqtl.ai

Patent pending.