Status (2026-07-07): no weights published yet. This repository currently contains only the model card — it marks a planned variant that has not been released. Follow the repo to be notified when files land.

Leanstral-TurboQuant

TurboQuant KV cache compression for mistralai/Leanstral-2603.

This is a documentation repository that explains how to combine Leanstral's weights with TurboQuant inference-time KV cache compression. No weights are stored here — use the base model directly and apply TurboQuant via the Python package or llama.cpp fork.

Hardware compatibility

Device	VRAM / RAM	Recommendation
Any host that runs the base model	baseline + runtime savings	RotorQuant/TurboQuant is a KV-cache runtime modifier; pair with any weight variant

What is this?

KV cache compression reduces the memory used by the attention cache during inference. Unlike weight quantization (which is baked into the GGUF/MLX file), KV cache compression is applied at runtime — so the same base weights can be used with or without compression.

Technique	Where it's applied	Savings
TurboQuant KV cache	At inference time	Reduces attention memory (critical for long context)

Both can be combined for maximum efficiency.

Quickstart

Option A — Python / transformers

Install the turboquant package:

pip install turboquant

Then use it with the base model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from turboquant import TurboQuantCache

tokenizer = AutoTokenizer.from_pretrained("mistralai/Leanstral-2603", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Leanstral-2603",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# Apply TurboQuant to the KV cache
cache = TurboQuantCache(bits=4)  # or bits=2 for more aggressive compression

inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    past_key_values=cache,
    use_cache=True,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Option B — llama.cpp / LM Studio / Ollama (with fork)

TurboQuant KV cache types (planar3) are not in upstream llama.cpp. They require:

• llama-cpp-turboquant fork

Once built:

llama-cli -m Leanstral.gguf \
  --cache-type-k planar3 --cache-type-v planar3 \
  -ngl 99 -fa \
  -p "Hello"

For standard runtimes (LM Studio, Ollama, upstream llama.cpp), use conventional KV cache types (q8_0, q4_0). You lose the TurboQuant-specific benefits but keep GGUF weight quantization.

Model Specifications

Property	Value
Base Model	mistralai/Leanstral-2603
Architecture	Sparse MoE (128 experts, 4 active)
Parameters	119B total (MoE)
Context Length	256K
BF16 Size	~238 GB
Modalities	Text
License	apache-2.0

What is TurboQuant?

TurboQuant (ICLR 2026) applies random orthogonal rotations followed by optimal scalar quantization to the KV cache. Bit-identical prefill logits at 4-bit, up to 4-8× memory savings for long sequences.

Benchmarks (from the TurboQuant repository, Llama 3.1 8B on RTX 5090 — results vary by model and hardware):

• 4-bit KV cache: bit-identical prefill logits
• ~1.4-1.7× speedup on Apple Silicon
• Up to 8× KV memory savings

Benchmarks are from the TurboQuant repository using Llama 3.1 8B. Performance on Leanstral will differ. Please open a discussion if you have independent results.

Current Ecosystem Support

Runtime	TurboQuant Support	Notes
Python transformers + turboquant	✅ Full	Drop-in cache class
llama.cpp upstream	❌ Not merged	Use fork below
llama-cpp-turboquant fork	✅ planar3, iso3	GitHub
LM Studio	❌ Requested	Use q8_0 as alternative
Ollama	❌ Not supported	Use OLLAMA_KV_CACHE_TYPE=q8_0
vLLM	❌ Not supported	—
koboldcpp	❌ Not supported	—

Pre-quantized weight variants

If you want combined weight + KV cache compression, majentik hosts pre-quantized versions:

• MLX (Apple Silicon)
• GGUF (llama.cpp / Ollama / LM Studio)

See Also

• RotorQuant GitHub
• TurboQuant paper (arXiv 2504.19874)
• llama-cpp-turboquant fork
• Base model: mistralai/Leanstral-2603
• Leanstral announcement

Variants in this family

(Showing 8 sibling variants under majentik/leanstral-*. The current variant — TurboQuant — is bolded.)

Variant	Runtime	Approx size	Use case
RotorQuant	runtime modifier	n/a	KV-cache root (weight-agnostic)
RotorQuant-MLX-2bit	mlx-lm	card-only	Apple Silicon, smallest
RotorQuant-MLX-4bit	mlx-lm	card-only	Apple Silicon balanced
RotorQuant-MLX-8bit	mlx-lm	card-only	Apple Silicon reference
TurboQuant	runtime modifier	n/a	KV-cache root (weight-agnostic)
TurboQuant-MLX-2bit	mlx-lm	card-only	Apple Silicon, smallest
TurboQuant-MLX-4bit	mlx-lm	card-only	Apple Silicon balanced
TurboQuant-MLX-8bit	mlx-lm	card-only	Apple Silicon reference