patrickbdevaney/MiMo-V2.6-Flash-REAP50-GGUF

Xiaomi MiMo-V2.6-Flash REAP-50 — GGUF

Official GGUF quantisations of MiMo-V2.6-Flash-REAP50, a 50% routed-expert pruned checkpoint of XiaomiMiMo/MiMo-V2.6-Flash created with REAP and HOPE second-order saliency pruning.

• Base HF Checkpoint: patrickbdevaney/MiMo-V2.6-Flash-REAP50
• Experts Retained: 128 of 256 routed experts per layer across 47 MoE layers (1 dense layer, 47 MoE layers).
• Base Architecture: Native packed MXFP4 (U8, block size 32) experts with unquantized pure BF16 attention and embeddings.
• Towers Included: Vision & Audio multimodal projectors (mmproj) and Multi-Token Prediction speculative draft heads (mtp).

---

Quantization Ladder

Filename	Quant Type	Size	Description	Recommended VRAM / RAM
MiMo-V2.6-Flash-REAP50-MXFP4_MOE.gguf	MXFP4_MOE	86.06 GiB	Flagship: 1-to-1 native packed MXFP4 experts (32 blk) + BF16 attention/trunk. Exact bit-level fidelity to REAP base.	96 GiB+ / 1x 128GB Thor or 2x 48GB
MiMo-V2.6-Flash-REAP50-Q2_K.gguf	Q2_K	61.64 GiB	Optimal Hybrid MoE: sensitive down-projections kept in native MXFP4, gate/up in Q2_K, trunk in Q8_0 (~3.36 BPW).	64 GiB+ / 3x 24GB GPUs (72GB) or Mac 64-96GB

Supporting Towers (Vision, Audio & MTP)

Filename	Size	Description
mmproj-MiMo-V2.6-Flash-REAP50-BF16.gguf	2.56 GiB	Multimodal projector (Vision + Audio) in BF16
mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf	1.46 GiB	Multimodal projector (Vision + Audio) quantized to Q8_0
mtp-MiMo-V2.6-Flash-REAP50-BF16.gguf	4.17 GiB	Multi-Token Prediction (MTP) draft head (3 next-n layers) in BF16
mtp-MiMo-V2.6-Flash-REAP50-Q8_0.gguf	2.22 GiB	Multi-Token Prediction (MTP) draft head (3 next-n layers) in Q8_0

---

Key Features

1. Native MXFP4 MoE Preservation: In the base model, 92.9% of weights are stored as native packed mxfp4 (32 block size). Our GGUF converter natively repacks these blocks directly into GGMLQuantizationType.MXFP4, avoiding costly lossy dequantization cycles while preserving exact native numerical precision.

2. Multimodal Projectors (mmproj): Xiaomi MiMo-V2.6-Flash incorporates both visual and audio processing towers:

   • Vision encoder (28-layer ViT, 560px patch representation)
   • Audio tokenizer / RVQ speech representations Both are packed into standard GGUF multimodal projectors (mmproj-*-BF16.gguf and mmproj-*-Q8_0.gguf) compatible with llama.cpp's multimodal pipeline.

3. Multi-Token Prediction (mtp): MiMo-V2.6-Flash includes 3 trained MTP layers for speculative decoding. We ship standalone MTP draft models (mtp-*-BF16.gguf and mtp-*-Q8_0.gguf) that can be loaded alongside the trunk model with --draft-model to accelerate generation.

---

Running with llama.cpp

1. Standard Text Inference (Optimal Hybrid Q2_K)

./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    -p "You are MiMo, an AI assistant developed by Xiaomi. Explain how MoE expert pruning works:" \
    -n 512 --temp 0.6

Or run the flagship bit-for-bit native MXFP4 checkpoint:

./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-MXFP4_MOE.gguf \
    -p "You are MiMo, an AI assistant developed by Xiaomi. Explain how MoE expert pruning works:" \
    -n 512 --temp 0.6

2. Speculative Decoding with MTP Draft Head

./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    --draft-model mtp-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
    -p "Explain quantum teleportation in detail:" \
    -n 512

3. Multimodal Inference (Vision & Audio)

./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    --mmproj mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
    --image input.jpg \
    -p "Describe the contents of this image in detail."

4. OpenAI-Compatible API Server

./llama-server \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    --mmproj mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
    --port 8080 \
    -ngl 99

---

Background & Pruning Method

Pruned using HOPE (Higher-Order Pruning of Experts) over a diverse calibration corpus spanning code, math, conversational text, and multimodal reasoning tasks. Rather than relying solely on first-order activation frequencies, HOPE accounts for inter-expert interaction terms: $$\mathrm{\Delta }\mathcal{L}\approx \sum _i{g}_i^T\mathrm{\Delta }{w}_i+\frac{1}{2}\sum _{i,j}\mathrm{\Delta }{w}_i^T{H}_{ij}\mathrm{\Delta }{w}_j\backslash Delta\; \backslash mathcal\{L\}\; \backslash approx\; \backslash sum\_\{i\}\; g\_i^T\; \backslash Delta\; w\_i\; +\; \backslash frac\{1\}\{2\}\; \backslash sum\_\{i,j\}\; \backslash Delta\; w\_i^T\; H\_\{ij\}\; \backslash Delta\; w\_j$$ By computing cross-expert Hessian blocks during the calibration pass, 128 experts per layer were optimally selected to minimize perplexity loss under 50% parameter reduction.

---

Created by patrickbdevaney.