--- language: - en - ru tags: - dflash2 - speculative-decoding - llama.cpp - tesla-p40 - pascal - gguf license: apache-2.0 library_name: gguf base_model: Qwen/Qwen3.8-27B-UD --- # llama.cpp DFlash2 P40 (Pascal) optimization Speculative decoding (DFlash2 block-diffusion drafter) tuned specifically for **NVIDIA Tesla P40** (Pascal, compute capability 6.1, no FP16 tensor cores). Fork of `ggml-org/llama.cpp` (PR #27342 lineage) with DFlash2 support, optimized for the Pascal backend: int8 DP4A (not FP16), multi-stage quantization of the draft, and an adaptive margin that shortens the verify batch where the draft's selector is uncertain. ``` Mini-article: Title placeholder -> 30.26 tok/s on Pascal P40 with 27B dense ``` --- ## What this repo is A runnable, measured build of llama.cpp + DFlash2 speculative decoding on a single Tesla P40 (24 GB, sm_61). Everything in this repo's `DFLASH2_RING_PORT.md` is a real measurement on the P40; no other GPUs / cloud numbers are claimed. Primary pairing tested: - **Target:** Qwen3.8-27B-UD `Qwen3.8-27B-UD-Q4_K_XL.gguf` (qwen35 hybrid, Gated DeltaNet) - **Draft:** DFlash2 `Qwen3.8-27B-DFlash2-q4mix-self.gguf` (our q4-mix quant) --- ## Best command (hit it) See `run_best.sh`. Summary: ```bash ./build-p40-ring/bin/llama-server \ -m Qwen3.8-27B-UD-Q4_K_XL.gguf \ -md Qwen3.8-27B-DFlash2-q4mix-self.gguf \ -ngl 999 -ngld 999 -c 8192 -b 512 -ub 512 -np 1 \ --load-mode mlock --cache-ram 32768 --checkpoint-min-step 512 \ -fa 1 -ctk q8_0 -ctv q8_0 -ctkd q8_0 -ctvd q8_0 --kv-unified \ --spec-type draft-dflash \ --spec-draft-n-max 7 --spec-draft-n-min 1 --spec-draft-p-min 0.35 \ --spec-draft-ctx 0 --temp 0 --jinja --reasoning off -bs \ -lv 4 --host 0.0.0.0 --port 8080 ``` Key flags: - `--spec-type draft-dflash` - DFlash2 block-diffusion drafter - `--spec-draft-n-max 7` - sweep optimum (Test 26: 7=30.26 > 8=30.18 > 5=27.4 > 4=27.8) - `--spec-draft-p-min 0.35` - adaptive margin (top1-top2 on raw selector logits), cuts the chain where the selector is a coin flip. Shortens the verify batch. - `--spec-draft-n-min 1` - keep short chains alive - `-bs` - backend (GPU) argmax sampling, no CPU logit readback - `-fa 1`, `-ctk/ctv q8_0`, `--kv-unified` - flash attention + q8 KV - `--reasoning off` - fastest mode; use `on` for real reasoning (see below) To switch to the reasoning scenario replace `--reasoning off` with `--reasoning on`. --- ## Measured results (Tesla P40, single GPU) All numbers are our own runs, logged in `DFLASH2_RING_PORT.md`. ### Code prompt, reasoning OFF, greedy | Draft quant | size | tok/s | acceptance | mean len | |---|---|---|---|---| | **q4-mix (recommended)** | 1.2 GB | **30.26** | 0.83 | 5.25 | | Q8 (lucebox) | 2.0 GB | ~30 | 0.85 | 5.27 | | Q2-hybrid (fc Q2, head) | 0.96GB | 29.0 | 0.80 | 5.17 | | Q2-all | 0.87GB | 27.1 | 0.76 | 4.69 | Draft quant is q4-mix = backbone q4_0 + dflash.* heads q8_0 (protects the selector). q4-mix wins: same acceptance as Q8 with a smaller, faster draft; Q2 is slower because the 2-bit selector heads lose near-tie accuracy. ### n_max sweep (q4-mix, code, reasoning OFF) | n_max | tok/s | |---|---| | 4 | 27.8 | | 5 | 27.4 | | **7** | **30.26** | | 8 | 30.18 | n_max=7 is the optimum; adaptive margin already caps the useful chain at ~5.25. ### Thinking scenario (code, reasoning ON, greedy) - ours (DFlash2 q4-mix + margin): **~18 t/s** with real reasoning tokens. - Luce chain with real thinking: 13.3 t/s (their spec is killed by a budget hook), i.e. our port is **faster than Luce on the reasoning scenario**. ### Reference (same 27B, code, reasoning OFF) - our DFlash2 q4-mix + margin: **30.26**; plain Luce chain: ~32; ngram-mod / ngram-map-k4v on top of DFlash2: **28.2** (drags us down, not used). --- ## What was done / optimizations (P40 Pascal) 1. **MMQ DP4A (int8), no FP16.** Pascal has no FP16 tensor cores. `ggml-cuda` selects the MMQ DP4A kernel (`__dp4a`, cc >= 610) for quantized weights via `(!fp16_mma_hardware_available(cc) || ...)`, compile arch `61`. Draft and target both run the int8 matmul path. No FP16 path is used for quant weights. 2. **Draft quantization pipeline** (`server/scripts/quantize_dflash_draft.py` + `llama-quantize`): - `q4-mix`: backbone q4_0, `dflash.*` heads + conv q8_0 -> 1.2 GB, acceptance-neutral. - Q2 available only via `llama-quantize` (python-gguf has no K-quant *write*). 3. **DFlash2 adaptive chain (margin)** (`common/speculative.cpp`): raw selector logits top1-top2 margin vs a threshold from `p_min`; stops the chain when the pick is a coin flip. This is what shortens the verify batch and makes n_max=7 optimal. 4. **DFlash2 is block-diffusion**: builds one noise block (id_last + n_max masks) and decodes it in a **single pass** (`draft_dflash::draft`, one `llama_decode`) - same approach as upstream PR #27342. 5. **Backend (GPU) argmax** (`-bs`): no CPU-side logit readback during draft/verify. 6. **Flash attention** (`-fa`) + `q8_0` KV for both models, unified KV. 7. **Duplicate draft quant tooling** and honest per-scenario comparison vs Luce (see DFLASH2_RING_PORT.md). --- ## HF model Our recommended draft (q4-mix, 1.2 GB): - Filename: `Qwen3.8-27B-DFlash2-q4mix-self.gguf` - HF repo: **https://huggingface.co/maxwelhelp/llama.cpp-DFlash2-pascal6-optimized** Target: `Qwen3.8-27B-UD-Q4_K_XL.gguf` (Qwen3.8-27B-UD, qwen35 hybrid). --- ## Building for P40 ```bash cmake -B build-p40-ring -DCMAKE_CUDA_ARCHITECTURES=61 -DLLAMA_CUDA=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build build-p40-ring --config Release -j ``` --- ## Notes / honesty - No cloud numbers; all measurements are local on the P40 (see the log). - ngram-mod / ngram-map-k4v did **not** help on top of DFlash2 and are not used. - On the reasoning scenario our port already beats Luce; q4-mix + margin is the recommended release config. --- License note: this is a private fork for experimental Pascal tuning; it wraps upstream llama.cpp (MIT) and DFlash2 draft weights (Apache-2.0 / z-lab).