thread13/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF-MTP

About

• WIP, more GGUF quantizations are coming after some tests

What you probably want instead is one of the APEX gguf builds by gbuzhf:

• gbuzhf/KAT-Coder-V2.5-Dev-APEX-MTP-GGUF

However, sometimes you want to see if a more classic quantization will be more suitable for your specific task, or simply want to fit (or use) a specific vram target. In this case, "one can't go wrong with bartowski builds (c) reddit" -- except for these are sadly missing any draft models, and a 20-30 %% gain in inference speed is not something to easily give up on.

Hence, I'll be adding here some "franken-merges" of original bartowski builds with 20 MTP layers added to it from unsloth/Qwen3.6-35B-A3B-MTP-GGUF model using a [conversion script] made by buzz.

All uploaded models were tested and add a ~20-30 % gain in inference speed compared to original ("ungrafted") models on AMD hardware.

(It is strongly recommended to use at most q8_0 quantizations for KV cache if your hardware supoorts it, especially if it has a chance to be more performant on it.)

llama.cpp invocation examples

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.q8_k_xl.gguf

• this fits 24 GiB VRAM due to using quantized KV cache

llama-server  \
--model Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.q8_k_xl.gguf  \
--no-mmproj  \
--flash-attn on  \
--n-gpu-layers all  \
--n-gpu-layers-draft all  \
--kv-offload  \
--kv-unified  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--chat-template-kwargs '{"enable_thinking":false}'  \
--reasoning-preserve  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

llama.cpp invocation examples - find optimal mtp length

As for --spec-draft-n-max 4 setting above -- the exact number would obviously depend on your GPU model.

You might want to do a log scale search by trying

• spec-draft-n-max := 1,2,4,8,... (unlikely more) while the inference speed grows or stays the same,
• and then using binary search to narrow down the exact value that works best for you on your typical tasks;
• for a quick control check, your server log will show something like:
  • nb: showing results for the same model running two consequitive coding tasks

[37813] 812.35.434.815 I slot launch_slot_: id  0 | task 56272 | processing task, is_child = 0
[37813] 812.37.748.230 I slot print_timing: id  0 | task 56272 | prompt eval time =     310.31 ms /   177 tokens (    1.75 ms per token,   570.40 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id  0 | task 56272 |        eval time =    2002.97 ms /   400 tokens (    5.01 ms per token,   199.70 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id  0 | task 56272 |       total time =    2313.28 ms /   577 tokens
[37813] 812.37.748.235 I slot print_timing: id  0 | task 56272 |    graphs reused =      52595
[37813] 812.37.748.238 I slot print_timing: id  0 | task 56272 | draft acceptance = 0.95482 (  317 accepted /   332 generated), mean len =  4.82
[37813] 812.37.749.091 I slot      release: id  0 | task 56272 | stop processing: n_tokens = 29120, truncated = 0

[37813] 812.38.114.222 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.988 (> 0.100 thold), f_keep = 1.000
[37813] 812.38.115.131 I slot launch_slot_: id  0 | task 56358 | processing task, is_child = 0
[37813] 812.38.767.224 I slot print_timing: id  0 | task 56358 | prompt eval time =     440.40 ms /   341 tokens (    1.29 ms per token,   774.29 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id  0 | task 56358 |        eval time =     211.57 ms /    20 tokens (   10.58 ms per token,    94.53 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id  0 | task 56358 |       total time =     651.97 ms /   361 tokens
[37813] 812.38.767.229 I slot print_timing: id  0 | task 56358 |    graphs reused =      52603
[37813] 812.38.767.232 I slot print_timing: id  0 | task 56358 | draft acceptance = 0.33333 (   12 accepted /    36 generated), mean len =  2.33

As you can see, we could have also tried --spec-draft-n-max 5 and see if that would improve the benchmarks: basically as long as "mean len" frequently shows values which are greater than your spec-draft-n-max parameter -- you can still increment it by at least 1 and see if it improves your statistics (and waiting times).

Also, in some scenarios there could be some use for additional settings like --spec-draft-p-min 0.5.

I am no expert, but in my tests I had some luck with first determining the optimal --spec-draft-n-max value, and then seeing if raising up the acceptance min level (and/or the "split" values such as --spec-draft-p-split) will improve things a little more.