Draft model as accelerator for DeepSeek-R1?
Is there a compatible draft model for use with llama.cpp for speculative decoding as an accelerator for Deepseek R1?
I have tested some but llama.cpp does not accept them. Is a Draft model even possible in principle with Deepseek R1?
Meh, perhaps it was just my own negative experience, but when I tested speculative decoding on a different set of models, the quality of the output was actually worse for me. The model generated very low quality words for the given context, words it would never generate by itself. It felt like chatting with a very small and dumb model and yet the main model was a 14B model. With that small model loaded in memory, I was wasting more memory for lower quality and the increase of speed that's supposed to happen with it? Nope, didn't happen either. In fact, due to having more stuff loaded in the memory, the system was actually slower overall. I'd rather them focus on new better models, pushing boundaries of what's possible with smaller models. 😉
No the quality cannot be degraded by speculative decoding, the results of the outputs always correspond to those that the main model alone would have produced. If the draft model predicts incorrectly then the main model corrects it. It only affects the speed and memory consumption. Speculative decoding is particularly worthwhile for large models.
No the quality cannot be degraded by speculative decoding, the results of the outputs always correspond to those that the main model alone would have produced. If the draft model predicts incorrectly then the main model corrects it. It only affects the speed and memory consumption. Speculative decoding is particularly worthwhile for large models.
So, you're basically negating everything I wrote there, huh? Look, I described my own experience, something I saw for myself in action. The quality of the output was objectively much lower with draft model loaded alongside the big model and the supposed speed increase was questionable. If I didn't speak from my own experience, maybe then such negating would leave me doubtful, but that's not the case here obviously. Perhaps someone else had better luck with it, but my experience with speculative decoding was very bad. The big model loaded alone (without the smaller draft model) performed much better for me.
Hey all is well, what i am writing is not a personal opinion and not based on experience or conjecture, it is a scientific fact ;-) I didn't make this up, the information can be found in the relevant publications:
"Speculative Decoding is a widely used technique to speed up inference for Large Language Models (LLMs) without sacrificing quality. When performing inference, speculative decoding uses a smaller draft model to generate speculative tokens and then uses the target LLM to verify those draft tokens."
https://arxiv.org/html/2402.01528v3
"will speculative decoding in LLMs harm the accuracy of the original model? The simple answer is no."
https://blogs.novita.ai/will-speculative-decoding-harm-llm-inference-accuracy/ (with mathematical proof)
In your case, a 14B model main model is already very small, I suspect that the technology only tends to be worthwhile from a certain size 32B/70B/etc. upwards.
In my experience it pays off enormously so it would be great if there was a draft model for R1. But I am not sure if a Draft model even possible in principle with Deepseek R1 hence the question here.
On the original question, since it hasn't been answered directly: there's a concrete reason llama.cpp rejected the draft models you tried, and it isn't that R1 is unsuitable in principle.
Speculative decoding needs the draft and target to share a vocabulary, because the target has to score the draft's exact token IDs. R1's config lists vocab_size: 129280. The R1-Distill models are built on different bases and don't match — Distill-Qwen-1.5B is 151936 (Qwen2 architecture) and Distill-Llama-8B is 128256 (Llama). So despite the naming, none of the distills can act as a draft model for R1, and a vocabulary mismatch is the usual reason llama.cpp refuses one.
The only model I could find sharing R1's 129280 vocabulary is DeepSeek-V3 itself, which obviously doesn't help as a small draft. V2-Lite is 102400 and deepseek-coder-1.3b is 32256, so neither works either.
One thing that might be more promising: R1's config also has num_nextn_predict_layers: 1, the V3-architecture multi-token-prediction module. That's DeepSeek's own built-in speculative path rather than an external draft model, and engine support for it varies. Probably a better bet than hunting for a compatible small model that doesn't seem to exist.
On the quality disagreement, for what it's worth I think you're both describing something real.
The guarantee @inputout is pointing at is solid, and it's on firmer ground than the blog post: Leviathan et al. (https://arxiv.org/abs/2211.17192) state that "the output distribution is guaranteed to remain unchanged", and Chen et al. (https://arxiv.org/abs/2302.01318) prove the same property for their modified rejection sampling scheme.
But the DeepMind paper adds a caveat that rarely gets quoted alongside it: "because the different computation graphs lead to different numerics, we cannot not expect identical outputs" (the doubled "not" is theirs). The distribution is preserved. The individual sample isn't guaranteed to be the same sample. So on a given prompt you can genuinely get different, sometimes worse-looking text without the theory being violated — you're drawing from the same distribution through a different computation graph.
@MrDevolver , your memory and speed observations also line up with the papers rather than contradicting them. Speculative decoding uses more memory by construction, since you're holding two models. And Leviathan is explicit that the method "assume[s] that we have enough compute resources to support the increased concurrency" — total arithmetic actually goes up, and it only pays back when there's idle compute to spend on it. On a local setup where both models are already competing for memory, that assumption doesn't hold. Both papers measured at batch size 1 with hardware headroom, which is a fairly different situation.
So "no speedup on a memory-constrained 14B" is closer to an expected result than a strange one, and it's consistent with @inputout 's hunch that this pays off more at larger sizes.
I went through both papers in detail while writing https://diffstudy.com/speculative-decoding-vs-autoregressive-decoding/ — every quote above is sourced there if anyone wants the full context.