AngelSlim/Hy3-DFly-Block8-Think-High

A more accessible, comprehensive, and efficient toolkit for large model compression.

✒️ TechnicalReport   |    📖 Documentation   |   🤗 Hugging Face   |   🤖 ModelScope

💬 WeChat |   🫨 Discord

Hy3-A21B DFly High-Thinking Drafter

Model Summary

This repository contains a DFly drafter model trained for speculative decoding with Hy3-A21B under the high-thinking inference setting. The model is trained with AngelSpec and is designed to propose candidate tokens that are verified by the Hy3-A21B target model.

This drafter is not a standalone chat model. It should be deployed with the matching Hy3-A21B target model and the corresponding high-thinking target-side inference mode.

• Target model: Hy3-A21B
• Drafter type: DFly
• Inference mode: High-thinking
• Training framework: AngelSpec
• Recommended serving backend: vLLM speculative decoding
• Recommended speculative tokens: 8

Training and Thinking-Mode Specialization

High-thinking and no-thinking decoding induce different hidden-state and acceptance distributions. As a result, a drafter trained for one thinking mode should not be assumed to transfer optimally to the other.

This checkpoint is the high-thinking DFly drafter: it is trained with high-thinking target-side data and should be evaluated and deployed with high-thinking prompts / target-side generation behavior. The matched high-thinking configuration is important for preserving acceptance length and speculative-decoding efficiency.

Evaluation: Accepted Length

The following table reports this model's mean accepted length on Hy3-A21B under the high-thinking setting at temperature 1. The average is computed over the six displayed benchmarks.

Target Model	Drafter	Train Data	Test Data	Math500	GSM8K	HumanEval	MBPP	LiveCodeBench	MT-Bench	Avg.
Hy3-A21B	DFly	High-think	High-think	4.29	4.47	4.76	4.74	3.92	3.42	4.27

The high-thinking drafter is specialized for high-thinking inference. It is especially important for reasoning- and chat-oriented workloads where the target-side thinking mode changes the acceptance distribution.

Deployment

Set TARGET to the Hy3-A21B target model path or repository and DRAFT to the corresponding drafter checkpoint. The relevant inference code can be found in PR https://github.com/vllm-project/vllm/pull/50246.

export TARGET="<path-or-hf-repo-to-hy3-a21b-target>"
export DRAFT="<path-or-hf-repo-to-hy3-dfly-high-thinking-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 8 \
  --speculative-config "{"method":"dspark", "model":"$DRAFT", "num_speculative_tokens":7}"

Technical Report

For more details, see the technical report:

AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
arXiv: https://arxiv.org/abs/2607.25852