dMoE-16B: dLLMs with Learnable Block Experts

dMoE is a block-level Mixture-of-Experts (MoE) framework designed for Diffusion Large Language Models (dLLMs). By aggregating token-level expert distributions within each block into a unified block-level distribution, dMoE substantially reduces the number of uniquely activated experts during inference, mitigating memory-bound bottlenecks without sacrificing performance.

• Paper: dMoE: dLLMs with Learnable Block Experts
• Project Page: https://fscdc.github.io/dMoE/
• Repository: https://github.com/fscdc/dMoE

Highlights

• Learnable Block Experts: Introduces block-level MoE routing into dLLMs, drastically compressing the number of activated unique experts across diffusion steps.
• Reduced MoE Bandwidth: Significantly reduces memory bandwidth consumed by expert weight loading during the block diffusion process.
• Improved Efficiency-Accuracy Trade-off: Achieves 1.14x to 1.66x end-to-end latency speedup while maintaining competitive performance on benchmarks.
• Plug-and-play on LLaDA-2.0: Built directly on top of LLaDA-2.0-mini without architectural changes.

Sample Usage

Please refer to the GitHub: https://github.com/fscdc/dMoE

Citation

@article{feng2026dmoe,
  title={dMoE: dLLMs with Learnable Block Experts},
  author={Feng, Sicheng and Chen, Zigeng and Fang, Gongfan and Ma, Xinyin and Wang, Xinchao},
  journal={arXiv preprint arXiv:2605.30876},
  year={2026}
}