---
license: mit
base_model: moonshotai/Kimi-K2.7-Code
tags:
  - speculative-decoding
  - eagle3
  - eagle3-mla
  - draft-model
  - vllm
language:
  - en
---

# Kimi-K2.7-Code Eagle3-MLA Draft

Eagle3-MLA speculative-decoding draft model for **Kimi-K2.7-Code**, trained natively on
K2.7-Code data. Pairs with the Kimi-K2.7-Code verifier under vLLM speculative decoding.

## What this is

- **Algorithm:** EAGLE-3 with MLA (multi-head latent attention), single draft decoder layer.
- **Verifier:** `Kimi-K2.7-Code` (DeepSeek-V3-class architecture; arch is identical across
  K2.5 / K2.6 / K2.7). The draft reuses the verifier's frozen embedding / lm_head / norm.
- **Training data:** real K2.7-Code serving traffic (agentic / coding / tool, oversampled 5x)
  mixed with kimi-mtp prompts re-answered by K2.7-Code.
- **Recipe:** ttt_steps=4, ttt_step_loss_decay=1.0, off-policy tokens, l2sp_lambda=1e-4,
  cosine LR 2e-5, seq_length 8192, max_steps 120000.

## Evaluation

Final checkpoint, speculative-decoding eval against the Kimi-K2.7-Code verifier
(vLLM 0.20.0, TP=8, `num_speculative_tokens=3`, c=4, greedy). Mean accepted-token length:

| Draft | Real K2.7-Code traffic | K2.6-distribution held-out |
|---|---|---|
| **This model (final)** | **2.345** | 2.246 |

## Usage (vLLM)

```bash
vllm serve /path/to/Kimi-K2.7-Code \
  --tensor-parallel-size 8 \
  --speculative-config '{"model": "k-l-lambda/kimi-k2.7-code-eagle3-mla", "num_speculative_tokens": 3, "method": "eagle3"}'
```

## Checkpoint

Final checkpoint of the K2.7-native run (step 118800; val_loss had plateaued, so the run was
stopped just short of the 120000 budget). Best by validation full-sequence accept rate among
retained checkpoints, and the eval winner on real K2.7 traffic above.