Add K2.7-Code Eagle3-MLA draft (ckpt53/step32400, best val_loss)

README.md

@@ -1,3 +1,48 @@
 ---
 license: mit
+base_model: moonshotai/Kimi-K2.7-Code
+tags:
+  - speculative-decoding
+  - eagle3
+  - eagle3-mla
+  - draft-model
+  - vllm
+language:
+  - en
 ---
+
+# Kimi-K2.7-Code Eagle3-MLA Draft
+
+Eagle3-MLA speculative-decoding draft model for **Kimi-K2.7-Code**, trained natively on
+K2.7-Code data. Pairs with the Kimi-K2.7-Code verifier under vLLM speculative decoding.
+
+## What this is
+
+- **Algorithm:** EAGLE-3 with MLA (multi-head latent attention), single draft decoder layer.
+- **Verifier:** `Kimi-K2.7-Code` (DeepSeek-V3-class architecture; arch is identical across
+  K2.5 / K2.6 / K2.7). The draft reuses the verifier's frozen embedding / lm_head / norm.
+- **Init:** lightseek K2.6 Eagle3-MLA export, then fine-tuned on K2.7-native data.
+- **Training data:** real K2.7-Code serving traffic (agentic / coding / tool, oversampled 5x)
+  mixed with kimi-mtp prompts re-answered by K2.7-Code.
+- **Recipe:** ttt_steps=4, ttt_step_loss_decay=1.0, off-policy tokens, l2sp_lambda=1e-4,
+  cosine LR 2e-5, seq_length 8192.
+
+## Why K2.7-native
+
+A K2.6-teacher draft over-fit the K2.6 distribution and lost to the lightseek init on real
+K2.7-Code traffic. Training on K2.7-native data reverses that: on held-out K2.7 traffic this
+draft matches or beats the lightseek init on accepted-token length.
+
+## Usage (vLLM)
+
+```bash
+vllm serve /path/to/Kimi-K2.7-Code \
+  --tensor-parallel-size 8 \
+  --speculative-config '{"model": "k-l-lambda/kimi-k2.7-code-eagle3-mla", "num_speculative_tokens": 3, "method": "eagle3"}'
+```
+
+## Checkpoint
+
+This is an **intermediate** checkpoint from an in-progress run (step 32400, the best by
+validation loss among retained checkpoints at upload time). It is published for evaluation;
+a final checkpoint will follow when the run reaches its step budget.

config.json

@@ -0,0 +1,47 @@
+{
+  "architectures": [
+    "Eagle3DeepseekV2ForCausalLM"
+  ],
+  "model_type": "kimi_k2",
+  "hidden_size": 7168,
+  "intermediate_size": 18432,
+  "num_hidden_layers": 1,
+  "num_attention_heads": 64,
+  "num_key_value_heads": 64,
+  "q_lora_rank": 1536,
+  "kv_lora_rank": 512,
+  "qk_nope_head_dim": 128,
+  "qk_rope_head_dim": 64,
+  "v_head_dim": 128,
+  "hidden_act": "silu",
+  "rms_norm_eps": 1e-05,
+  "vocab_size": 163840,
+  "draft_vocab_size": 163840,
+  "torch_dtype": "bfloat16",
+  "rope_theta": 50000.0,
+  "rope_scaling": {
+    "beta_fast": 1.0,
+    "beta_slow": 1.0,
+    "factor": 64.0,
+    "mscale": 1.0,
+    "mscale_all_dim": 1.0,
+    "original_max_position_embeddings": 4096,
+    "type": "yarn"
+  },
+  "eagle_config": {
+    "eagle_aux_hidden_state_layer_ids": [
+      2,
+      30,
+      58
+    ],
+    "use_aux_hidden_state": true,
+    "use_input_layernorm_in_first_layer": true,
+    "use_last_layernorm": true,
+    "use_mtp_layernorm": false
+  },
+  "bos_token_id": 163584,
+  "eos_token_id": 163585,
+  "pad_token_id": 0,
+  "_torchspec_version": "0.1.0",
+  "max_position_embeddings": 262144
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94d725abbe70ed552adad9cdb4e0fc0e02782a7fba4a55ae49e004598c3c03a2
+size 6031210296