| --- |
| license: apache-2.0 |
| --- |
| Eagle model for [CohereLabs/North-mini-code-1.0](https://huggingface.co/CohereLabs/North-Mini-Code-1.0). |
|
|
|
|
| Key highlights include: |
| - three layers with dense transformer blocks |
| - all draft layers use sliding window attention |
| - compatible with fp8/w4a16 target model checkpoints |
|
|
| ## vllm |
| To start vllm server with this draft model, run the following |
| ``` |
| vllm serve CohereLabs/North-Mini-Code-1.0 \ |
| -tp 2 \ |
| --max-model-len 320000 \ |
| --tool-call-parser cohere_command4 \ |
| --reasoning-parser cohere_command4 \ |
| --enable-auto-tool-choice \ |
| --speculative_config "{\"method\": \"eagle\", \"model\": \"CohereLabs/North-Mini-Code-1.0-eagle\", \"num_speculative_tokens\": 3, \"draft_tensor_parallel_size\": 2}" |
| ``` |
