sdougbrown/North-Mini-Code-1.0-dflash

North Mini Code 1.0 DFlash

This is a research-preview auxiliary draft checkpoint for speculative decoding with CohereLabs/North-Mini-Code-1.0-w4a16. It is not a standalone language model and it cannot generate text without the North verifier/target model. It will likely also work with CohereLabs/North-Mini-Code-1.0-fp8 with slightly lower acceptance rates.

Training

This DFlash draft model was trained on 10,000 Magicoder prompts with on-policy supervision from North Mini Code.

Important Notes

• not a base model
• not a fine-tuning target for standalone inference
• not a correctness guarantee
• not byte-parity with target-only greedy decoding

Architecture

• speculator type: DFlashDraftModel
• draft layers: 5 Qwen3 sliding-window layers
• hidden size: 2048
• attention heads: 32
• key/value heads: 4
• head dim: 128
• MLP intermediate size: 768
• sliding window: 2048
• draft vocabulary: 32,000
• target vocabulary: 262,144
• block size: 8
• sample_from_anchor: false
• auxiliary target hidden-state taps: layers [2, 24, 46]
• tensors: 62
• serialized tensor elements: 733,274,624
• BF16 parameters: 732,980,480
• selected weight hash: 0ac5964c48003f2eab08c4691357dedc11227d88302ea7c76fc47aaddab6d5b9

Largest tensors include the target embedding matrix, lm_head.weight, and the draft fc.weight projection. The tensor inventory is recorded in tensor_info.json.

Training data and supervision

• prompt dataset: ise-uiuc/Magicoder-Evol-Instruct-110K
• license: Apache-2.0
• training rows: 10,000
• selected rows: 0-499, 600-5099, and 6100-11099
• locked holdout: rows 500-599, never trained
• supervision: on-policy responses and target hidden states generated once from CohereLabs/North-Mini-Code-1.0-w4a16
• response cap: 2,048 tokens
• target layers used for supervision: [2, 24, 46]
• later acceptance-mined continuation data: not included in this release checkpoint

Training metadata is recorded in training_metadata.json.

Matched GB10 benchmarks

These numbers come from the same matched serving contract on a GB10 (DGX Spark) machine running Marlin (which is suboptimal but predictable). Compare the rows within the same host and contract only. These absolute token/s values are not representative of maximum performance, simply a demonstration that the draft model works. :)

Contract	Target-only	DFlash K3	DFlash / target	DSpark K4	DSpark / target
C1	39.9129 tok/s	75.5106 tok/s	1.8919x	79.7341 tok/s	1.9977x
C2	78.9486 tok/s	121.5232 tok/s	1.5393x	127.6174 tok/s	1.6165x

Model	C1 EAL	C1 draft acceptance	C2 EAL	C2 draft acceptance
DFlash K3	2.2947	43.16%	2.2856	42.85%
DSpark K4	2.4800	37.00%	2.4779	36.95%

Expected acceptance length (EAL) is the mean emitted tokens per speculative verification step, including the target bonus token.

Additional result: NVIDIA RTX 3090

DFlash was also tested on an NVIDIA RTX 3090 under the same family of matched conditions. That result is specific to DFlash; do not infer DSpark RTX 3090 portability from it.

• target-only: 110.2714 tok/s
• DFlash K3: 153.7188 tok/s
• official Eagle K3: 77.9583 tok/s

Also being realistic, you wouldn't try to serve this on a single 3090 anyway the model weights leave no room for context.

Default proposal depth

Default proposal depth is K3. That is the selected deployment setting in config.json and the configuration used for the release evidence. More speculative tokens hit diminishing returns past position 3 for this DFlash drafter. You're likely to see the best speed here.

Runtime boundary

This checkpoint is served as the top-level model because config.json already embeds the verifier reference. Use it only with the companion North runtime, published at https://github.com/sdougbrown/north-mini-code-draft-runtime; this card does not describe a separate target-model --speculative_config deployment. The companion is a thin overlay on official vLLM 0.27.1 carrying PRs #49819 and #50937 (Cohere2MoE auxiliary hidden states, and an expert-bias loading fix), published as ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937.

export DRAFT_MODEL="${DRAFT_MODEL:?set this to the downloaded DFlash directory or Hub model ID}"
export VLLM_USE_V2_MODEL_RUNNER=1
vllm serve "${DRAFT_MODEL}" \
  --tensor-parallel-size 1 \
  --max-model-len 320000 \
  --tool-call-parser cohere_command4 \
  --tokenizer-mode cohere \
  --cohere-format cmd4 \
  --reasoning-config '{"reasoning_start_str":"<|START_THINKING|>","reasoning_end_str":"<|END_THINKING|>"}' \
  --enable-auto-tool-choice

Limitations

• 32K draft vocabulary, not the 262,144-token target vocabulary
• target-dependent: cannot run standalone
• quantized greedy outputs are not guaranteed to match target-only greedy byte-for-byte
• throughput and acceptance do not establish correctness or response quality
• absolute throughput is host-specific

References

• North base model: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-w4a16
• Magicoder dataset: https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K
• Companion North runtime: https://github.com/sdougbrown/north-mini-code-draft-runtime
• Companion image: ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937
• Official Eagle card: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-eagle
• Speculators: pinned commit f7ec34182826bc89934ce710283421778022b74d, version 0.6.0

Files in this repo

• README.md
• config.json
• config.py
• model.safetensors
• tensor_info.json
• training_metadata.json
• SHA256SUMS
• LICENSE
• .gitattributes