Accio-Lab/occamy-1.0-MTP

Occamy 1.0 MTP

A BF16 MTP draft head adapted on Occamy data for 512 steps, using frozen Occamy 1.0 predictions as targets. This replaces the earlier 32-step pilot. Download a base checkpoint separately; this is a custom MTP head, not a standalone model or PEFT adapter.

Native three-step MTP now passes the recorded BF16 H200 TP1 correctness checks with the supplied SGLang patch: 4/4 cold/warm comparisons and 20/20 fixed-prompt comparisons have identical output tokens and logprobs. The head weights are unchanged. See Native MTP3 setup and results; performance after the final fix has not been measured.

Training

For NVFP4 with vLLM, see the community setup and assembly notes, including BF16-head quantization exclusions and container mounts. vLLM MTP output parity remains unverified; the results below apply to SGLang.

Initialized from Qwen/Qwen3.6-35B-A3B. Only 8,392,704 fusion and normalization parameters were trained; the rest of the 844,640,768-parameter head and the Occamy base stayed frozen. Training used 256 text examples over two epochs, with 32 heldout examples and assistant-only loss. Windows contain up to 128 context and 382 assistant tokens. Images and unnormalized tool-call roles were excluded; this is bounded text adaptation, not full-head or full-conversation training.

Exported-head target top-1 agreement rose from 73.57% to 74.66%. Data-answer NLL worsened from 2.2850 to 2.6034: matching the target model's greedy choices differs from predicting the recorded answers. No external teacher was used. Exact revisions and settings are in TRAINING.json.

Earlier MTP1 validation

The same head was tested on BF16 and Occamy NVFP4. Each variant used 10 authored prompts and 8 Occamy heldout prefixes, repeated twice. Comparisons use the matching precision's no-MTP baseline.

Combination	Occamy heldout accepted / proposed	Heldout acceptance	Exact token sequences	Exact token logprobs
BF16 + original donor head	1128 / 1292	87.31%	36/36	36/36
BF16 + released head	1130 / 1290	87.60%	36/36	36/36
NVFP4 + released head	1178 / 1372	85.86%	36/36	36/36

Across authored and heldout prompts combined, acceptance was 1470/1668 (88.13%) for BF16 and 1506/1756 (85.76%) for NVFP4. The small BF16 acceptance gain does not establish a meaningful speed improvement. Timings had concurrent activity and are diagnostic only. NVFP4 has no same-run original-donor-head comparison. Some heldout generations hit the 160-token limit; parity applies to the measured outputs, not unbounded continuations. These checks do not establish broad task accuracy.

The earlier validation below covers single-request greedy MTP1, with two verification nodes, context length 2048, deterministic inference, and the supplied hooks. CUDA graphs, overlap and prefix caching were disabled. FP8, GGUF, sampling, concurrency, MTP2, multimodal MTP and other engines are not validated with this head. Results for older heads do not apply to it.

MTP1 runtime

Tested with SGLang 0.5.13.post1, and PyTorch 2.11.0+cu130. Use canonical_attention:install from the included runtime/ directory. The hooks prevent stale prefill-state initialization during verification and reuse decode kernels for recurrent and full attention. Unsupported batch/tree shapes fail closed. Exact parity is bounded evidence for this configuration, not a universal guarantee or a claim about unmodified SGLang.

MTP1 loading

Download this repository and the desired base checkpoint separately. Run the following from this repository's directory. The helper creates a separate assembled directory with links to existing files; it does not modify the base.

MTP_BASE_DIR=/path/to/occamy-bf16
python assemble_head.py \
  --base "$MTP_BASE_DIR" \
  --head ./mtp-trained.safetensors \
  --out ./occamy-with-mtp

PYTHONPATH="$PWD/runtime${PYTHONPATH:+:$PYTHONPATH}" python -m sglang.launch_server \
  --model-path ./occamy-with-mtp --tokenizer-path "$MTP_BASE_DIR" \
  --host 127.0.0.1 --port 30000 --dtype bfloat16 \
  --context-length 2048 --max-running-requests 1 --max-total-tokens 2048 \
  --max-mamba-cache-size 4 --mem-fraction-static 0.50 \
  --attention-backend triton --mamba-backend triton --mm-attention-backend sdpa \
  --disable-cuda-graph --disable-overlap-schedule --disable-radix-cache \
  --random-seed 42 --enable-deterministic-inference --moe-runner-backend triton \
  --speculative-algorithm NEXTN --speculative-num-steps 1 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
  --forward-hooks '[{"name":"mtp-strict","target_modules":[""],"hook_factory":"canonical_attention:install","config":{"output":"./mtp-strict-audit.jsonl"}}]'

For Occamy NVFP4, assemble a new directory using that base instead. Replace --moe-runner-backend triton with:

--quantization modelopt_fp4 --moe-runner-backend flashinfer_cutlass --speculative-moe-runner-backend triton

The base needs sufficient device memory for the selected memory fraction. The runtime hooks use SGLang internals; different builds or settings require revalidation.

Files

• mtp-trained.safetensors: the draft head; download the base model separately.
• assemble_head.py, mtp_head_config.json, runtime/: loading helpers and required runtime hooks.
• Validation results and training details.

Apache 2.0. Full reproduction notes are in Technical details.