kingjones777/OxCoder-9B-MTP-ROCmFP4-GGUF

OxCoder-9B — ROCmFP4 for AMD Strix Halo (gfx1151)

ROCmFP4 / ROCmFPX quantizations of OrionLLM/OxCoder-9B — 9,409,813,744 parameters (BF16), a 9B coding model (32-layer dense Qwen3.5 hybrid: 24 Gated DeltaNet linear-attention and 8 full-attention layers, every 4th being full attention; hidden 4096, FFN 12288, 16 attention / 4 KV heads, 262,144-token context), text + image — built and measured on an AMD Ryzen AI Max+ 395 (Radeon 8060S, gfx1151). The checkpoint ships no mtp.* tensors (it declares mtp_num_hidden_layers: 1); every file here carries a grafted in-file MTP head from the declared base model.

• Grafted MTP head inside every file. Measured on ROCm0 at n-max 3: 64.36 tok/s vs 38.50 without it (1.67×), draft acceptance 0.743 (Speed).
• Prompt caching with MTP needs the included server patch in recipe/patches/ — without it, a build of this ROCmFPX line reuses nothing while the head is loaded (measured).
• Vision projector included.
• Importance-matrix builds of the same three 4-bit tiers: kingjones777/OxCoder-9B-MTP-ROCmFP4-imatrix-GGUF.

Which file should I use?

AMD Ryzen AI Max+ 395 (Radeon 8060S) (MAX-1), ROCm 7.2.4, patched llama-server (see Quick start), -c 65536, one request at a time (--parallel 1), greedy (temp 0, top_k 1), ignore_eos so every arm generates exactly 256 tokens after a code prompt of 7,094–7,102 tokens (the first 30,000 characters of convert_hf_to_gguf.py plus an instruction), a unique nonce per request and cache_prompt: false (cache_n = 0 asserted on every timed request), 1 warm-up then the median of 3. Decode numbers are the server's own predicted_per_second. Box iced: no other model loaded (except the rows marked †, see Vulkan0 prefill and CPU activity).

File	ftype	Size	BPW⁴	KLD vs BF16 ↓²	Same top-1 ↑	PPL (× BF16)	TG MTP ROCm0	TG MTP Vulkan0	PP ROCm0
OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf	106	5.03 GiB	4.69	0.0879 ± 0.0034	89.29 %	6.7713 ± 0.0829 (×1.0185)	65.80	60.98‡	898
OxCoder-9B-MTP-Q4_0-ROCmFP4-COHERENT.gguf	102	5.34 GiB	4.97	0.0857 ± 0.0033	89.31 %	6.6389 ± 0.0802 (×0.9986)	57.40†	60.53‡	803†
OxCoder-9B-MTP-Q4_0-ROCmFP4-FAST.gguf	103	4.85 GiB	4.52	0.0890 ± 0.0033	88.94 %	6.8166 ± 0.0835 (×1.0253)	64.81†	61.72	899†
BF16 reference	32	17.14 GiB³	16.00	0	100 %	6.6482 ± 0.0810¹	—	—	—

TG = decode tokens/s after the 7,094–7,102-token code prompt, grafted MTP head, --spec-draft-n-max 3. PP = prefill tokens/s on ROCm0. ¹ The BF16 PPL shown is the paired base every "×" ratio is computed against (averaged over the same scored tokens in the KL-divergence runs). The standalone BF16 run's own summary line reads 6.6532 ± 0.0813. ² Quality columns: see Where the quality numbers come from below. ³ BF16 conversion of the grafted checkpoint (MTP block included); not published. ⁴ BPW as printed by llama-quantize: bits per weight over the 9,197,093,888 weights in each GGUF — the checkpoint's 8,953,803,264 text weights plus the grafted 243,290,624-weight MTP block. The 9,409,813,744-parameter count above is the checkpoint's: it includes the 456,010,480-weight vision tower, which ships in the projector file, and no MTP block. † Quiet re-run: the first measurement had requests inside a window when an unrelated job ran on this box; re-run later with other model servers loaded but idle — see Vulkan0 prefill and CPU activity. ‡ Has requests inside a window when an unrelated job ran on this box; kept as measured — see Vulkan0 prefill and CPU activity.

Start with STRIX_LEAN. COHERENT's KLD is within noise of it (-2.4 %, 0.4σ) for 315 MiB more; STRIX_LEAN decodes 14.6 % faster than COHERENT on ROCm0 (65.80 vs 57.40† tok/s; per-run ranges 64.36–65.83 and 55.67–58.10) and within 0.7 % of it on Vulkan0 (60.98‡ vs 60.53‡), and prefills 11.8 % faster than it on ROCm0 (897.8 vs 803.4† tok/s).

FAST does not buy a clear speed gain here: FAST decodes within 1.5 % of STRIX_LEAN on ROCm0 (64.81† vs 65.80 tok/s; per-run ranges 62.61–68.03 and 64.36–65.83) and within 1.2 % of it on Vulkan0 (61.72 vs 60.98‡), and prefills within 0.1 % of it on ROCm0 (898.6† vs 897.8 tok/s); its KLD is within noise of STRIX_LEAN's (0.0890 vs 0.0879, +1.3 %, 0.2σ).

KLD order (lower is closer to BF16; KLD gaps under 2σ are called within noise): COHERENT 0.0857, STRIX_LEAN 0.0879 (+2.5 % vs COHERENT), FAST 0.0890 (+3.8 % vs COHERENT). STRIX_LEAN is 315 MiB smaller than COHERENT; FAST is 178 MiB smaller than STRIX_LEAN. Speed gaps are called a tie below 10.3 % for decode and below 3.0 % (ROCm0) / 3.0 % (Vulkan0) for prefill (3.0 % when the two backends are compared): the larger of 3 % and the widest gap measured between two files that run identical tensors — each imatrix file and its standard twin at the MTP knee (decode 10.3 %; prefill 1.3 % on ROCm0, 0.2 % on Vulkan0). The decode floor is the widest because each twin's drafts are accepted at a different rate (STRIX_LEAN on ROCm0: 0.750 standard vs 0.641 imatrix): the imatrix changes the main model's weights, so the head sees different hidden states and is checked against different target tokens.

The imatrix build is measurably closer to BF16 at the same size: STRIX_LEAN 0.0879 → 0.0610 (-30.6 %, 6.2σ); COHERENT 0.0857 → 0.0550 (-35.9 %, 7.4σ); FAST 0.0890 → 0.0637 (-28.4 %, 5.8σ).

Quality is graded against the BF16 GGUF (reference logits computed on CPU) on a held-out corpus (wikitext-2 test, -c 2048, 40 chunks × 1,023 scored tokens each — the second half of every window, less its first token — = 40,920), never on the imatrix calibration text. KLD is the per-token KL divergence of each quant's next-token distribution from BF16's on the same tokens — far more sensitive than perplexity.

Where the quality numbers come from. Measured directly on these files, against BF16 logits computed on CPU in the same session. The BF16 model's own first-window perplexity was CPU 6.6290, Vulkan0 6.6346 (+0.08 %), ROCm0 6.6463 (+0.26 %): the two GPU backends disagree with CPU and with each other, so the reference logits come from pure CPU (-dev none -ngl 0 --no-op-offload), and that run's own first-window value (6.6290) equals the CPU control's. Every file was graded on ROCm0 (the columns above); the standard STRIX_LEAN file was also graded on Vulkan0. The standard STRIX_LEAN grade was run twice on ROCm0 and all 40 per-chunk rows matched.

Same file, same reference, graded on each backend — standard STRIX_LEAN KLD ROCm0 / Vulkan0: 0.0879 / 0.0886 (+0.9 %, 0.2σ). The two backends agree within noise on this file.

Quick start

1. Download

hf download kingjones777/OxCoder-9B-MTP-ROCmFP4-GGUF --local-dir ~/models/oxcoder

2. Build llama-server with the prompt-cache patch — ROCmFPX at the measured commit (ROCm and Vulkan prerequisites: the project's build guide)

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout d3ca53726109b46fd5cd6fa42f688c707079b44e
git apply ~/models/oxcoder/recipe/patches/mtp-prompt-cache-fix.patch
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF
cmake --build build --target llama-server -j

(The CMake options of the measured build are listed in Reproduction.)

d3ca537 is also in the history of the official ROCmFPX/ROCmFPX repository; the patch was verified against d3ca53726109b46fd5cd6fa42f688c707079b44e only.

3. Serve — the MTP head is inside the model file, so there is no --model-draft:

Text and tool calls, with MTP:

env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
build/bin/llama-server \
  -m ~/models/oxcoder/OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.0 \
  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
  -c 65536 --host 127.0.0.1 --port 8080

Images (--mmproj), without MTP:

env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
build/bin/llama-server \
  -m ~/models/oxcoder/OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj ~/models/oxcoder/mmproj-OxCoder-9B-BF16.gguf \
  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
  -c 65536 --host 127.0.0.1 --port 8080

⛔ Images: with the MTP head loaded, image requests fail on this build (measured). For image input, drop the two --spec-* lines.

(LD_LIBRARY_PATH avoids a soname clash on machines that also have a Vulkan-only llama.cpp build.) The exact measured argv is in Reproduction.

4. Call — thinking off (OxCoder's SWE-bench sampling: temperature 1.0, top_p 0.95):

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Flag	Why
--spec-type draft-mtp	Uses the grafted MTP layer inside the model file; no second file to download or place.
--spec-draft-n-max 3	llama.cpp's default is 16. On this model the measured knee is n-max 3 — see Speed.
--spec-draft-p-min 0.0	The default 0.75 gate throttles drafting.
--jinja	Already on by default in this build; keep it on — the thinking controls (chat_template_kwargs) and tool calling rely on the Jinja chat template.
-fit off	Autofit reads MemAvailable on integrated GPUs and can silently shrink context or push tensors to CPU.
-cram <MiB>	Not set above (default 8 GiB of host RAM for saved prompts). Set it on a shared box — see Known issues.
--mmproj	Loads the 27-layer vision tower. Drop the flag for text-only.

⚠️ On an unpatched build the in-file head still drafts, but prompt caching turns off while it is loaded — every turn re-prefills the whole conversation (measured). For multi-turn / agentic use there, drop the two --spec-* lines.

Requires a llama.cpp build with ROCmFP4 / ROCmFPX tensor-type support; stock llama.cpp rejects these tensor types.

Reasoning controls

OxCoder ships its own chat template (template_version qwen3.6-froggeric-v21.3, in the source repo's chat_template.jinja). It reads these chat_template_kwargs:

kwarg	default	effect
enable_thinking	true	true: the generation prompt opens <think>\n; false: it emits an empty <think>\n\n</think> block (no reasoning)
preserve_thinking	true	earlier assistant turns are re-rendered with their reasoning, so multi-turn contexts grow faster than with templates that drop it
auto_disable_thinking_with_tools	false	true switches thinking off whenever tools are passed
tool_call_format	xml	the tool-call format the system prompt asks for: <tool_call><function=…><parameter=…>; json asks for a JSON object inside <tool_call> instead (not tested here)
max_tool_arg_chars / max_tool_response_chars	0 (off)	truncate long arguments of earlier tool calls / long tool results, noting the original length

A <|think_off|> or <|think_on|> tag inside a system or user message overrides enable_thinking (the last tag wins) and is removed from the text.

A short tool result that starts like an error (error:, Traceback, fatal:, command not found …) gets a "⚠️ SYSTEM WARNING" appended by the template; after two such results in a row (no user turn in between), the next generation prompt also gets an empty think block, regardless of enable_thinking. A developer message is treated as system; any other unknown role is rendered as a user turn [role]: content.

With thinking on and a small max_tokens, the whole budget can go to reasoning and content comes back empty — raise max_tokens before concluding the model is broken.

Tool calls in the default format are parsed by llama.cpp natively through the Jinja chat template (on by default). OxCoder's own card reports SWE-bench at temperature 1.0 / top_p 0.95 and Terminal-Bench / NL2Repo at temperature 1.0 / top_p 1.0. The tools gate on this card used temperature 1.0, top_p 0.95, top_k disabled, max_tokens 8192.

Speed: making it fast

Measured on OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf (patched server; 7,094–7,102-token code prompt):

Config	Decode tok/s (min–max)	Draft acceptance	Prefill tok/s
ROCm0 · no draft head	38.50 (38.14–38.54)	—	1089
Vulkan0 · no draft head	38.65 (38.63–38.65)	—	668
ROCm0 · grafted MTP, n-max 1	55.06 (54.99–55.15)	0.937	906
ROCm0 · grafted MTP, n-max 2	62.32 (60.82–62.40)	0.826	904
ROCm0 · grafted MTP, n-max 3	64.36 (64.32–65.00)	0.743	901
ROCm0 · grafted MTP, n-max 4	60.91 (60.53–61.13)	0.651	898
ROCm0 · grafted MTP, n-max 5	58.31 (58.19–58.79)	0.584	896
ROCm0 · grafted MTP, n-max 6	55.45 (55.26–56.73)	0.526	896
Vulkan0 · grafted MTP, n-max 2	60.98 (60.95–61.10)	0.770	605
Vulkan0 · grafted MTP, n-max 3	61.95 (61.89–61.97)	0.684	615
Vulkan0 · grafted MTP, n-max 4	60.82 (59.47–62.03)	0.628	620
ROCm0 · DFlash drafter, n-max 4	48.99 (47.81–49.24)	0.615	982
ROCm0 · DFlash drafter, n-max 8	41.26 (40.14–41.90)	0.341	981
ROCm0 · DFlash drafter, n-max 15	37.74 (34.47–38.74)	0.168	977

On ROCm0, n-max 3 is the fastest depth of the n-max 1–6 sweep: 64.36 tok/s, 1.67× the 38.50 tok/s no-draft baseline, with 0.743 of drafted tokens accepted. On Vulkan0 (n-max 2–4 measured, around the ROCm0 knee) the fastest is n-max 3: 61.95 tok/s, 1.60× its own 38.65 tok/s no-draft baseline, acceptance 0.684 — within 3.7 % of ROCm0 at its knee, a tie. The separate DFlash drafter (z-lab/Qwen3.5-9B-DFlash, Q8_0; n-max 4, 8, 15 measured on ROCm0) peaks at n-max 4: 48.99 tok/s, 1.27× the no-draft baseline, acceptance 0.615 — 23.9 % slower than the grafted MTP head at its knee. Prefill with the head loaded vs without it, same prompt: ROCm0 897.8 vs 1089.2 tok/s; Vulkan0 614.2‡ vs 668.3 tok/s. Speed gaps are called a tie below 10.3 % for decode and below 3.0 % (ROCm0) / 3.0 % (Vulkan0) for prefill (3.0 % when the two backends are compared): the larger of 3 % and the widest gap measured between two files that run identical tensors — each imatrix file and its standard twin at the MTP knee (decode 10.3 %; prefill 1.3 % on ROCm0, 0.2 % on Vulkan0). The decode floor is the widest because each twin's drafts are accepted at a different rate (STRIX_LEAN on ROCm0: 0.750 standard vs 0.641 imatrix): the imatrix changes the main model's weights, so the head sees different hidden states and is checked against different target tokens.

Draft acceptance depends on how predictable the text is, so the same file was also run on prose. STRIX_LEAN, code prompt (7,095–7,101 tokens: the first 30,000 characters of convert_hf_to_gguf.py plus a coding instruction) vs prose prompt (7,450–7,458 tokens: the first 34,000 characters of wikitext-2 train plus a writing instruction) — ROCm0 with the head: code 65.80 tok/s vs prose 48.00 (prose decodes 27.1 % slower than code); Vulkan0 with the head: code 60.98‡ tok/s vs prose 49.75 (prose decodes 18.4 % slower than code); ROCm0 without the head: code 38.50 tok/s vs prose 38.46 (prose decodes within 0.1 % of code); Vulkan0 without the head: code 38.65 tok/s vs prose 38.55 (prose decodes within 0.3 % of code). (Same tie rule as above.)

Every 4-bit file at the MTP knee (n-max 3), both backends — separate runs from the depth sweep above (STRIX_LEAN at n-max 3 read 64.36 tok/s on ROCm0 and 61.95 on Vulkan0 there):

File	Backend	Decode tok/s (min–max)	Acceptance	Prefill tok/s
OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf	ROCm0	65.80 (64.36–65.83)	0.750	898
OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf	Vulkan0‡	60.98 (60.86–61.24)	0.672	614
OxCoder-9B-MTP-Q4_0-ROCmFP4-COHERENT.gguf	ROCm0†	57.40 (55.67–58.10)	0.628	803
OxCoder-9B-MTP-Q4_0-ROCmFP4-COHERENT.gguf	Vulkan0‡	60.53 (58.45–60.74)	0.673	709
OxCoder-9B-MTP-Q4_0-ROCmFP4-FAST.gguf	ROCm0†	64.81 (62.61–68.03)	0.738	899
OxCoder-9B-MTP-Q4_0-ROCmFP4-FAST.gguf	Vulkan0	61.72 (61.72–61.81)	0.672	618
OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf	ROCm0	59.03 (57.78–59.12)	0.641	898
OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf	Vulkan0	59.18 (59.18–59.80)	0.639	613
OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-COHERENT.gguf	ROCm0	59.28 (58.16–59.53)	0.665	813
OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-COHERENT.gguf	Vulkan0	63.83 (63.62–64.40)	0.737	609
OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-FAST.gguf	ROCm0	60.01 (58.76–60.11)	0.656	910
OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-FAST.gguf	Vulkan0	61.13 (60.50–61.13)	0.659	617

STRIX_LEAN: ROCm0 decodes within 7.9 % of Vulkan0 (65.80 vs 60.98‡ tok/s; ranges 64.36–65.83 / 60.86–61.24) and prefills 46.2 % faster than it (897.8 vs 614.2‡ tok/s). COHERENT: ROCm0 decodes within 5.2 % of Vulkan0 (57.40† vs 60.53‡ tok/s; ranges 55.67–58.10 / 58.45–60.74) and prefills 13.3 % faster than it (803.4† vs 709.3‡ tok/s; all three Vulkan0 runs overlapped the unrelated job — a quiet re-run read 621.6 tok/s prefill). FAST: ROCm0 decodes within 5.0 % of Vulkan0 (64.81† vs 61.72 tok/s; ranges 62.61–68.03 / 61.72–61.81) and prefills 45.4 % faster than it (898.6† vs 618.0 tok/s). (Same tie rule as above.)

Vulkan0 prefill and CPU activity

While the per-file rows were being measured, an unrelated job ran on this box from 01:50:50 to 01:53:40 UTC: one niced, single-threaded process re-rendering another model's cards, no GPU use (recipe/logs/O6_background_job_journal.txt). Rows with requests inside that window, each against the closest like-for-like requests outside it: STRIX_LEAN on Vulkan0, run 3: 718 tok/s prefill vs 610–614 for runs 1 and 2 of the same row; COHERENT on Vulkan0, all three runs: 709 tok/s prefill vs 604–609 for its imatrix twin (identical tensor types, outside the window); COHERENT on ROCm0, all three runs: 809–811 tok/s prefill vs 810–815 for its imatrix twin (identical tensor types, outside the window); FAST on ROCm0, run 1: 910 tok/s prefill vs 906–910 for runs 2 and 3 of the same row.

Re-run with the same harness and flags, quiet vs with one niced busy-loop thread (whole-box CPU busy 1.2–3.6 % quiet, 4.4–6.7 % with the thread; this re-run was not on an iced box: 5 other model servers were loaded, all idle before and after every arm):

file	backend	quiet: prefill / decode tok/s (acceptance)	busy thread: prefill / decode tok/s (acceptance)	prefill change
STRIX_LEAN	Vulkan0	625 / 62.46 (0.703)	721 / 62.58 (0.684)	+15.3 %
COHERENT	Vulkan0	622 / 59.36 (0.662)	716 / 60.78 (0.669)	+15.1 %
COHERENT	ROCm0	803 / 57.40 (0.628)	801 / 59.34 (0.644)	-0.2 %
imatrix STRIX_LEAN (control)	Vulkan0	632 / 60.39 (0.646)	720 / 60.37 (0.642)	+14.0 %

On this box a single busy CPU thread made Vulkan0 prefill with the MTP head 14–15 % faster; ROCm0 prefill moved -0.2 %. Vulkan0 prefill numbers therefore depend on what else the CPU is doing. A second quiet COHERENT Vulkan0 run after the busy-thread arms read 620 tok/s (first quiet run: 622). The ROCm0 quiet control (imatrix COHERENT, from an undisturbed window) reproduced its original row (prefill 812.9 → 807.4 tok/s, -0.7 %; decode +0.6 %; tolerances ±3 % / ±5 %, set before the re-run), so the COHERENT and FAST ROCm0 rows show the quiet re-run (marked †). The Vulkan0 quiet control (imatrix STRIX_LEAN, from an undisturbed window) did not reproduce its original row within the tolerances set before the re-run (prefill 612.9 → 631.6 tok/s, +3.05 % against ±3 %; decode +2.0 % against ±5 %; the re-run's box state differed: other model servers were loaded), so the STRIX_LEAN and COHERENT Vulkan0 rows keep the original values (marked ‡): STRIX_LEAN's reported medians are values from its runs outside the window (its in-window run 3 is the extreme of the three in both prefill and decode); all three COHERENT runs were inside the window, so its reported values were measured with the job running (its quiet re-run read 622 tok/s prefill and 59.36 tok/s decode). The replaced rows stay in recipe/results_summary.json under bench_superseded.

The grafted MTP head

The OxCoder checkpoint has no mtp.* tensors. The 15 mtp.* tensors of Qwen/Qwen3.5-9B (the declared base_model) were fetched by HTTP range read and written byte-for-byte as an extra safetensors shard next to the untouched OxCoder file; the converter keeps mtp.* as the in-file nextn block. The head was trained for the base model, not for OxCoder.

donor	Qwen/Qwen3.5-9B
donor revision	c202236235762e1c871ad0ccb60c8ee5ba337b9a
tensors	15
total bytes	486,581,248
graft file	model-mtp-graft.safetensors
graft sha256	6a1197089cf65461ac0ebec179e172f922884f13c3fe10e565e520ab4bd76051
every tensor BF16 and shape-checked before writing (the script refuses to write otherwise)	yes

tensor	shape	bytes	sha256	shape checked against
mtp.layers.0.mlp.down_proj.weight	4096×12288	100,663,296	52c72564f7da59c25233b2194a79239cc1e69dd6694130aae87d5fffc472707c	model.language_model.layers.0.mlp.down_proj.weight
mtp.layers.0.mlp.gate_proj.weight	12288×4096	100,663,296	dd3e6d05e9c519ebb16c5eee64b4d4d217d7efb8ae7c88e8dfa96f6e6f7d3eac	model.language_model.layers.0.mlp.gate_proj.weight
mtp.layers.0.mlp.up_proj.weight	12288×4096	100,663,296	79e02e92d4cb775120f48cd523577345b277dacb749e56e2d52532583f92800e	model.language_model.layers.0.mlp.up_proj.weight
mtp.fc.weight	4096×8192	67,108,864	a4639d8f4b81cbdc65c61f1cba82816ae1534ec494a95a0b677dba98f04f4017	2*hidden -> hidden
mtp.layers.0.self_attn.q_proj.weight	8192×4096	67,108,864	17d3aac05cb017e9ef98f93fe567f1c1ffae08fbe5001b80e67829bda189a584	model.language_model.layers.3.self_attn.q_proj.weight
mtp.layers.0.input_layernorm.weight	4096	8,192	c8913bfe7cef186fb59b7f1ca80d391ba5eaec45c9c8db54f54e63f6410ae633	model.language_model.layers.0.input_layernorm.weight
mtp.layers.0.post_attention_layernorm.weight	4096	8,192	63500aa0ed93918f6f3d5f1bfd1656921543931392a06267f81c0b3ad680fdcf	model.language_model.layers.0.post_attention_layernorm.weight
mtp.layers.0.self_attn.k_norm.weight	256	512	8f5a8071649fa9a7b6b6afcea6d5404cd849640be2c4f90e6759c4c0a60120ac	model.language_model.layers.3.self_attn.k_norm.weight
mtp.layers.0.self_attn.k_proj.weight	1024×4096	8,388,608	1b0129a3c6f4add6c8417d67e84050f8595f96f7693511beaace9ef56d1ad30a	model.language_model.layers.3.self_attn.k_proj.weight
mtp.layers.0.self_attn.o_proj.weight	4096×4096	33,554,432	31140d974d94190127d0f3d908a18bafe5fba0d8c3d7f522e012013795c89924	model.language_model.layers.3.self_attn.o_proj.weight
mtp.layers.0.self_attn.q_norm.weight	256	512	10e6c9fa42ceb72373c207422c305663a89dfd08f5332cbbe6d1057301fe57d1	model.language_model.layers.3.self_attn.q_norm.weight
mtp.layers.0.self_attn.v_proj.weight	1024×4096	8,388,608	c21332947c44203ca197d014de23559bf6e88d7c0debca8d4ecad735d717e2c9	model.language_model.layers.3.self_attn.v_proj.weight
mtp.norm.weight	4096	8,192	7e4daf06ad25b834b3b95c9592fa690b75a4a90fb9f4128ccee8537d7f15988e	hidden-size norm
mtp.pre_fc_norm_embedding.weight	4096	8,192	8b9bca62497def4783b20dfd58ddf32573a255b44a79f0acf8ca9c611b7bea15	hidden-size norm
mtp.pre_fc_norm_hidden.weight	4096	8,192	a7410736f9962dd6ebd8ac2b4c355898d057296a03c47795223b7298cac8988c	hidden-size norm

File	nextn tensors (read-back)	MTP block (blk.32.*)	MTP block bytes
BF16	4	15	486,623,232
OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf	4	15	147,408,896
OxCoder-9B-MTP-Q4_0-ROCmFP4-COHERENT.gguf	4	15	153,700,352
OxCoder-9B-MTP-Q4_0-ROCmFP4-FAST.gguf	4	15	147,146,752
OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf	4	15	147,408,896
OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-COHERENT.gguf	4	15	153,700,352
OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-FAST.gguf	4	15	147,146,752

At n-max 3 on ROCm0, the grafted head's measured acceptance was 0.743 and STRIX_LEAN decoded at 64.36 tok/s — 1.67× the 38.50 tok/s no-draft baseline on the same backend and the same prompt. The head was trained for Qwen3.5-9B, not for OxCoder; those figures are measurements on the benchmark prompts above, not a claim about other prompts.

Is the output the same as without MTP?

Greedy output with MTP matched greedy output without a draft head on 2 of 6 prompts (every pair run cold at temperature 0). The 6 prompts are the identity gate's fixed prompt (1) plus the 5 prompts the aligned cache gate also ran without a draft head (o-c3-* rows; cold reply compared with cold reply). Without a draft head, the unpatched and patched servers produced identical output on the fixed prompt. A mismatch is not a quality regression: with a draft head loaded the server evaluates the same math in different batch shapes (drafts are verified several tokens at a time, and the head needs the hidden state of every prompt position), so float rounding can differ and a near-tie can break the other way; from that token on, the two replies diverge. If you need output that is byte-for-byte reproducible against a no-MTP run, serve without --spec-type.

Prompt caching with MTP

Measured: a second turn that shares the first turn's long prefix and differs only in its closing instruction, each exchange run warm (cache_prompt: true) and then cold (cache_prompt: false) and the two replies compared byte for byte. Every prompt in the patched gate is padded to one token length so warm and cold see identical chunking.

server	draft head	second-turn prompt tokens reused	processed	warm reply = cold reply
d3ca537, unpatched	grafted MTP	0 of 7,067	7,067	—
d3ca537 + patch	grafted MTP	6,011 of 7,039 (at the aligned checkpoint in 5/5 exchanges)	1,028	5/5
d3ca537 + patch	none (control)	6,011 of 7,039 (at the aligned checkpoint in 5/5 exchanges)	1,028	5/5

⛔ Unpatched, loading the draft head turns prompt reuse off. On the unmodified d3ca537 server with the grafted head loaded, a second turn sharing a long prefix reused 0 of its 7,067 prompt tokens.

✅ Patched, reuse works. In 5 two-turn exchanges at a fixed prompt length of 7,039 tokens, every second turn resumed from the checkpoint 1,028 tokens before its end — 6,011 tokens reused (85 %), 1,028 processed — median prefill 1.3 s instead of 7.8 s cold (6.2× faster), with the draft head still loaded.

Resuming does not change the output: each warm reply was byte-identical to a cold run of the same prompt in 5/5 exchanges with the grafted head. Patched with no draft head (control): warm = cold in 5/5 exchanges.

One behaviour to know about, which belongs to llama-server rather than to this patch: the server processes the last n_ubatch + 4 tokens of every prompt as two separate batches so it can checkpoint there (1,024 + 4 = 1,028 tokens with the -ub 1024 used in these measurements, where the server default is -ub 512; upstream PR #20288). A turn that resumes from a checkpoint left by a prompt of a different length therefore splits its tail differently from a cold run, and float rounding can flip a greedy token. Both replies are ordinary greedy decodes; they are just not bit-identical to each other. That is why every prompt in this gate has the same length.

Tool calling

The template emits the XML-style <tool_call><function=…><parameter=…> format, which llama.cpp parses natively through the Jinja chat template (on by default). Suite run through llama-server in the deployed configuration (grafted MTP on):

14/14 on native tool_calls (raw XML left in content counts as a failure), run on OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf with the grafted head on, at OxCoder's SWE-bench sampling (temperature 1.0, top_p 0.95, top_k disabled, max_tokens 8192).

check	thinking ON (enable_thinking: true)	thinking OFF (enable_thinking: false)
multi-arg	✅	✅
nested-object	✅	✅
enum	✅	✅
correct-decline	✅	✅
multi-turn	✅	✅
streaming	✅	✅
parallel	✅	✅

Every check passed in both thinking modes (enable_thinking: true and false).

Vision

mmproj-OxCoder-9B-BF16.gguf is the 27-layer vision tower (width 1152), loaded with --mmproj. Its attention follows the server's -fa setting, so both settings were checked, with and without the MTP head.

Probe: a synthetic 448×448 image with a red circle and a blue square (a model that ignores the image cannot name both), sent to OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf on the patched server with --mmproj, temperature 0. Pass = the reply names every expected term (red, blue, circle, square).

MTP head	-fa on	-fa off
loaded	❌ server stopped	❌ server stopped
not loaded	✅ 4/4 terms	✅ 4/4 terms
loaded, unpatched server (control)	❌ server stopped	—

⛔ With the MTP head loaded, image requests fail with -fa on and with -fa off — the server aborted (process: missing MTP boundary for seq_id=0 pos=17 (current=3/1 previous=2/1) → update_slots: failed to process speculative batch). Without the head they work. For image input, serve without --spec-type. The unmodified d3ca537 server aborts the same way, so this is not caused by the prompt-cache patch; the server log points to the MTP state not being advanced across an image's positions, which jump (M-RoPE).

Reply (no head, -fa on; first 300 characters as stored by the harness):

The image contains two distinct geometric shapes placed side by side on a white background:

1. Left Shape: A solid red circle. It is perfectly round with no visible outline or gradient — just a flat, uniform red fill.

2. Right Shape: A solid blue square. It has sharp, straight edges and ri

Memory

Measured footprint (drop in MemAvailable) of STRIX_LEAN on ROCm0 with the grafted MTP head loaded (n-max 3, --spec-draft-p-min 0.0), the vision projector, q8_0 KV cache, --no-mmap, -cram 512, one slot, patched server. After one request is after a single request whose prompt is the first 30,000 characters of convert_hf_to_gguf.py plus a one-line instruction, with 64 tokens generated.

context	after load	after one request
65,536	10.82 GiB	16.46 GiB
262,144	17.37 GiB	22.45 GiB

No row was decode-benchmarked beyond that one request. Nothing beyond these rows was measured.

Quantization methodology

# 0. graft the 15 mtp.* tensors of Qwen/Qwen3.5-9B onto the OxCoder checkpoint (byte-for-byte extra shard):
#    hf/ is the untouched OxCoder download, graft/ a hard-linked copy of it plus the new shard
cp -al hf graft && rm -rf graft/.cache
python3 oxcoder_graft.py hf graft/model-mtp-graft.safetensors

# 1. convert: grafted checkpoint WITH its MTP layer (the converter's default), and the vision projector
python convert_hf_to_gguf.py graft --outtype bf16 --model-name OxCoder-9B --outfile OxCoder-9B-MTP-BF16.gguf
python convert_hf_to_gguf.py graft --outtype bf16 --mmproj --model-name OxCoder-9B --outfile mmproj-OxCoder-9B-BF16.gguf

# 2. quantize from BF16 only; the LM head is forced up on every tier and read back by exact tensor name
llama-quantize --output-tensor-type q6_K                            OxCoder-9B-MTP-BF16.gguf OUT Q4_0_ROCMFP4_STRIX_LEAN 8
llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K OxCoder-9B-MTP-BF16.gguf OUT Q4_0_ROCMFP4_COHERENT 8
llama-quantize --output-tensor-type q6_K                            OxCoder-9B-MTP-BF16.gguf OUT Q4_0_ROCMFP4_FAST 8

# 3. BF16 reference logits on CPU (chunk-1 control, BF16 first-window PPL: CPU 6.6290, Vulkan0 6.6346, ROCm0 6.6463)
llama-perplexity -m OxCoder-9B-MTP-BF16.gguf -f wikitext-2-raw/wiki.test.raw -c 2048 -b 2048 --chunks 40 --kl-divergence-base bf16.kld \
  -dev none -ngl 0 --no-op-offload -t 16
# 4. grade each shipped file against those logits on ROCm0 (STRIX_LEAN also on Vulkan0)
llama-perplexity -m OUT --kl-divergence-base bf16.kld --kl-divergence -c 2048 -b 2048 -ngl 999 -fa on -dio -dev ROCm0
llama-perplexity -m OUT --kl-divergence-base bf16.kld --kl-divergence -c 2048 -b 2048 -ngl 999 -fa on -dio -dev Vulkan0

# 5. DFlash drafter (not redistributed): this fork's converter, then Q8_0
python convert_hf_to_gguf.py dflash-hf --outtype bf16 --target-model-dir graft \
  --outfile gguf/dflash-Qwen3.5-9B-BF16.gguf
llama-quantize gguf/dflash-Qwen3.5-9B-BF16.gguf out/dflash-Qwen3.5-9B-Q8_0.gguf Q8_0 16

Receipts (the built file is the receipt — exact tensor names, never a substring match; recipe/logs/):

File	output.weight	token_embd.weight	tensors	nextn
OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf	Q6_K	Q5_K	442	4
OxCoder-9B-MTP-Q4_0-ROCmFP4-COHERENT.gguf	Q6_K	Q6_K	442	4
OxCoder-9B-MTP-Q4_0-ROCmFP4-FAST.gguf	Q6_K	Q4_0_ROCMFP4_FAST	442	4

Where the three tiers differ, read from the standard files (blk.N = every layer that has the tensor, the MTP block included; families not listed have the same type in all three):

tensor family	tensors	STRIX_LEAN	COHERENT	FAST
blk.N.attn_gate.weight	24	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.attn_k.weight	9	Q4_0_ROCMFP4	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.attn_output.weight	9	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.attn_q.weight	9	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.attn_qkv.weight	24	Q4_0_ROCMFP4	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.attn_v.weight	9	Q4_0_ROCMFP4	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.ffn_down.weight	33	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.ffn_gate.weight	33	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.ffn_up.weight	33	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.ssm_alpha.weight	24	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.ssm_beta.weight	24	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
blk.N.ssm_out.weight	24	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4	Q4_0_ROCMFP4_FAST
output.weight	1	Q6_K	Q6_K	Q6_K
token_embd.weight	1	Q5_K	Q6_K	Q4_0_ROCMFP4_FAST

tie_word_embeddings is false, so the output head is a real tensor and --output-tensor-type q6_K does real work. All three tiers pin output.weight to q6_K; COHERENT also pins token_embd.weight to q6_K, while STRIX_LEAN and FAST keep their tier's own embedding type (shown in the receipts).

Reproduction

server   : https://github.com/charlie12345/ROCmFPX @ d3ca53726109b46fd5cd6fa42f688c707079b44e
           + recipe/patches/mtp-prompt-cache-fix.patch (sha256 412a0173f60faa937018f0153efe9b5d9230c1606735502485e7e520fffb68dd)
           built target llama-server with the same CMake options as below; sha256 14e9f1299240c891478c820b8c20811ca5721044d2e63e23799eaa9be26b30bd
tools    : same commit, unpatched (quantize / imatrix / perplexity; the patch touches only common/ and tools/server/)
           build dir /opt/llama-rocm/rocmfpx-724/build-hipvk, Release, Unix Makefiles, GGML_HIP=ON GGML_VULKAN=ON
           GGML_HIP_GRAPHS=ON GGML_HIP_NO_VMM=ON GGML_NATIVE=ON AMDGPU_TARGETS=gfx1151 LLAMA_CURL=OFF
           (this CMake forwards AMDGPU_TARGETS to GPU_TARGETS, the spelling used in Quick start)
           CMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang
           sha256 llama-quantize   1559189968a30781f837e4d25ca3cdf26c8eb0351023675406535eddcbe0db13
           sha256 llama-imatrix    bfdce5c21a01bdc8043d69335b74a7b68b8335ae989991a1e10e21d5faeaeb8e
           sha256 llama-perplexity c2175925bb0c61df89ff787845d8e03a35403f684be52719ac2987b8ca636a53
           sha256 llama-server     d077b98f3f04df4b482a5826c8922e39cda3aa8d2e2b2fbe1870f36d245cebe7   (unpatched; used for the unpatched control rows: prompt cache and vision)
source   : OrionLLM/OxCoder-9B  revision 07ba04e2149bace1b43c66448471a70378a7a708
model    : OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf (the argv below; every file was measured the same way)
argv     : /opt/llama-rocm/rocmfpx-724-mtpcache/build-hipvk/bin/llama-server -m /mnt/models/oxcoder-9b/out/OxCoder-9B-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf -dev ROCm0 -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -c 65536 -b 2048 -ub 1024 --host 127.0.0.1 --port 18600 --no-webui --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.0
env      : LD_LIBRARY_PATH=<patched build>/bin:/opt/rocm-7.2.4/lib
           HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1
box      : aimax — AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (gfx1151), 124 GiB RAM, GTT 131072 MiB,
           kernel 6.17.6-061706-generic, ROCm 7.2.4
protocol : 7,094–7,102-token code prompt, 256 generated tokens, temp 0 / top_k 1, ignore_eos, cache_prompt false,
           1 warm-up + median of 3, no co-resident models (box iced)
           (rows marked † were re-run later, quiet, with other model servers loaded but idle)
measured : 2026-09-16 to 2026-09-17, by the pipeline in recipe/ (every raw number in recipe/results_summary.json and recipe/raw/)

Files

File	Size	sha256
OxCoder-9B-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf	5.03 GiB	1af457fc6f292b98cdaa072295fe3cb6c11293d275914fd706146fbbd23ae5fb
OxCoder-9B-MTP-Q4_0-ROCmFP4-COHERENT.gguf	5.34 GiB	46c811b654aba00c1b4d1f89ee09f39868a892d2aa344950205123bdea03f817
OxCoder-9B-MTP-Q4_0-ROCmFP4-FAST.gguf	4.85 GiB	5b96e01da43361c21f56d37d121e448f2715aa082b9c6b9bc0d128fbaa65bf39
mmproj-OxCoder-9B-BF16.gguf	0.86 GiB	5c3e4e46a7d7d6fc860ba3e044100aac852f61bbfdefee8a6b4cc60fe520d1ee

The DFlash drafter (dflash-Qwen3.5-9B-Q8_0.gguf, from z-lab/Qwen3.5-9B-DFlash @ 5fc3b3d474760f18c516db87d84c37edbfd3ede6) is not redistributed. The methodology block shows how to convert it with this fork (--target-model-dir) and quantize Q8_0.

SHA256SUMS covers every model file. recipe/ holds the measurement pipeline (recipe/pipeline/), the prompt-cache patch (recipe/patches/), the graft provenance (recipe/graft/), raw per-run results (recipe/raw/), build and receipt logs (recipe/logs/), and results_summary.json with every measured value on this card. Architecture facts (layer counts, vocabulary, vision depth) come from the checkpoint's config.json at revision 07ba04e2149bace1b43c66448471a70378a7a708.

Known issues and limits

• The MTP head is grafted from the base model. The 15 mtp.* tensors come from Qwen/Qwen3.5-9B at revision c20223623576 (recipe/graft/GRAFT_PROVENANCE.json). They were trained for that checkpoint, not for OxCoder's fine-tune; their usefulness here is the measured acceptance and speed on this box, not a task-accuracy claim.
• MTP + image requests fail on this build (they work with the head unloaded) — see Vision. For image input, drop the two --spec-* lines.
• Prompt caching with MTP needs the included server patch. On the unpatched server the in-file head still drafts (40 of 62 drafted tokens accepted in that control), but the measured second turn reused 0 of its 7,067 prompt tokens; for multi-turn use on an unpatched build, drop the two --spec-* lines.
• preserve_thinking defaults to true in this chat template: earlier assistant turns are re-rendered with their reasoning, so multi-turn contexts grow faster than with templates that drop it. Pass "preserve_thinking": false in chat_template_kwargs to drop it.
• Tool calling was measured only with the template's default tool_call_format (XML parameters); the JSON variant ("tool_call_format": "json") was not tested.
• llama-server's host-RAM prompt cache defaults to 8 GiB (-cram 8192). On a shared box, set -cram explicitly.
• Measured on Linux only (AMD RYZEN AI MAX+ 395 w/ Radeon 8060S, ROCm 7.2.4, patched d3ca537).
• Not measured: decode beyond a 7,458-token prompt (a 262,144-token context was loaded in the memory test, not benchmarked), long-context quality, video input, concurrency above 1, and task-level accuracy. Perplexity/KLD measure next-token fidelity to BF16 on prose, not reasoning or code correctness. OxCoder's own SWE-bench / Terminal-Bench / NL2Repo numbers are for the unquantized checkpoint, not these files.

License and attribution

Apache-2.0, inherited from the base model. Weights: OrionLLM (OrionLLM/OxCoder-9B). MTP head weights: Qwen (Qwen/Qwen3.5-9B, Apache-2.0). DFlash drafter: z-lab (not redistributed). ROCmFP4 / ROCmFPX quantization format and runtime: the ROCmFPX project. Patch, graft, quantization and measurements: kingjones777.