Copy files from models/unsloth/Qwen3.8-Flash-Next-GGUF

MTP/README.md

@@ -0,0 +1,149 @@
+# MTP draft heads for Qwen3.8-Flash-Next
+
+An MTP head guesses the next few tokens; the main model verifies them in one pass. Verification is
+exact, so the output is unchanged - only the speed. Worth about **1.3x to 1.7x** at low concurrency.
+
+## Which file
+
+**Use `mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` (2.60 GB).** Fastest of the set.
+
+| file | size | |
+|---|---|---|
+| `shared-Q8_0` | 2.60 GB | **recommended** |
+| `shared-Q4_K_M` | 1.78 GB | smaller, ~2 points less acceptance |
+| `shared-BF16` | 4.87 GB | bigger *and* slower than Q8_0 |
+| `Q8_0` / `Q4_K_M` / `BF16` | 3.85 / 2.60 / 7.24 GB | self-contained variants |
+
+`shared-` heads borrow the token embedding and output projection from the model you are already
+running, saving about 1.3 GB. They draft identically to the self-contained files, which carry their
+own copies and are only needed on builds without borrowing support.
+
+BF16 is bigger and slower: a draft step is dominated by the output projection, which is cheaper to
+execute at 8 bits.
+
+## Requirements
+
+**A stock `ggml-org/llama.cpp` build cannot use these.** Mainline has no MTP graph for the
+`qwen4exp` architecture, no cross-model tensor borrowing, and no `--spec-type draft-mtp` option, so
+passing a head to it does nothing. Pick one of the three below.
+
+### Option 1: prebuilt binaries (easiest)
+
+From [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases), tag
+`b10715-mix-86bd2d3` or newer. Assets are `app-<tag>-<os>-<arch>-<backend>.tar.gz` (`.zip` on
+Windows) for CPU, CUDA 12/13, ROCm and Vulkan.
+
+### Option 2: build the upstream pull request
+
+[ggml-org/llama.cpp#28243](https://github.com/ggml-org/llama.cpp/pull/28243) is the MTP support
+going to mainline. Building it:
+
+```bash
+apt-get update
+apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
+git clone --branch qwen4exp/mtp https://github.com/danielhanchen/llama.cpp
+cmake llama.cpp -B llama.cpp/build \
+    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
+cmake --build llama.cpp/build --config Release -j --clean-first \
+    --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
+cp llama.cpp/build/bin/llama-* llama.cpp
+```
+
+Omit `-DGGML_CUDA=ON` for a CPU build. To track the pull request rather than the branch, which keeps
+working if the branch is renamed or deleted, replace the clone with:
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp
+git -C llama.cpp fetch origin refs/pull/28243/head
+git -C llama.cpp checkout FETCH_HEAD
+```
+
+Then fetch a head:
+
+```bash
+pip install -U "huggingface_hub[cli]"
+hf download unsloth/Qwen3.8-Flash-Next-GGUF \
+    --local-dir unsloth/Qwen3.8-Flash-Next-GGUF \
+    --include "*mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf*"
+```
+
+It lands in an `MTP/` subfolder, so the path to pass is
+`unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`:
+
+```bash
+llama.cpp/llama-server \
+    -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \
+    -md unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
+    --spec-type draft-mtp --spec-draft-n-max 2
+```
+
+### Option 3: build the unsloth fork
+
+[unslothai/llama.cpp#144](https://github.com/unslothai/llama.cpp/pull/144):
+
+```bash
+git clone https://github.com/unslothai/llama.cpp && cd llama.cpp
+git fetch origin pull/144/head:mtp && git checkout mtp
+cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON   # omit -DGGML_CUDA for CPU
+cmake --build build -j
+```
+
+## Usage
+
+```bash
+llama-cli \
+  -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
+  -md mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
+  --spec-type draft-mtp --spec-draft-n-max 2 -ngl 999
+```
+
+Same flags for `llama-server`. `--spec-draft-n-max 2` is a good default; higher drafts more but
+each guess is accepted less often.
+
+**Always pass `-md` explicitly.** The heads live in an `MTP/` subfolder, which sidecar
+auto-discovery does not search, so `--spec-type draft-mtp` on its own finds nothing and you get the
+main model's speed with no error saying why.
+
+**A `shared-` head logs one error line at startup and then works.** The automatic memory fit sizes
+the draft by loading it on its own, before the main model exists, so there is nothing for it to
+borrow from and the measurement fails:
+
+```
+E llama_model_load: error loading model: borrow_shared_tensor: this model is a draft head ...
+W operator(): failed to measure the memory of the extra model, fitting without it
+```
+
+Speculation still runs. The only real consequence is that the fit does not count the draft's
+memory, so on a card with little headroom it may choose a context size that does not fit. Pass
+`-c` and `-ngl` yourself, or use a self-contained head, if that matters to you. Self-contained
+heads measure cleanly and produce no such line.
+
+To confirm it is running, look for this in the log. If it never appears, speculation is off and you
+are probably on a build without MTP support:
+
+```
+draft acceptance = 0.66139 (325 accepted / 491 generated), mean len = 2.76
+```
+
+## Measured
+
+Single stream, greedy, one B200, `shared-Q8_0`:
+
+| main model | off | on | |
+|---|---|---|---|
+| `UD-Q4_K_XL` | 83.2 tok/s | **138.8 tok/s** | **1.67x** |
+| `UD-IQ1_S` | 90.1 tok/s | **120.9 tok/s** | **1.34x** |
+
+Acceptance: `shared-BF16` 66.5%, `shared-Q8_0` 66.1%, `shared-Q4_K_M` 64.4%.
+
+**These are greedy numbers.** Higher temperature makes the target less predictable, so fewer
+guesses are accepted and the speedup shrinks. A speculative decoding figure means nothing without
+the sampler it was measured with.
+
+## When not to use it
+
+**Skip MTP for concurrent serving.** A win at concurrency 1, but measured as a net loss (~0.81x to
+0.87x) at concurrency 8: a busy model has little idle capacity for a draft to exploit.
+
+These heads are for Qwen3.8-Flash-Next only; a mismatched pairing is rejected with an error rather
+than producing bad output. Loading a head on its own, without a main model, is also rejected.