Instructions to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Use Docker

docker model run hf.co/magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "magiccodingman/Qwen3.8-27B-MagicQuant-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-MagicQuant-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Ollama
How to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with Ollama:
```
ollama run hf.co/magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M
```

Unsloth Studio

How to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for magiccodingman/Qwen3.8-27B-MagicQuant-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for magiccodingman/Qwen3.8-27B-MagicQuant-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for magiccodingman/Qwen3.8-27B-MagicQuant-GGUF to start chatting

How to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with Docker Model Runner:
```
docker model run hf.co/magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M
```

Lemonade

How to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.8-27B-MagicQuant-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use magiccodingman/Qwen3.8-27B-MagicQuant-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "magiccodingman/Qwen3.8-27B-MagicQuant-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

MagicQuant Hybrids (v2.0) - Qwen3.8-27B

MagicQuant is a benchmark driven GGUF hybrid discovery and validation system focused on finding real, practical GGUF quants specific to each architecture.

Whether it's a pure baseline model built by llama.cpp, learned tensor configurations from Unsloth, or a custom built MagicQuant hybrid, the model table below shows quants that have won dominance checks, survived collapse spaces, and/or were found to be nonlinearly better. Instead of dumping every quant type possible, MagicQuant tests, validates, and brutally murders anything deemed unworthy.

This combined release utilizes Unsloth Dynamic V3 configurations, while retaining the distinct Dynamic V2 discoveries that still occupy unique positions on the measured size/KLD frontier.

Support MagicQuant

I’m a solo developer working full time for myself to achieve my dream. I build open source code on the side. If you like any of my work, buying me a coffee is always appreciated. Otherwise, I hope you enjoy, maybe give me a star or something. Or just send me good vibes. Either way, thank you!

Click here to see ways to support - BTC, Paypal, GitHub sponsors.

Final survivors

Name	Provider	KLD	Size (GB)	Download
~~LM-Q8_0~~	~~llama.cpp~~	~~0.000712~~	~~29.05~~
MQ-Q6_K_1	MagicQuant	0.000703	29.03	Link
MQ-Q6_K_2	MagicQuant	0.000873	27.26	Link
MQ-Q6_K_3	MagicQuant	0.001047	25.94	Link
UD-Unsloth-UD-Q6_K_XL	Unsloth	0.001238	25.33	Link
UD-Unsloth-UD-Q6_K_L	Unsloth	0.001439	24.23	Link
MQ-Q6_K_4	MagicQuant	0.001518	23.21	Link
MQ-Q5_K_1	MagicQuant	0.002427	22.00	Link
MQ-Q5_K_2	MagicQuant	0.003146	20.91	Link
MQ-Q5_K_3	MagicQuant	0.003562	20.10	Link
MQ-Q5_K_S_1	MagicQuant	0.004818	18.98	Link
MQ-Q4_K_M_1	MagicQuant	0.007412	17.62	Link
UD-Unsloth-UD-Q4_K_M	Unsloth	0.011205	16.50	Link
MQ-IQ4_XS_1	MagicQuant	0.013723	16.34	Link
UD-Unsloth-UD-Q4_K_S	Unsloth	0.014502	15.39	Link
UD-Unsloth-UD-IQ4_XS	Unsloth	0.020127	14.29	Link
UD-Unsloth-UD-Q3_K_XL	Unsloth	0.030770	13.18	Link
UD-Unsloth-UD-IQ3_S	Unsloth	0.046499	12.08	Link
MQ-IQ2_M_1	MagicQuant	0.057811	11.96	Link
UD-Unsloth-UD-IQ3_XXS	Unsloth	0.070084	10.97	Link
MQ-IQ2_M_2	MagicQuant	0.092394	10.77	Link
UD-Unsloth-UD-Q2_K_XL	Unsloth	0.105510	9.86	Link
UD-Unsloth-UD-IQ2_XXS	Unsloth	0.182364	9.05	Link
MQ-IQ2_XXS_1	MagicQuant	0.270304	8.27	Link

Provider credits

llama.cpp — Baseline quantization formats and llama.cpp tooling.
Unsloth — External learned baseline source (Dynamic V2 and Dynamic V3 configurations).

Warning - Is MagicQuant Better? (hint: how you frame the question matters)

External/custom baselines are normalized into MagicQuant's controlled comparison flow. MagicQuant rebuilds a learned baseline under native-source / MagicQuant-controlled conditions, including its own imatrix handling, so hybrids or external baselines (like Unsloth) can be judged on a more equal footing. That does not mean MagicQuant proved the original upstream artifact or upstream imatrix was worse. These comparisons exist for internal hybrid-search consistency and equal playing field comparisons, not as a universal judgment of the original creator's exact release artifact.

Easier to digest explanation:

MagicQuant compares and benchmarks the model's quant-to-tensor configurations, not the original external artifact. Different winners are retained for different reasons, and not every winner is simply “better” in all contexts. The manifests document what the automated system selected and why.

So, MagicQuant can confidently tell you: under the same quantization-to-tensor configurations, identical imatrix, and this benchmark, this candidate earned its place.

Re-Uploading External Provider Baselines

By default, if an external provider like Unsloth is deemed the winner, the repository links directly to the original provider instead of re-hosting the quant. External GGUFs are normally only re-uploaded when a specific winning variant does not already exist.

Release metadata

Manifest guide — combined-file semantics, provenance fields, and archive layout.
Final survivor metrics — final names, provenance, KLD, PPL, sizes, download targets, and replacement lineage.
Hybrid tensor map — tensor-group assignments and effective-state details for every retained MagicQuant hybrid.
Clone tensor configs — exact per-GGUF tensor quantization maps for the reconciled output list.
Isolation samples — V2 and V3 isolated base/group probes, tagged by sourceRun.
Bad trade details — both runs' structured pruning decisions, tagged by run.
Replacement details — both original decision sets plus the cross-run strict-dominance reconciliation.
Original run manifests — untouched Dynamic V2 and Dynamic V3 manifests and source model cards.

Replacement reason codes

STRICT_DOMINANCE — the winner was no larger and had lower real KLD than the removed anchor.
NEAR_BASELINE_PREMIUM — the winner used only the configured near-baseline size premium and beat the real linear KLD trade line.
INTERIOR_DISCOVERY — the winner was selected as a useful interior point inside a size/KLD gap between anchors.
SPACING_COLLAPSE — two candidates were too close in practical output space, so the stronger one was kept.
FINAL_DOMINANCE — a later validated survivor dominated this artifact in final real benchmark comparison.

Underlined names in the table replaced or ultimately inherited the replacement of another artifact. Hover the name for a short summary, or inspect magicquant-manifest/magicquant.replacements.json for exact deltas.

Dynamic V2 + V3 combined release

This repository is the reconciled frontier from two complete MagicQuant runs against the same Qwen3.8-27B source model and imatrix identity. The runs used different generations of Unsloth's learned Dynamic tensor configurations, and that difference produced meaningfully different winners—interesting enough that collapsing everything into only the newer run would discard useful measured trade points.

Dynamic V3 contributes the main frontier, including MQ-Q6_K_1, MQ-Q6_K_2, the Q5 series, MQ-Q4_K_M_1, MQ-IQ2_XXS_1, and the V3 Unsloth reference points. Dynamic V2 contributes MQ-Q6_K_3, MQ-Q6_K_4, MQ-IQ4_XS_1, MQ-IQ2_M_1, MQ-IQ2_M_2, and the UD-Unsloth-UD-IQ2_XXS reference point.

The final Q6_K suffixes describe fidelity order, not run generation: V3's source _1 and _2 remain _1 and _2; V2's source _1 and _2 become _3 and _4. V3's source MQ-Q6_K_3 is excluded because the V2 source MQ-Q6_K_2 (now MQ-Q6_K_4) is both smaller (23.21 vs 23.43 GB) and better (0.001518 vs 0.001761 KLD).

The combined manifests preserve a sourceRun, sourceName, and sourceFileName trail wherever a final artifact was selected or renamed. The original generated manifests remain untouched under magicquant-manifest/archive/, so the V2 and V3 decision histories can still be inspected independently.

Generic vs Unsloth imatrix A/B benchmark

After the combined survivor frontier had already been discovered with the generic MagicQuant imatrix, the same 23 final tensor configurations were cloned, rebuilt, and benchmarked again using the imatrix published in Unsloth's Qwen3.8-27B GGUF repository. This was a fixed-configuration A/B benchmark, not a second full MagicQuant evolution/search.

Because the generic-imatrix search happened first, these results do not establish whether beginning the full search with the Unsloth imatrix would have discovered different hybrids. The table reports only what was actually measured: identical survivor tensor configurations rebuilt under the two imatrices and evaluated by the same benchmark flow.

Δ KLD is Unsloth-imatrix KLD minus generic-imatrix KLD. Negative values favor the Unsloth imatrix; positive values favor the generic imatrix.

Quant	Size GB	Generic Imatrix	Unsloth Imatrix	Δ KLD	Change	Published result
MQ-Q6_K_1	29.03	0.000703	0.000705	+0.000002	+0.28%	🟦 Generic
MQ-Q6_K_2	27.26	0.000874	0.000873	-0.000001	-0.11%	🟩 Unsloth
MQ-Q6_K_3	25.94	0.001052	0.001047	-0.000005	-0.48%	🟩 Unsloth
UD-Unsloth-UD-Q6_K_XL	25.33	0.001238	0.001264	+0.000026	+2.10%	🟦 Generic
UD-Unsloth-UD-Q6_K_L	24.23	0.001439	0.001474	+0.000035	+2.43%	🟦 Generic
MQ-Q6_K_4	23.21	0.001518	0.001560	+0.000042	+2.77%	🟦 Generic
MQ-Q5_K_1	22.00	0.002427	0.002454	+0.000027	+1.11%	🟦 Generic
MQ-Q5_K_2	20.91	0.003212	0.003146	-0.000066	-2.05%	🟩 Unsloth
MQ-Q5_K_3	20.10	0.003742	0.003562	-0.000180	-4.81%	🟩 Unsloth
MQ-Q5_K_S_1	18.98	0.004818	0.005053	+0.000235	+4.88%	🟦 Generic
MQ-Q4_K_M_1	17.62	0.007412	0.008052	+0.000640	+8.63%	🟦 Generic
UD-Unsloth-UD-Q4_K_M	16.50	0.011351	0.011205	-0.000146	-1.29%	🟩 Unsloth
MQ-IQ4_XS_1	16.34	0.013794	0.013723	-0.000071	-0.51%	🟩 Unsloth
UD-Unsloth-UD-Q4_K_S	15.39	0.014502	0.014936	+0.000434	+2.99%	🟦 Generic
UD-Unsloth-UD-IQ4_XS	14.29	0.020127	0.020214	+0.000087	+0.43%	🟦 Generic
UD-Unsloth-UD-Q3_K_XL	13.18	0.031140	0.030770	-0.000370	-1.19%	🟩 Unsloth
UD-Unsloth-UD-IQ3_S	12.08	0.047485	0.046499	-0.000986	-2.08%	🟩 Unsloth
MQ-IQ2_M_1	11.96	0.057811	0.058362	+0.000551	+0.95%	🟦 Generic
UD-Unsloth-UD-IQ3_XXS	10.97	0.070084	0.071898	+0.001814	+2.59%	🟦 Generic
MQ-IQ2_M_2	10.77	0.092394	0.094737	+0.002343	+2.54%	🟦 Generic
UD-Unsloth-UD-Q2_K_XL	9.86	0.106262	0.105510	-0.000752	-0.71%	🟩 Unsloth
UD-Unsloth-UD-IQ2_XXS	9.05	0.182364	0.183547	+0.001183	+0.65%	🟦 Generic
MQ-IQ2_XXS_1	8.27	0.275224	0.270304	-0.004920	-1.79%	🟩 Unsloth

What was published

For locally hosted MagicQuant hybrids, the repository now carries the better-scoring rebuild for each configuration. Six files use the Unsloth-provided imatrix: MQ-Q6_K_2, MQ-Q6_K_3, MQ-Q5_K_2, MQ-Q5_K_3, MQ-IQ4_XS_1, and MQ-IQ2_XXS_1. The other seven locally hosted MagicQuant GGUFs retain their generic-imatrix builds.

External Unsloth rows remain direct links to Unsloth rather than locally re-hosted GGUFs. Their selected top-table KLD is still the better controlled rebuild measurement from this A/B test; it is not a benchmark claim about the original upstream artifact.

The combined magicquant-manifest/ files remain unchanged. They preserve the original search/evolution decisions, generic-imatrix metrics, byte sizes, hybrid recipes, and clone provenance. The README's primary table is the publication view after per-artifact imatrix selection, so its selected KLD values can differ from the archival manifest values. The six Unsloth-imatrix rebuilds are also 64 bytes smaller than their corresponding generic builds due to GGUF metadata differences; this does not affect the displayed two-decimal GB sizes.

The repository-root imatrix.dat remains the generic imatrix associated with the original search manifests. The six named replacement GGUFs were built with the Unsloth-provided imatrix.

Downloads last month: 4,519

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

2-bit

4-bit

5-bit

6-bit

Model tree for magiccodingman/Qwen3.8-27B-MagicQuant-GGUF

Base model

Qwen/Qwen3.8-27B

Quantized

(764)

this model

Collection including magiccodingman/Qwen3.8-27B-MagicQuant-GGUF

Magic Quant

Collection

MagicQuant is a benchmark-driven GGUF evaluation and hybrid-discovery system. https://github.com/magiccodingman/MagicQuant-Wiki • 6 items • Updated 8 days ago • 34