Instructions to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF
# Run inference directly in the terminal:
llama cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF
# Run inference directly in the terminal:
llama cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF
# Run inference directly in the terminal:
./llama-cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Use Docker

docker model run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

LM Studio
Jan

vLLM

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Ollama
How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF with Ollama:
```
ollama run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF
```
Unsloth Desktop

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF with Docker Model Runner:
```
docker model run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF
```

Lemonade

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Run and chat with the model

lemonade run user.KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

KAT-Coder-V2.5-Dev APEX-I-MiniPlus-V2.1 GGUF

The Definitive Frontier MoE · Blistering +24 to 28+ tok/s with Aggressive Partial Offload (DDR4 RAM) · Full 256K Context on 24GB Workstations

🏆 THE DEFINITIVE SPECIFICATION IN THE 13–14 GB CEILING

This APEX-I-MiniPlus-V2.1 release represents the absolute technological limit of sparse Mixture-of-Experts quantization within the 13–14 GB envelope. Every single tensor of its 40 layers and 256 micro-experts has been mathematically audited to maximize reasoning precision, eliminate recurrence state drift, and prevent AVX2 CPU dequantization stalls.

⚠️ DO NOT CONFUSE WITH GENERIC COMMUNITY APEX-I-MINI RELEASES!

Our APEX-I-MiniPlus builds (engineered by IsValorum) are completely custom, handcrafted, tensor-by-tensor architectures designed specifically for hybrid MoE models. Generic community APEX-I-Mini recipes uniformly compress all core experts down to 2-bit IQ2_S, leave the token output head unarmored at 3-bit Q3_K_M, and compress attention projections down to Q3_K. In deep reasoning models, that causes severe perplexity degradation, broken code syntax/brackets, and collapsed reasoning. APEX-I-MiniPlus was built to eliminate this flaw permanently.

📢 Optimization History & Transparency Notice

We maintain our previous releases publicly as a transparent engineering record of continuous optimization. Below is the exact evolutionary roadmap of our MiniPlus architectures:

Specification	Core Experts (10–29)	Edge Experts (0–9, 30–39)	Shared Expert (`shexp`)	Full Attention (L3, 7, 11, ...)	Output Head (`output.weight`)	Routers (`gate_inp`)	Size / Overhead	Real-World Impact
Generic APEX Mini	`IQ2_S` (2.50 bpw)	`Q3_K` (only 5 layers)	`Q4_K` / `Q3_K`	`Q3_K`	`Q3_K_M`	Compressed	Baseline (~12.5 GB)	Severe syntax errors, broken code indentation, high perplexity in `<think>`.
MiniPlus V1	`IQ3_XXS` (3.06 bpw)	`Q3_K` (5 layers)	`Q4_K` / `IQ4_NL`	`Q3_K`	`Q6_K`	`F32` (uncompressed)	+1.1 GB vs generic	Rescued core reasoning; zero router drift; eliminated vocabulary hallucinations.
MiniPlus V2 (Legacy)	`IQ3_XXS`	`IQ3_S` (10 layers)	`IQ4_NL`	`Q3_K` + `Q8_0` gates	`Q6_K`	`F32`	+1.2 GB vs generic	Expanded protective edge envelope; enhanced long-context attention gating.
🔥 MiniPlus V2.1 (CURRENT)	`IQ3_XXS`	`Q3_K` (10 layers)	`Q5_K` (All 40 layers)	`Q4_K` (`q/k/v`) + `Q6_K` (`output`)	`Q6_K`	`F32`	< 100 MB extra over V2 (`13.74 GiB` total)	Zero AVX2 CPU stalls; +24 to 28+ tok/s streaming under aggressive offload (very few layers in VRAM, bulk in DDR4 RAM); rock-solid foundation knowledge; flawless needle-in-a-haystack retrieval.

⚡ Quick Navigation Index

📦 Model Files & Specifications
🔬 Comparative Quantization Analysis (vs. Flat Quants & Generic APEX)
⚡ Extreme Offload Benchmarks (Minimal VRAM + Standard DDR4 RAM)
🔥 The 24GB Miracle: Full 256K Context Runs In VRAM!
🏎️ Hardware Throughput Projections (RTX 30 / 40 / 50)
🛠️ Surgical Tensor Quantization Map
📖 Recommended Configuration & Setup

📦 Model Files & Specifications

File Name	File Size	Memory Footprint	BPW	Description
`KAT-Coder-V2.5-Dev.APEX-I-MiniPlus-V2.1.gguf`	`14.75 GB` (`13.74 GiB`)	`13.74 GiB`	3.40 BPW	Dedicated deep coding, algorithm synthesis, test generation & software engineering MoE

Base Model: KAT-Dev/KAT-Coder-V2.5-Dev
Parameters: 35.2B total (approx. 2.6B to 3.2B active per token)
Architecture: 40 layers, 256 micro-experts (8 active per token) + hybrid linear attention / DeltaNet recurrent layers
Context Length: 262,144 tokens (native 256K)

🔬 Comparative Quantization Analysis (vs. Flat Quants & Generic APEX)

Architectural Component	Generic Automated Quants (Flat `Q3_K_S` / `IQ3_S`)	Generic APEX-I-Mini (Community Baseline)	Our Handcrafted APEX-I-MiniPlus-V2.1 (IsValorum)	Perceived Quality & Real-World Impact
Output Head (`output.weight`)	Flat `IQ3_S` / `Q3_K_S` (3.44 BPW)	Inherits base type `Q3_K_M` (3.44 BPW unarmored)	`Q6_K` (6.56 BPW uncompromised)	Eliminates Syntax & Vocabulary Hallucinations: Prevents bracket drops (`{}`, `[]`), broken markdown, and code indentation collapse.
Expert Routers (`ffn_gate_inp.weight`)	Blindly quantized to 3-bit	Inherits base type `Q3_K_M`	`F32` uncompressed (32.0 BPW, 2 MB/layer)	Zero Router Drift: Guarantees 100% routing fidelity with virtually zero memory overhead (~80 MB total).
*Shared Foundation Expert (`ffn__shexp`)**	Flat `IQ3_S` / `Q3_K_S` (3.44 BPW)	Linear `Q4_K`	`Q5_K` (5.50 BPW high-precision linear)	Foundational Knowledge Armor: Shared expert executes on 100% of tokens. `Q5_K` protects reasoning representations across all 40 layers.
Full Attention Layers (L3, 7, 11, 15, 19, 23, 27, 31, 35, 39)	Flat `IQ3_S` / `Q3_K_S`	`Q3_K`	`Q4_K` for `attn_q/k/v` + `Q6_K` for `attn_output`	Contextual Retrieval Precision: Periodic anchor layers retain pristine query-key mapping and high-precision output projection over deep context.
Attention Gates (`attn_gate.weight`)	Blindly compressed to 3-bit	Compressed to `Q3_K`	`Q8_0` (8.50 BPW)	Attention Head Stability: Modulates query-key routing across hybrid attention layers without crosstalk.
Core MoE Layers (10–29)	Flat `IQ3_S`	Aggressive `IQ2_S` (2.50 BPW)	`IQ3_XXS` (3.06 BPW) + calibrated `imatrix`	Above the Quality Threshold: Avoids the 2-bit perplexity cliff of generic APEX Mini while maintaining deep memory savings.
Edge MoE Layers (0–9 & 30–39)	Flat `IQ3_S`	`Q3_K` (limited to 5 layers)	`Q3_K` (expanded to 10 input & 10 output layers)	Zero CPU Stalls: Native linear quantization executes seamlessly with AVX2 vectorization during aggressive hybrid CPU RAM streaming.
Recurrent Scales (`ssm_alpha`)	Degraded by flat quants	Degraded	`F32` uncompressed (guarded by llama.cpp)	Prevents Recurrent Drift: Protects the state transition math of DeltaNet linear recurrence.

⚡ Extreme Offload Benchmarks (Minimal VRAM + Standard DDR4 RAM)

Empirically Verified in Unsloth Studio & llama.cpp under Aggressive Offload

Offload Configuration: Aggressive hybrid offload — very few layers offloaded to VRAM (~4.2 GB VRAM footprint), with the bulk of the model actively streaming from standard DDR4 system RAM.
Prompt Processing (Prefill): 385.70 to 407.24 tokens/second sustained.
Streaming Text Generation: 24.25 to 28.37 tokens/second sustained with reasoning enabled.
Memory Footprint: 13.74 GiB weights fits effortlessly within typical 16GB / 32GB DDR4 configurations without paging crashes or stutter.

🔥 The 24GB Miracle: Full 256K Context Runs In VRAM!

KAT-Coder-V2.5-Dev APEX-I-MiniPlus-V2.1 fits the entire 256K context window within 24GB VRAM:

Context Length	Model Weights (Est.)	KV Cache (q8_0, 4 slots)	Compute Buffers	Total GPU VRAM (Est.)	Feasibility
32,768 (32k)	`13.74 GiB`	`0.58 GiB`	`1.80 GiB`	`16.12 GiB`	Full offload on 24GB; partial on 16GB
65,536 (64k)	`13.74 GiB`	`0.92 GiB`	`1.95 GiB`	`16.61 GiB`	Effortless fit on 24GB GPUs
131,072 (128k)	`13.74 GiB`	`1.58 GiB`	`2.22 GiB`	`17.54 GiB`	Effortless fit on 24GB GPUs
262,144 (256k)	`13.74 GiB`	`2.92 GiB`	`2.80 GiB`	`19.46 GiB`	🔥 FULL 256K NATIVE IN VRAM!

Note: Leaves comfortable headroom for display drivers and compute buffers on standard 24GB GPUs (RTX 3090, RTX 4090, RTX 5090).

🏎️ Hardware Throughput Projections (RTX 30 / 40 / 50)

Hardware Target	Offload Mode	Generation Speed (Est.)	Prompt Prefill Speed (Est.)	Highlights
NVIDIA RTX 5080 / 5090 (Blackwell)	Full GPU (`-ngl 99`)	120 – 145+ tok/s	2,800 – 3,900+ tok/s	Blistering throughput on GDDR7 bandwidth
NVIDIA RTX 4090 (24GB GDDR6X)	Full GPU (`-ngl 99`)	90 – 115+ tok/s	2,000 – 2,800+ tok/s	Linear attention layers slash prefill latency
NVIDIA RTX 3090 (24GB GDDR6)	Full GPU (`-ngl 99`)	72 – 88+ tok/s	1,500 – 2,200+ tok/s	Full 256k native window in VRAM
Consumer Laptop (RTX 3050 / 4050 + DDR4)	Hybrid (~4.2GB VRAM)	24.25 – 28.37 tok/s	385 – 410+ tok/s	Zero AVX2 CPU stalls; fast DDR4 RAM streaming

🛠️ Surgical Tensor Quantization Map

Layer Group	Sub-Component / Tensor	Precision Type	Rationale
Global Head	`output.weight`	`Q6_K`	Output classification head. Preserves near-FP16 token projection.
Global Embeddings	`token_embd.weight`	`Q4_K`	Guarded vocabulary projection.
All Normalizations	`output_norm`, `attn_*_norm`, `post_attention_norm`	`F32`	100% uncompressed numerical stability across 40 layers.
Shared Foundation Experts	`blk..ffn__shexp` (All 40 Layers)	`Q5_K`	Executes on 100% of tokens. High-precision knowledge backbone.
Expert Routers	`blk.*.ffn_gate_inp` (All 40 Layers)	`F32`	Eliminates router drift across 256 micro-experts.
Periodic Full Attention	`blk.{3,7,11,15,19,23,27,31,35,39}.attn_q/k/v`	`Q4_K`	Periodic anchor checkpoints for long-context stability.
Periodic Full Attention	`blk.{3,7,11,15,19,23,27,31,35,39}.attn_output`	`Q6_K`	High-precision attention projection.
Recurrent SSM Scales	`blk.*.ssm_alpha`	`F32`	Guarded by llama.cpp to prevent DeltaNet recurrence drift.
Edge MoE Experts	Layers 0–9 and 30–39 (`ffn_*_exps`)	`Q3_K`	High-speed linear execution; zero AVX2 CPU stalls.
Core MoE Experts	Layers 10–29 (`ffn_*_exps`)	`IQ3_XXS`	Calibrated with importance matrix (`imatrix`) for maximum compactness.

📖 Recommended Configuration & Setup

llama-server.exe \
  -m KAT-Coder-V2.5-Dev.APEX-I-MiniPlus-V2.1.gguf \
  --port 8080 \
  --parallel 4 \
  --flash-attn on \
  --fit on \
  -c 104960 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

Downloads last month: -

GGUF

Model size

35B params

Architecture

qwen35moe

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Collection including IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF

APEX-I-MiniPlus V2.1 (Current)

Collection

Definitive frontier V2.1 specification within the 13-14 GB envelope with zero AVX2 stalls. • 5 items • Updated 37 minutes ago