Instructions to use Tech2wild/GLM-5.3-Int4-Int8Mix with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Tech2wild/GLM-5.3-Int4-Int8Mix with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Tech2wild/GLM-5.3-Int4-Int8Mix")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Tech2wild/GLM-5.3-Int4-Int8Mix")
model = AutoModelForCausalLM.from_pretrained("Tech2wild/GLM-5.3-Int4-Int8Mix", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Tech2wild/GLM-5.3-Int4-Int8Mix with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Tech2wild/GLM-5.3-Int4-Int8Mix"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Tech2wild/GLM-5.3-Int4-Int8Mix",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Tech2wild/GLM-5.3-Int4-Int8Mix

SGLang

How to use Tech2wild/GLM-5.3-Int4-Int8Mix with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Tech2wild/GLM-5.3-Int4-Int8Mix" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Tech2wild/GLM-5.3-Int4-Int8Mix",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Tech2wild/GLM-5.3-Int4-Int8Mix" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Tech2wild/GLM-5.3-Int4-Int8Mix",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Tech2wild/GLM-5.3-Int4-Int8Mix with Docker Model Runner:
```
docker model run hf.co/Tech2wild/GLM-5.3-Int4-Int8Mix
```

Tony DeAngelo commited on 1 day ago

Commit

a101464

verified ·

1 Parent(s): 4b9b54b

Stage model card (weights uploading after quant)

Browse files

Files changed (1) hide show

README.md +85 -0

README.md ADDED Viewed

	@@ -0,0 +1,85 @@

+---
+license: other
+license_name: glm-5.3
+license_link: https://huggingface.co/zai-org/GLM-5.3/blob/main/LICENSE
+base_model:
+- zai-org/GLM-5.3
+base_model_relation: quantized
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- compressed-tensors
+- int4
+- int8
+- w4a16
+- w8a16
+- moe
+- glm
+- dgx-spark
+- gb10
+- vllm
+---
+# GLM-5.3 Int4-Int8Mix (743B) — the quant that fits 4x NVIDIA DGX Spark
+> ### 🚧 Weights uploading — page staged (2026-08-28)
+> This card is live; the model shards are being quantized and will land here shortly. Everything below describes exactly what is being published. If a number is marked TBD it is being measured on our own hardware and will be filled in with a date next to it.
+An **Int4-Int8 mixed-precision** quantization of the full **[zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3)** (743B total / ~40B active MoE, `glm5_next`, 78 layers, 1M context), in the **`compressed-tensors`** format for **vLLM**. Produced with the QuantTrio-style recipe (data-free, mixed W4A16 / W8A16), it is, as far as we can tell, the **first quant of the big GLM-5.3 that actually serves on four DGX Spark (GB10) nodes** with real KV-cache headroom.
+## Why this one fits when the NVFP4s do not
+The existing full-GLM-5.3 NVFP4 quants are all ~465 GB because they keep the entire attention stack at bf16. Across 4 Sparks that is ~116 GB/rank, leaving only ~12 GB/rank, which is not enough for the KV pool plus framework, so they OOM in practice.
+This quant takes the attention to **Int8** instead of bf16. That single change drops the footprint to **~378 GB** (~94.5 GB/rank at TP4), leaving **~33 GB/rank** for a real KV pool. It is the difference between "loads on paper" and "serves under real traffic."
+| | weights | GB/rank (TP4) | KV headroom/rank | serves on 4 Sparks? |
+|---|---|---|---|---|
+| NVFP4 (attention bf16) | ~465 GB | ~116 | ~12 GB | no (OOM) |
+| **This — Int4-Int8Mix** | **~378 GB** | **~94.5** | **~33 GB** | **yes** |
+## The recipe (data-free, mixed precision)
+Static, symmetric, weight-only, `compressed-tensors` / `pack-quantized`, group size 128. No calibration data. Per-layer map:
+- **MoE experts (layers 3-77):** W4A16, group 128 — the bulk of the weights, where the size savings come from.
+- **Dense + attention (layers 1-77):** W8A16, group 128 — `self_attn.*`, `mlp.gate_up/gate/up/down`, `shared_experts.*`.
+- **MTP block (layer 78):** W8A16, channelwise.
+- **Kept at full precision (this is what protects accuracy):** layer 0 entirely, every `mlp.gate` (the MoE router), `self_attn.indexer` / `indexers_proj` (the DSA sparse-attention selector), the MTP `eh_proj`/`enorm`/`hnorm`, and `shared_head.norm` / `shared_head.head` (the LM head).
+The trick in one line: **Int4 only the bulky experts, Int8 the dense and attention, and leave the routing, indexer, and head alone.**
+## Serving (vLLM, 4x DGX Spark GB10, TP4)
+```bash
+vllm serve <path-to-this-model> \
+  --quantization compressed-tensors \
+  --kv-cache-dtype fp8 \
+  --tensor-parallel-size 4 --enable-expert-parallel \
+  --served-model-name glm-5.3 \
+  --max-model-len 131072   # raise per your KV pool
+```
+- **Speculative decode:** pairs with **[incoai/GLM-5.3-DFlash2](https://huggingface.co/incoai/GLM-5.3-DFlash2)** (a 4.9 GB DFlash2 drafter) at ~zero extra KV pool — the flagship stack. Native MTP (layer 78) is preserved as a fallback.
+- **Long context:** add **Decode Context Parallel** (`--decode-context-parallel-size 4`) to split the KV across all four nodes for a ~600K single-request context (the tradeoff is a small cross-node cost per decode step).
+## Benchmarks
+Measured on 4x DGX Spark (GB10, sm121, aarch64) over a RoCE fabric. **TBD — filling in after the serve-test.**
+| config | decode tok/s | context | KV pool | date |
+|---|---|---|---|---|
+| Int4-Int8Mix, fp8 KV, TP4 | TBD | TBD | TBD | TBD |
+| + DFlash2 | TBD | TBD | TBD | TBD |
+| + DCP4 | TBD | ~600K | TBD | TBD |
+## Credits
+- **Base model:** [zai-org/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) (743B / ~40B active MoE).
+- **Quantization recipe:** the data-free Int4-Int8Mix compressed-tensors method popularized by [QuantTrio](https://huggingface.co/QuantTrio) (e.g. GLM-5.2-Int4-Int8Mix); reproduced here for GLM-5.3 with a verified layer-map coverage check.
+- **Speculative drafter:** [IncoAI](https://huggingface.co/incoai) (DFlash2).
+- **4x DGX Spark deployment + this quant:** [@tonyd2wild](https://github.com/tonyd2wild).
+## License
+Inherits the [GLM-5.3 license](https://huggingface.co/zai-org/GLM-5.3/blob/main/LICENSE) from the base model. Quantization changes weights only, not the license terms.