GGUF kernels: dequantize + fused gemv over packed blocks, 9 CUDA variants

.gitattributes

@@ -33,3 +33,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+build/torch210-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch210-cxx11-cu128-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch210-cxx11-cu130-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch28-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch28-cxx11-cu128-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch28-cxx11-cu129-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch29-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch29-cxx11-cu128-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch29-cxx11-cu130-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text

.pytest_cache/.gitignore

@@ -0,0 +1,2 @@
+# Created by pytest automatically.
+*

.pytest_cache/CACHEDIR.TAG

@@ -0,0 +1,4 @@
+Signature: 8a477f597d28d172789f06886806bc55
+# This file is a cache directory tag created by pytest.
+# For information about cache directory tags, see:
+#	https://bford.info/cachedir/spec.html

.pytest_cache/README.md

@@ -0,0 +1,8 @@
+# pytest cache directory #
+
+This directory contains data from the pytest's cache plugin,
+which provides the `--lf` and `--ff` options, as well as the `cache` fixture.
+
+**Do not** commit this to version control.
+
+See [the docs](https://docs.pytest.org/en/stable/how-to/cache.html) for more information.

.pytest_cache/v/cache/lastfailed

@@ -0,0 +1 @@
+{}

.pytest_cache/v/cache/nodeids

@@ -0,0 +1,23 @@
+[
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype0-Q4_K]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype0-Q5_K]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype0-Q6_K]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype0-Q8_0]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype1-Q4_K]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype1-Q5_K]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype1-Q6_K]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype1-Q8_0]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype2-Q4_K]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype2-Q5_K]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype2-Q6_K]",
+  "tests/test_gguf_kernels.py::test_dequantize_matches_reference[dtype2-Q8_0]",
+  "tests/test_gguf_kernels.py::test_gemv_is_compileable",
+  "tests/test_gguf_kernels.py::test_mul_mat_vec_matches_matmul[1-Q4_K]",
+  "tests/test_gguf_kernels.py::test_mul_mat_vec_matches_matmul[1-Q5_K]",
+  "tests/test_gguf_kernels.py::test_mul_mat_vec_matches_matmul[1-Q6_K]",
+  "tests/test_gguf_kernels.py::test_mul_mat_vec_matches_matmul[1-Q8_0]",
+  "tests/test_gguf_kernels.py::test_mul_mat_vec_matches_matmul[8-Q4_K]",
+  "tests/test_gguf_kernels.py::test_mul_mat_vec_matches_matmul[8-Q5_K]",
+  "tests/test_gguf_kernels.py::test_mul_mat_vec_matches_matmul[8-Q6_K]",
+  "tests/test_gguf_kernels.py::test_mul_mat_vec_matches_matmul[8-Q8_0]"
+]

README.md

@@ -0,0 +1,62 @@
+---
+license: mit
+tags:
+  - kernel
+  - gguf
+  - quantization
+---
+
+# GGUF kernels
+
+Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values that share a scale. These kernels read that
+layout as it is, so a quantized model runs without ever materializing a dense copy of its weights —
+which is where the memory saving comes from. A 4B model at Q4_K_M stays at 3.1 GB instead of the
+8.4 GB it would take unpacked.
+
+Ported from [llama.cpp](https://github.com/ggml-org/llama.cpp)'s `ggml-cuda`. `vendor/UPSTREAM`
+pins the revision the sources come from; `vendor.py` refreshes them.
+
+## Usage
+
+```python
+import torch
+from kernels import get_kernel
+
+gguf_kernels = get_kernel("marcsun13/gguf-kernels")
+
+# `blocks` is a GGUF weight exactly as stored: (out_features, bytes_per_row) uint8
+out = gguf_kernels.mul_mat_vec(blocks, x, ggml_type=12, out_features=4096)   # x @ blocks.T
+dense = gguf_kernels.dequantize(blocks, ggml_type=12, rows=4096, cols=2560, dtype=torch.bfloat16)
+```
+
+| | |
+| --- | --- |
+| `mul_mat_vec(blocks, x, ggml_type, out_features)` | Fused dequantize-gemv, for `x` of at most `MAX_GEMV_ROWS` rows (decode). Returns f32 whatever `x` was: the kernel writes an f32 destination, and casting at the call site lets the cast fuse into the consumer. |
+| `dequantize(blocks, ggml_type, rows, cols, dtype)` | Blocks to values, in `dtype` directly. For more rows than the gemv handles (prefill), unpack and use an ordinary matmul. |
+| `MAX_GEMV_ROWS` | Upstream's `MMVQ_MAX_BATCH_SIZE`. Beyond this the gemv has no implementation. |
+| `GEMV_TYPES` | The ggml type ids `mul_mat_vec` implements. `dequantize` covers more, so check this before choosing the fused path. |
+
+Both ops carry `register_fake` implementations, so they trace under `torch.compile` instead of
+breaking the graph — which matters, since a break at every quantized linear costs more than the
+kernels save.
+
+## Backends
+
+CUDA today. The ops are named for what they do rather than for a backend: the schema is declared
+once in `torch-ext/torch_binding.cpp` and each backend registers its own implementation of it, so
+adding Metal means adding a `[kernel.*]` section and a source directory — no change to the schema or
+to the Python API, and callers dispatch on the tensor's device as usual.
+
+`mmq` (quantized gemm) and `mmf` (dense gemm) are deliberately not ported: both take a
+`ggml_backend_cuda_context` for their pool allocator, which would pull in the whole ggml backend.
+Above the gemv's row limit, unpacking and using cuBLAS measures at parity with llama.cpp anyway.
+
+## Building
+
+```bash
+nix run .#build-and-copy -L    # every variant, into build/
+nix build .#ci                 # just this system's variant
+pytest tests/                  # against a built variant on PYTHONPATH
+```

build.toml

@@ -0,0 +1,74 @@
+# One backend per `[kernel.*]` section, all implementing the same two entry points declared in
+# torch-ext/torch_binding.h. Adding Metal means adding a section and a `gguf_metal/` directory —
+# no change to the bindings' schema or to the Python API.
+#
+# `vendor/` is a pinned subset of llama.cpp (revision in vendor/UPSTREAM); refresh it with
+# `python vendor.py --rev <sha>`.
+
+[general]
+name = "gguf-kernels"
+version = 1
+backends = ["cuda"]
+
+[torch]
+src = [
+    "torch-ext/torch_binding.cpp",
+    "torch-ext/torch_binding.h",
+]
+
+[kernel.gguf_cuda]
+backend = "cuda"
+depends = ["torch"]
+cuda-capabilities = ["7.5", "8.0", "8.6", "8.9", "9.0", "10.0", "12.0"]
+include = ["gguf_cuda", "torch-ext", "vendor/include", "vendor/src", "vendor/src/ggml-cuda"]
+src = [
+    "gguf_cuda/gguf_cuda.cu",
+    "gguf_cuda/ggml_dispatch.cu",
+    "gguf_cuda/ggml_stubs.cu",
+    "vendor/src/ggml-cuda/convert.cu",
+    "vendor/src/ggml-cuda/quantize.cu",
+    # headers are listed so the builder stages them; only the files above are compiled
+    "vendor/include/ggml-alloc.h",
+    "vendor/include/ggml-backend.h",
+    "vendor/include/ggml-cuda.h",
+    "vendor/include/ggml.h",
+    "vendor/include/gguf.h",
+    "vendor/src/ggml-common.h",
+    "vendor/src/ggml-cuda/common.cuh",
+    "vendor/src/ggml-cuda/convert.cuh",
+    "vendor/src/ggml-cuda/dequantize.cuh",
+    "vendor/src/ggml-cuda/mma.cuh",
+    "vendor/src/ggml-cuda/mmq-config-ampere.cuh",
+    "vendor/src/ggml-cuda/mmq-config-blackwell.cuh",
+    "vendor/src/ggml-cuda/mmq-config-cdna.cuh",
+    "vendor/src/ggml-cuda/mmq-config-pascal.cuh",
+    "vendor/src/ggml-cuda/mmq-config-rdna2.cuh",
+    "vendor/src/ggml-cuda/mmq-config-rdna3-5.cuh",
+    "vendor/src/ggml-cuda/mmq-config-rdna3.cuh",
+    "vendor/src/ggml-cuda/mmq-config-rdna4.cuh",
+    "vendor/src/ggml-cuda/mmq-load-tiles.cuh",
+    "vendor/src/ggml-cuda/mmq-vec-dot.cuh",
+    "vendor/src/ggml-cuda/mmq.cuh",
+    "vendor/src/ggml-cuda/mmvf.cuh",
+    "vendor/src/ggml-cuda/mmvq.cuh",
+    "vendor/src/ggml-cuda/mmvq-impl.cuh",
+    "vendor/src/ggml-cuda/quantize.cuh",
+    "vendor/src/ggml-cuda/unary.cuh",
+    "vendor/src/ggml-cuda/vecdotq.cuh",
+    "vendor/src/ggml-cuda/vendors/cuda.h",
+    "vendor/src/ggml-impl.h",
+]
+# ggml's half/bfloat16 arithmetic needs the operators torch's build disables
+cuda-flags = [
+    "-DGGML_USE_CUDA",
+    "-DNDEBUG",
+    "-U__CUDA_NO_HALF_OPERATORS__",
+    "-U__CUDA_NO_HALF_CONVERSIONS__",
+    "-U__CUDA_NO_HALF2_OPERATORS__",
+    "-U__CUDA_NO_BFLOAT16_CONVERSIONS__",
+    "-U__CUDA_NO_BFLOAT16_OPERATORS__",
+    "-U__CUDA_NO_BFLOAT162_OPERATORS__",
+    "--expt-relaxed-constexpr",
+    "--expt-extended-lambda",
+]
+cxx-flags = ["-DGGML_USE_CUDA", "-DNDEBUG"]

build/torch210-cxx11-cu126-x86_64-linux/__init__.py

@@ -0,0 +1,55 @@
+"""Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
+layout as it is, so a quantized model can be loaded and run without ever materializing a dense
+copy — which is the whole memory saving.
+
+Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
+they do rather than for a backend: each backend registers its own implementation of the same
+schema, so calls dispatch on the tensor's device.
+"""
+
+import torch
+
+from ._ops import add_op_namespace_prefix, ops
+
+
+__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
+
+# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
+# caller with more (prefill) dequantizes and uses an ordinary matmul.
+MAX_GEMV_ROWS = 8
+
+# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
+# `dequantize` covers more, so check this one before choosing the fused path.
+GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
+
+
+def dequantize(
+    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
+) -> torch.Tensor:
+    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
+    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
+
+
+def mul_mat_vec(
+    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
+) -> torch.Tensor:
+    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
+
+    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
+    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
+    where the cast can be fused into whatever consumes it.
+    """
+    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
+
+
+# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
+@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
+def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
+    return blocks.new_empty((rows, cols), dtype=dtype)
+
+
+@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
+def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
+    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch210-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:deef7081403683a708ba874d97321243281a1b6e9e1f23ae06c930f06459f3b1
+size 26692640

build/torch210-cxx11-cu126-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_20260805124229
+ops = torch.ops._gguf_kernels_20260805124229
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch210-cxx11-cu126-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import sys
+
+import importlib
+from pathlib import Path
+from types import ModuleType
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch210-cxx11-cu126-x86_64-linux/metadata.json

@@ -0,0 +1,4 @@
+{
+  "version": 1,
+  "python-depends": []
+}

build/torch210-cxx11-cu128-x86_64-linux/__init__.py

@@ -0,0 +1,55 @@
+"""Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
+layout as it is, so a quantized model can be loaded and run without ever materializing a dense
+copy — which is the whole memory saving.
+
+Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
+they do rather than for a backend: each backend registers its own implementation of the same
+schema, so calls dispatch on the tensor's device.
+"""
+
+import torch
+
+from ._ops import add_op_namespace_prefix, ops
+
+
+__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
+
+# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
+# caller with more (prefill) dequantizes and uses an ordinary matmul.
+MAX_GEMV_ROWS = 8
+
+# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
+# `dequantize` covers more, so check this one before choosing the fused path.
+GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
+
+
+def dequantize(
+    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
+) -> torch.Tensor:
+    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
+    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
+
+
+def mul_mat_vec(
+    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
+) -> torch.Tensor:
+    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
+
+    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
+    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
+    where the cast can be fused into whatever consumes it.
+    """
+    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
+
+
+# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
+@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
+def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
+    return blocks.new_empty((rows, cols), dtype=dtype)
+
+
+@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
+def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
+    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch210-cxx11-cu128-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6cba7d21629a2a2269bd06de30ace18b373266c7e74e75e11f093cc27eec066
+size 39064328

build/torch210-cxx11-cu128-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_20260805124229
+ops = torch.ops._gguf_kernels_20260805124229
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch210-cxx11-cu128-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import sys
+
+import importlib
+from pathlib import Path
+from types import ModuleType
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch210-cxx11-cu128-x86_64-linux/metadata.json

@@ -0,0 +1,4 @@
+{
+  "version": 1,
+  "python-depends": []
+}

build/torch210-cxx11-cu130-x86_64-linux/__init__.py

@@ -0,0 +1,55 @@
+"""Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
+layout as it is, so a quantized model can be loaded and run without ever materializing a dense
+copy — which is the whole memory saving.
+
+Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
+they do rather than for a backend: each backend registers its own implementation of the same
+schema, so calls dispatch on the tensor's device.
+"""
+
+import torch
+
+from ._ops import add_op_namespace_prefix, ops
+
+
+__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
+
+# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
+# caller with more (prefill) dequantizes and uses an ordinary matmul.
+MAX_GEMV_ROWS = 8
+
+# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
+# `dequantize` covers more, so check this one before choosing the fused path.
+GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
+
+
+def dequantize(
+    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
+) -> torch.Tensor:
+    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
+    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
+
+
+def mul_mat_vec(
+    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
+) -> torch.Tensor:
+    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
+
+    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
+    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
+    where the cast can be fused into whatever consumes it.
+    """
+    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
+
+
+# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
+@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
+def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
+    return blocks.new_empty((rows, cols), dtype=dtype)
+
+
+@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
+def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
+    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch210-cxx11-cu130-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4e935328add4b20b3761b3ab018967f2b61030491865b841c455e6a25702e4e
+size 39318224

build/torch210-cxx11-cu130-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_20260805124229
+ops = torch.ops._gguf_kernels_20260805124229
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch210-cxx11-cu130-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import sys
+
+import importlib
+from pathlib import Path
+from types import ModuleType
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch210-cxx11-cu130-x86_64-linux/metadata.json

@@ -0,0 +1,4 @@
+{
+  "version": 1,
+  "python-depends": []
+}

build/torch28-cxx11-cu126-x86_64-linux/__init__.py

@@ -0,0 +1,55 @@
+"""Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
+layout as it is, so a quantized model can be loaded and run without ever materializing a dense
+copy — which is the whole memory saving.
+
+Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
+they do rather than for a backend: each backend registers its own implementation of the same
+schema, so calls dispatch on the tensor's device.
+"""
+
+import torch
+
+from ._ops import add_op_namespace_prefix, ops
+
+
+__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
+
+# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
+# caller with more (prefill) dequantizes and uses an ordinary matmul.
+MAX_GEMV_ROWS = 8
+
+# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
+# `dequantize` covers more, so check this one before choosing the fused path.
+GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
+
+
+def dequantize(
+    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
+) -> torch.Tensor:
+    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
+    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
+
+
+def mul_mat_vec(
+    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
+) -> torch.Tensor:
+    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
+
+    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
+    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
+    where the cast can be fused into whatever consumes it.
+    """
+    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
+
+
+# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
+@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
+def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
+    return blocks.new_empty((rows, cols), dtype=dtype)
+
+
+@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
+def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
+    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch28-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6708d42e7390b3a6fe316852e3857716defd991e5861031c8d04a1e7b4e0260f
+size 26686512

build/torch28-cxx11-cu126-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_20260805124229
+ops = torch.ops._gguf_kernels_20260805124229
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch28-cxx11-cu126-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import sys
+
+import importlib
+from pathlib import Path
+from types import ModuleType
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch28-cxx11-cu126-x86_64-linux/metadata.json

@@ -0,0 +1,4 @@
+{
+  "version": 1,
+  "python-depends": []
+}

build/torch28-cxx11-cu128-x86_64-linux/__init__.py

@@ -0,0 +1,55 @@
+"""Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
+layout as it is, so a quantized model can be loaded and run without ever materializing a dense
+copy — which is the whole memory saving.
+
+Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
+they do rather than for a backend: each backend registers its own implementation of the same
+schema, so calls dispatch on the tensor's device.
+"""
+
+import torch
+
+from ._ops import add_op_namespace_prefix, ops
+
+
+__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
+
+# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
+# caller with more (prefill) dequantizes and uses an ordinary matmul.
+MAX_GEMV_ROWS = 8
+
+# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
+# `dequantize` covers more, so check this one before choosing the fused path.
+GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
+
+
+def dequantize(
+    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
+) -> torch.Tensor:
+    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
+    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
+
+
+def mul_mat_vec(
+    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
+) -> torch.Tensor:
+    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
+
+    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
+    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
+    where the cast can be fused into whatever consumes it.
+    """
+    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
+
+
+# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
+@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
+def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
+    return blocks.new_empty((rows, cols), dtype=dtype)
+
+
+@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
+def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
+    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch28-cxx11-cu128-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:964ea6025655311cfebe2fb58eca589ace6d208222d793134a48925309ccf910
+size 39054016

build/torch28-cxx11-cu128-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_20260805124229
+ops = torch.ops._gguf_kernels_20260805124229
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch28-cxx11-cu128-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import sys
+
+import importlib
+from pathlib import Path
+from types import ModuleType
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch28-cxx11-cu128-x86_64-linux/metadata.json

@@ -0,0 +1,4 @@
+{
+  "version": 1,
+  "python-depends": []
+}

build/torch28-cxx11-cu129-x86_64-linux/__init__.py

@@ -0,0 +1,55 @@
+"""Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
+layout as it is, so a quantized model can be loaded and run without ever materializing a dense
+copy — which is the whole memory saving.
+
+Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
+they do rather than for a backend: each backend registers its own implementation of the same
+schema, so calls dispatch on the tensor's device.
+"""
+
+import torch
+
+from ._ops import add_op_namespace_prefix, ops
+
+
+__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
+
+# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
+# caller with more (prefill) dequantizes and uses an ordinary matmul.
+MAX_GEMV_ROWS = 8
+
+# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
+# `dequantize` covers more, so check this one before choosing the fused path.
+GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
+
+
+def dequantize(
+    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
+) -> torch.Tensor:
+    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
+    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
+
+
+def mul_mat_vec(
+    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
+) -> torch.Tensor:
+    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
+
+    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
+    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
+    where the cast can be fused into whatever consumes it.
+    """
+    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
+
+
+# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
+@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
+def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
+    return blocks.new_empty((rows, cols), dtype=dtype)
+
+
+@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
+def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
+    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch28-cxx11-cu129-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:51b5f28bd29d9373e0a9a5046e7314af5061350b340646ab98dfa391fd4f8670
+size 38993592

build/torch28-cxx11-cu129-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_20260805124229
+ops = torch.ops._gguf_kernels_20260805124229
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch28-cxx11-cu129-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import sys
+
+import importlib
+from pathlib import Path
+from types import ModuleType
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch28-cxx11-cu129-x86_64-linux/metadata.json

@@ -0,0 +1,4 @@
+{
+  "version": 1,
+  "python-depends": []
+}

build/torch29-cxx11-cu126-x86_64-linux/__init__.py

@@ -0,0 +1,55 @@
+"""Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
+layout as it is, so a quantized model can be loaded and run without ever materializing a dense
+copy — which is the whole memory saving.
+
+Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
+they do rather than for a backend: each backend registers its own implementation of the same
+schema, so calls dispatch on the tensor's device.
+"""
+
+import torch
+
+from ._ops import add_op_namespace_prefix, ops
+
+
+__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
+
+# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
+# caller with more (prefill) dequantizes and uses an ordinary matmul.
+MAX_GEMV_ROWS = 8
+
+# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
+# `dequantize` covers more, so check this one before choosing the fused path.
+GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
+
+
+def dequantize(
+    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
+) -> torch.Tensor:
+    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
+    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
+
+
+def mul_mat_vec(
+    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
+) -> torch.Tensor:
+    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
+
+    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
+    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
+    where the cast can be fused into whatever consumes it.
+    """
+    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
+
+
+# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
+@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
+def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
+    return blocks.new_empty((rows, cols), dtype=dtype)
+
+
+@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
+def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
+    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch29-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7435e6aeaab83f149782dd9d4dd926012315c0e9162b2a141dcbc961da18ea9
+size 26690648

build/torch29-cxx11-cu126-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_20260805124229
+ops = torch.ops._gguf_kernels_20260805124229
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch29-cxx11-cu126-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import sys
+
+import importlib
+from pathlib import Path
+from types import ModuleType
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch29-cxx11-cu126-x86_64-linux/metadata.json

@@ -0,0 +1,4 @@
+{
+  "version": 1,
+  "python-depends": []
+}

build/torch29-cxx11-cu128-x86_64-linux/__init__.py

@@ -0,0 +1,55 @@
+"""Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
+layout as it is, so a quantized model can be loaded and run without ever materializing a dense
+copy — which is the whole memory saving.
+
+Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
+they do rather than for a backend: each backend registers its own implementation of the same
+schema, so calls dispatch on the tensor's device.
+"""
+
+import torch
+
+from ._ops import add_op_namespace_prefix, ops
+
+
+__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
+
+# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
+# caller with more (prefill) dequantizes and uses an ordinary matmul.
+MAX_GEMV_ROWS = 8
+
+# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
+# `dequantize` covers more, so check this one before choosing the fused path.
+GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
+
+
+def dequantize(
+    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
+) -> torch.Tensor:
+    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
+    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
+
+
+def mul_mat_vec(
+    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
+) -> torch.Tensor:
+    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
+
+    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
+    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
+    where the cast can be fused into whatever consumes it.
+    """
+    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
+
+
+# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
+@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
+def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
+    return blocks.new_empty((rows, cols), dtype=dtype)
+
+
+@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
+def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
+    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch29-cxx11-cu128-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c141779c8750668e8681e6b803bdac2b6105ee861ff3331b59c0375493392a8
+size 39058144

build/torch29-cxx11-cu128-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_20260805124229
+ops = torch.ops._gguf_kernels_20260805124229
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch29-cxx11-cu128-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import sys
+
+import importlib
+from pathlib import Path
+from types import ModuleType
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch29-cxx11-cu128-x86_64-linux/metadata.json

@@ -0,0 +1,4 @@
+{
+  "version": 1,
+  "python-depends": []
+}

build/torch29-cxx11-cu130-x86_64-linux/__init__.py

@@ -0,0 +1,55 @@
+"""Compute directly on the packed blocks of a GGUF checkpoint.
+
+A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
+layout as it is, so a quantized model can be loaded and run without ever materializing a dense
+copy — which is the whole memory saving.
+
+Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
+they do rather than for a backend: each backend registers its own implementation of the same
+schema, so calls dispatch on the tensor's device.
+"""
+
+import torch
+
+from ._ops import add_op_namespace_prefix, ops
+
+
+__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
+
+# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
+# caller with more (prefill) dequantizes and uses an ordinary matmul.
+MAX_GEMV_ROWS = 8
+
+# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
+# `dequantize` covers more, so check this one before choosing the fused path.
+GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
+
+
+def dequantize(
+    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
+) -> torch.Tensor:
+    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
+    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
+
+
+def mul_mat_vec(
+    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
+) -> torch.Tensor:
+    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
+
+    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
+    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
+    where the cast can be fused into whatever consumes it.
+    """
+    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
+
+
+# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
+@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
+def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
+    return blocks.new_empty((rows, cols), dtype=dtype)
+
+
+@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
+def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
+    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch29-cxx11-cu130-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41f95273f40f722f1e7e64d5a9784f769ff48cbce511f383e88bd89b70495a8b
+size 39312040