Hugging Face's logo

Kernels:
phanerozoic
/
binary-gemm

Kernel card Files
xet
 Community
phanerozoic commited on
Commit
31ec85e
·
verified ·
1 Parent(s): 905e624

Add kernel card

Browse files
Files changed (2) hide show
  1. CARD.md +152 -0
  2. README.md +152 -0
CARD.md ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: kernels
3
+ license: apache-2.0
4
+ ---
5
+
6
+ # binary-gemm
7
+
8
+ W1A8 matrix products for binary-weight language models, loadable through
9
+ `kernels`. Each weight is one bit meaning `+1` or `-1`, with one fp16 scale per
10
+ group of `G` weights. At `G = 128` that is `1 + 16/128 = 1.125` effective bits
11
+ per weight, against 2 bits for a ternary `{-1, 0, +1}` layer and 16 for bf16.
12
+ Activations are quantized per token to int8.
13
+
14
+ The ternary members of this family on the Hub are
15
+ [phanerozoic/bitnet-tc](https://huggingface.co/kernels/phanerozoic/bitnet-tc)
16
+ (CUDA) and
17
+ [phanerozoic/bitnet-cpu](https://huggingface.co/kernels/phanerozoic/bitnet-cpu)
18
+ (CPU). This is the binary member, and it is a different kernel rather than a
19
+ narrower case of those.
20
+
21
+ ## How it works
22
+
23
+ Binary weights collapse the inner product. Writing the stored bit as
24
+ `b = (w + 1) / 2`,
25
+
26
+ ```
27
+ dot(w, a) = 2 * sum_{i : b_i = 1} a_i - sum_i a_i
28
+ ```
29
+
30
+ so the weight never has to be materialized as a number. This kernel keeps the
31
+ int8 activation path and resolves four weights at a time through a 16-entry
32
+ constant table that expands a nibble of the bit pattern directly into four
33
+ packed int8 values of `±1`, which feeds `__dp4a`. The table is the whole decode:
34
+ no shifts, no selects, no unpack buffer, no scratch.
35
+
36
+ One warp owns each output column. Lanes stride the weight groups, accumulate an
37
+ int32 dot per group, apply that group's fp16 scale in fp32, and the warp
38
+ reduces. The batch dimension is tiled at compile time so a weight word is
39
+ fetched once and reused across the tile.
40
+
41
+ ## Measured
42
+
43
+ RTX 6000 Ada (48 GB, 96 MB L2), torch 2.10 + CUDA 12.6. Bonsai 27B shapes,
44
+ hidden size 5120, `G = 128`, against cuBLAS bf16 on the same shapes.
45
+
46
+ Decode, one token:
47
+
48
+ | layer | N | cuBLAS bf16 | binary | speedup |
49
+ |---|---|---|---|---|
50
+ | attention QKV | 8,192 | 0.1124 ms | 0.0404 ms | **2.78x** |
51
+ | attention out | 5,120 | 0.0733 ms | 0.0276 ms | **2.65x** |
52
+ | MLP gate | 17,408 | 0.2425 ms | 0.0756 ms | **3.21x** |
53
+ | MLP down | 5,120 | 0.0738 ms | 0.0272 ms | **2.71x** |
54
+ | LM head | 248,320 | 3.3622 ms | 0.9637 ms | **3.49x** |
55
+
56
+ These numbers rotate over enough distinct weight copies to exceed twice the
57
+ card's L2 on both paths. That matters: an 84 MB bf16 weight matrix fits inside
58
+ 96 MB of L2, so a naive loop over one matrix measures cache bandwidth and
59
+ reports the bf16 baseline as roughly three times faster than it is in a setting
60
+ where a 5.9 GB model streams a different layer every step. Measured
61
+ cache-resident, the small-`N` rows above invert.
62
+
63
+ Weight footprint at hidden 5120:
64
+
65
+ | layer | bf16 | ternary (2-bit) | binary (1.125-bit) | vs ternary |
66
+ |---|---|---|---|---|
67
+ | attention QKV | 80.0 MB | 10.6 MB | 5.6 MB | 1.89x |
68
+ | attention out | 50.0 MB | 6.6 MB | 3.5 MB | 1.89x |
69
+ | LM head | 2,425.0 MB | 322.1 MB | 170.5 MB | 1.89x |
70
+
71
+ 1.89x rather than a nominal 1.78x because the fp16 group scales are a fixed
72
+ overhead both formats pay, and they are a larger share of the ternary total.
73
+
74
+ ## Correctness
75
+
76
+ Verified on RTX 6000 Ada.
77
+
78
+ - **Packing is lossless.** `unpack(pack(W))` is `torch.equal` to `W` for every
79
+ shape tested, including 6144 x 5120. Zeros are treated as `-1`, so a ternary
80
+ tensor deliberately does not round-trip.
81
+ - **The integer path is exact.** With unit group scales the product is integer
82
+ arithmetic, and the output is `torch.equal` to an exact int32 reference
83
+ wherever bf16 can represent the accumulator, which is `|acc| < 256`. This is
84
+ the strongest available statement: not "close", equal.
85
+ - **The full path** matches a reference that dequantizes the weights and does
86
+ the product in fp32 to below `2^-8` normwise, which is the bf16 output
87
+ rounding step.
88
+ - **Activation quantization** is per-token absmax: the largest magnitude in each
89
+ row maps to `±127`, and reconstruction error is under 1% of the row maximum.
90
+ - **Group size is a parameter**, verified at 32, 64, 128 and 256.
91
+ - **Deterministic.** Repeated products are bitwise identical; the reduction tree
92
+ is fixed and there are no atomics.
93
+
94
+ ## Usage
95
+
96
+ ```python
97
+ import torch
98
+ from kernels import get_kernel
99
+
100
+ bg = get_kernel("phanerozoic/binary-gemm", version=1, trust_remote_code=True)
101
+
102
+ W = torch.where(torch.randn(N, K, device="cuda") >= 0, 1, -1).to(torch.int8)
103
+ wq = bg.pack_weights(W) # [N, K//32] int32
104
+ ws = group_scales.to(torch.float16) # [N, K//128] fp16
105
+
106
+ y = bg.binary_linear(x, wq, ws, group_size=128) # bf16 in, bf16 out
107
+ ```
108
+
109
+ `version` selects the release branch; `trust_remote_code` is required by
110
+ `kernels` for publishers without the trusted-publisher mark.
111
+
112
+ Module form, and conversion from an existing dense layer:
113
+
114
+ ```python
115
+ layer = bg.BinaryLinear(K, N, group_size=128).cuda()
116
+ layer = bg.BinaryLinear.from_dense(nn.Linear(K, N)) # sign + group mean-abs scale
117
+ ```
118
+
119
+ ## API
120
+
121
+ | Symbol | Purpose |
122
+ |---|---|
123
+ | `pack_weights(W)` | `{-1,+1}` int8 `[N,K]` -> bit-packed int32 `[N,K//32]` |
124
+ | `unpack_weights(wq, K)` | inverse, for tests and reference paths |
125
+ | `quantize_activation(x)` | bf16 `[...,K]` -> (int8 `[M,K]`, fp32 per-token scale) |
126
+ | `binary_gemm(act_q, act_scale, wq, wscale, bias, group_size)` | packed product -> bf16 |
127
+ | `binary_linear(x, wq, wscale, bias, group_size)` | one-shot forward |
128
+ | `BinaryLinear(in, out, bias, group_size)` | `nn.Module`; `from_dense` converts a dense layer |
129
+
130
+ ## Requirements and limits
131
+
132
+ - NVIDIA GPU with compute capability 8.0+ (`__dp4a`).
133
+ - `K` a multiple of 32 and of `group_size`; bf16 activations and output.
134
+ - **Decode only.** This is a warp-per-output-column GEMV structure and it is
135
+ built for `M = 1`. Above that it loses to cuBLAS and keeps losing: measured
136
+ 0.40x at `M = 2` and 0.02x at `M = 256`, because one warp per column cannot
137
+ use tensor cores and the arithmetic stops being bandwidth-bound. Prefill and
138
+ batched serving want a separate tile-and-`mma` path, which this kernel does
139
+ not implement. Route by batch size.
140
+ - Weights must already be binary. Nothing here trains or calibrates them;
141
+ `from_dense` is a sign quantizer for testing, not a compression method.
142
+
143
+ ## References
144
+
145
+ Rastegari et al., "XNOR-Net" (2016), for binary-weight networks; Courbariaux et
146
+ al., "BinaryConnect" (2015); Ma et al., "The Era of 1-bit LLMs" (2024) for the
147
+ ternary sibling. Group-wise scaling with 1-bit weights as shipped in the Bonsai
148
+ family (PrismML, 2026).
149
+
150
+ ## License
151
+
152
+ Apache-2.0.
README.md ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: kernels
3
+ license: apache-2.0
4
+ ---
5
+
6
+ # binary-gemm
7
+
8
+ W1A8 matrix products for binary-weight language models, loadable through
9
+ `kernels`. Each weight is one bit meaning `+1` or `-1`, with one fp16 scale per
10
+ group of `G` weights. At `G = 128` that is `1 + 16/128 = 1.125` effective bits
11
+ per weight, against 2 bits for a ternary `{-1, 0, +1}` layer and 16 for bf16.
12
+ Activations are quantized per token to int8.
13
+
14
+ The ternary members of this family on the Hub are
15
+ [phanerozoic/bitnet-tc](https://huggingface.co/kernels/phanerozoic/bitnet-tc)
16
+ (CUDA) and
17
+ [phanerozoic/bitnet-cpu](https://huggingface.co/kernels/phanerozoic/bitnet-cpu)
18
+ (CPU). This is the binary member, and it is a different kernel rather than a
19
+ narrower case of those.
20
+
21
+ ## How it works
22
+
23
+ Binary weights collapse the inner product. Writing the stored bit as
24
+ `b = (w + 1) / 2`,
25
+
26
+ ```
27
+ dot(w, a) = 2 * sum_{i : b_i = 1} a_i - sum_i a_i
28
+ ```
29
+
30
+ so the weight never has to be materialized as a number. This kernel keeps the
31
+ int8 activation path and resolves four weights at a time through a 16-entry
32
+ constant table that expands a nibble of the bit pattern directly into four
33
+ packed int8 values of `±1`, which feeds `__dp4a`. The table is the whole decode:
34
+ no shifts, no selects, no unpack buffer, no scratch.
35
+
36
+ One warp owns each output column. Lanes stride the weight groups, accumulate an
37
+ int32 dot per group, apply that group's fp16 scale in fp32, and the warp
38
+ reduces. The batch dimension is tiled at compile time so a weight word is
39
+ fetched once and reused across the tile.
40
+
41
+ ## Measured
42
+
43
+ RTX 6000 Ada (48 GB, 96 MB L2), torch 2.10 + CUDA 12.6. Bonsai 27B shapes,
44
+ hidden size 5120, `G = 128`, against cuBLAS bf16 on the same shapes.
45
+
46
+ Decode, one token:
47
+
48
+ | layer | N | cuBLAS bf16 | binary | speedup |
49
+ |---|---|---|---|---|
50
+ | attention QKV | 8,192 | 0.1124 ms | 0.0404 ms | **2.78x** |
51
+ | attention out | 5,120 | 0.0733 ms | 0.0276 ms | **2.65x** |
52
+ | MLP gate | 17,408 | 0.2425 ms | 0.0756 ms | **3.21x** |
53
+ | MLP down | 5,120 | 0.0738 ms | 0.0272 ms | **2.71x** |
54
+ | LM head | 248,320 | 3.3622 ms | 0.9637 ms | **3.49x** |
55
+
56
+ These numbers rotate over enough distinct weight copies to exceed twice the
57
+ card's L2 on both paths. That matters: an 84 MB bf16 weight matrix fits inside
58
+ 96 MB of L2, so a naive loop over one matrix measures cache bandwidth and
59
+ reports the bf16 baseline as roughly three times faster than it is in a setting
60
+ where a 5.9 GB model streams a different layer every step. Measured
61
+ cache-resident, the small-`N` rows above invert.
62
+
63
+ Weight footprint at hidden 5120:
64
+
65
+ | layer | bf16 | ternary (2-bit) | binary (1.125-bit) | vs ternary |
66
+ |---|---|---|---|---|
67
+ | attention QKV | 80.0 MB | 10.6 MB | 5.6 MB | 1.89x |
68
+ | attention out | 50.0 MB | 6.6 MB | 3.5 MB | 1.89x |
69
+ | LM head | 2,425.0 MB | 322.1 MB | 170.5 MB | 1.89x |
70
+
71
+ 1.89x rather than a nominal 1.78x because the fp16 group scales are a fixed
72
+ overhead both formats pay, and they are a larger share of the ternary total.
73
+
74
+ ## Correctness
75
+
76
+ Verified on RTX 6000 Ada.
77
+
78
+ - **Packing is lossless.** `unpack(pack(W))` is `torch.equal` to `W` for every
79
+ shape tested, including 6144 x 5120. Zeros are treated as `-1`, so a ternary
80
+ tensor deliberately does not round-trip.
81
+ - **The integer path is exact.** With unit group scales the product is integer
82
+ arithmetic, and the output is `torch.equal` to an exact int32 reference
83
+ wherever bf16 can represent the accumulator, which is `|acc| < 256`. This is
84
+ the strongest available statement: not "close", equal.
85
+ - **The full path** matches a reference that dequantizes the weights and does
86
+ the product in fp32 to below `2^-8` normwise, which is the bf16 output
87
+ rounding step.
88
+ - **Activation quantization** is per-token absmax: the largest magnitude in each
89
+ row maps to `±127`, and reconstruction error is under 1% of the row maximum.
90
+ - **Group size is a parameter**, verified at 32, 64, 128 and 256.
91
+ - **Deterministic.** Repeated products are bitwise identical; the reduction tree
92
+ is fixed and there are no atomics.
93
+
94
+ ## Usage
95
+
96
+ ```python
97
+ import torch
98
+ from kernels import get_kernel
99
+
100
+ bg = get_kernel("phanerozoic/binary-gemm", version=1, trust_remote_code=True)
101
+
102
+ W = torch.where(torch.randn(N, K, device="cuda") >= 0, 1, -1).to(torch.int8)
103
+ wq = bg.pack_weights(W) # [N, K//32] int32
104
+ ws = group_scales.to(torch.float16) # [N, K//128] fp16
105
+
106
+ y = bg.binary_linear(x, wq, ws, group_size=128) # bf16 in, bf16 out
107
+ ```
108
+
109
+ `version` selects the release branch; `trust_remote_code` is required by
110
+ `kernels` for publishers without the trusted-publisher mark.
111
+
112
+ Module form, and conversion from an existing dense layer:
113
+
114
+ ```python
115
+ layer = bg.BinaryLinear(K, N, group_size=128).cuda()
116
+ layer = bg.BinaryLinear.from_dense(nn.Linear(K, N)) # sign + group mean-abs scale
117
+ ```
118
+
119
+ ## API
120
+
121
+ | Symbol | Purpose |
122
+ |---|---|
123
+ | `pack_weights(W)` | `{-1,+1}` int8 `[N,K]` -> bit-packed int32 `[N,K//32]` |
124
+ | `unpack_weights(wq, K)` | inverse, for tests and reference paths |
125
+ | `quantize_activation(x)` | bf16 `[...,K]` -> (int8 `[M,K]`, fp32 per-token scale) |
126
+ | `binary_gemm(act_q, act_scale, wq, wscale, bias, group_size)` | packed product -> bf16 |
127
+ | `binary_linear(x, wq, wscale, bias, group_size)` | one-shot forward |
128
+ | `BinaryLinear(in, out, bias, group_size)` | `nn.Module`; `from_dense` converts a dense layer |
129
+
130
+ ## Requirements and limits
131
+
132
+ - NVIDIA GPU with compute capability 8.0+ (`__dp4a`).
133
+ - `K` a multiple of 32 and of `group_size`; bf16 activations and output.
134
+ - **Decode only.** This is a warp-per-output-column GEMV structure and it is
135
+ built for `M = 1`. Above that it loses to cuBLAS and keeps losing: measured
136
+ 0.40x at `M = 2` and 0.02x at `M = 256`, because one warp per column cannot
137
+ use tensor cores and the arithmetic stops being bandwidth-bound. Prefill and
138
+ batched serving want a separate tile-and-`mma` path, which this kernel does
139
+ not implement. Route by batch size.
140
+ - Weights must already be binary. Nothing here trains or calibrates them;
141
+ `from_dense` is a sign quantizer for testing, not a compression method.
142
+
143
+ ## References
144
+
145
+ Rastegari et al., "XNOR-Net" (2016), for binary-weight networks; Courbariaux et
146
+ al., "BinaryConnect" (2015); Ma et al., "The Era of 1-bit LLMs" (2024) for the
147
+ ternary sibling. Group-wise scaling with 1-bit weights as shipped in the Bonsai
148
+ family (PrismML, 2026).
149
+
150
+ ## License
151
+
152
+ Apache-2.0.