Hugging Face's logo

Kernels:
phanerozoic
/
binary-gemm

Kernel card Files
xet
 Community
phanerozoic commited on
Commit
ca7b082
·
verified ·
1 Parent(s): 31ec85e

Card: standardized form with hero

Browse files
Files changed (2) hide show
  1. CARD.md +87 -103
  2. README.md +87 -103
CARD.md CHANGED
@@ -6,90 +6,26 @@ license: apache-2.0
6
  # binary-gemm
7
 
8
  W1A8 matrix products for binary-weight language models, loadable through
9
- `kernels`. Each weight is one bit meaning `+1` or `-1`, with one fp16 scale per
10
- group of `G` weights. At `G = 128` that is `1 + 16/128 = 1.125` effective bits
11
- per weight, against 2 bits for a ternary `{-1, 0, +1}` layer and 16 for bf16.
12
- Activations are quantized per token to int8.
13
-
14
- The ternary members of this family on the Hub are
15
- [phanerozoic/bitnet-tc](https://huggingface.co/kernels/phanerozoic/bitnet-tc)
16
- (CUDA) and
17
- [phanerozoic/bitnet-cpu](https://huggingface.co/kernels/phanerozoic/bitnet-cpu)
18
- (CPU). This is the binary member, and it is a different kernel rather than a
19
- narrower case of those.
20
-
21
- ## How it works
22
-
23
- Binary weights collapse the inner product. Writing the stored bit as
24
- `b = (w + 1) / 2`,
25
-
26
- ```
27
- dot(w, a) = 2 * sum_{i : b_i = 1} a_i - sum_i a_i
28
- ```
29
-
30
- so the weight never has to be materialized as a number. This kernel keeps the
31
- int8 activation path and resolves four weights at a time through a 16-entry
32
- constant table that expands a nibble of the bit pattern directly into four
33
- packed int8 values of `±1`, which feeds `__dp4a`. The table is the whole decode:
34
- no shifts, no selects, no unpack buffer, no scratch.
35
-
36
- One warp owns each output column. Lanes stride the weight groups, accumulate an
37
- int32 dot per group, apply that group's fp16 scale in fp32, and the warp
38
- reduces. The batch dimension is tiled at compile time so a weight word is
39
- fetched once and reused across the tile.
40
-
41
- ## Measured
42
-
43
- RTX 6000 Ada (48 GB, 96 MB L2), torch 2.10 + CUDA 12.6. Bonsai 27B shapes,
44
- hidden size 5120, `G = 128`, against cuBLAS bf16 on the same shapes.
45
-
46
- Decode, one token:
47
-
48
- | layer | N | cuBLAS bf16 | binary | speedup |
49
- |---|---|---|---|---|
50
- | attention QKV | 8,192 | 0.1124 ms | 0.0404 ms | **2.78x** |
51
- | attention out | 5,120 | 0.0733 ms | 0.0276 ms | **2.65x** |
52
- | MLP gate | 17,408 | 0.2425 ms | 0.0756 ms | **3.21x** |
53
- | MLP down | 5,120 | 0.0738 ms | 0.0272 ms | **2.71x** |
54
- | LM head | 248,320 | 3.3622 ms | 0.9637 ms | **3.49x** |
55
-
56
- These numbers rotate over enough distinct weight copies to exceed twice the
57
- card's L2 on both paths. That matters: an 84 MB bf16 weight matrix fits inside
58
- 96 MB of L2, so a naive loop over one matrix measures cache bandwidth and
59
- reports the bf16 baseline as roughly three times faster than it is in a setting
60
- where a 5.9 GB model streams a different layer every step. Measured
61
- cache-resident, the small-`N` rows above invert.
62
-
63
- Weight footprint at hidden 5120:
64
-
65
- | layer | bf16 | ternary (2-bit) | binary (1.125-bit) | vs ternary |
66
- |---|---|---|---|---|
67
- | attention QKV | 80.0 MB | 10.6 MB | 5.6 MB | 1.89x |
68
- | attention out | 50.0 MB | 6.6 MB | 3.5 MB | 1.89x |
69
- | LM head | 2,425.0 MB | 322.1 MB | 170.5 MB | 1.89x |
70
-
71
- 1.89x rather than a nominal 1.78x because the fp16 group scales are a fixed
72
- overhead both formats pay, and they are a larger share of the ternary total.
73
-
74
- ## Correctness
75
-
76
- Verified on RTX 6000 Ada.
77
-
78
- - **Packing is lossless.** `unpack(pack(W))` is `torch.equal` to `W` for every
79
- shape tested, including 6144 x 5120. Zeros are treated as `-1`, so a ternary
80
- tensor deliberately does not round-trip.
81
- - **The integer path is exact.** With unit group scales the product is integer
82
- arithmetic, and the output is `torch.equal` to an exact int32 reference
83
- wherever bf16 can represent the accumulator, which is `|acc| < 256`. This is
84
- the strongest available statement: not "close", equal.
85
- - **The full path** matches a reference that dequantizes the weights and does
86
- the product in fp32 to below `2^-8` normwise, which is the bf16 output
87
- rounding step.
88
- - **Activation quantization** is per-token absmax: the largest magnitude in each
89
- row maps to `±127`, and reconstruction error is under 1% of the row maximum.
90
- - **Group size is a parameter**, verified at 32, 64, 128 and 256.
91
- - **Deterministic.** Repeated products are bitwise identical; the reduction tree
92
- is fixed and there are no atomics.
93
 
94
  ## Usage
95
 
@@ -104,18 +40,13 @@ wq = bg.pack_weights(W) # [N, K//32] int32
104
  ws = group_scales.to(torch.float16) # [N, K//128] fp16
105
 
106
  y = bg.binary_linear(x, wq, ws, group_size=128) # bf16 in, bf16 out
 
 
107
  ```
108
 
109
  `version` selects the release branch; `trust_remote_code` is required by
110
  `kernels` for publishers without the trusted-publisher mark.
111
 
112
- Module form, and conversion from an existing dense layer:
113
-
114
- ```python
115
- layer = bg.BinaryLinear(K, N, group_size=128).cuda()
116
- layer = bg.BinaryLinear.from_dense(nn.Linear(K, N)) # sign + group mean-abs scale
117
- ```
118
-
119
  ## API
120
 
121
  | Symbol | Purpose |
@@ -127,25 +58,78 @@ layer = bg.BinaryLinear.from_dense(nn.Linear(K, N)) # sign + group mean-abs sc
127
  | `binary_linear(x, wq, wscale, bias, group_size)` | one-shot forward |
128
  | `BinaryLinear(in, out, bias, group_size)` | `nn.Module`; `from_dense` converts a dense layer |
129
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  ## Requirements and limits
131
 
132
  - NVIDIA GPU with compute capability 8.0+ (`__dp4a`).
133
  - `K` a multiple of 32 and of `group_size`; bf16 activations and output.
134
- - **Decode only.** This is a warp-per-output-column GEMV structure and it is
135
- built for `M = 1`. Above that it loses to cuBLAS and keeps losing: measured
136
- 0.40x at `M = 2` and 0.02x at `M = 256`, because one warp per column cannot
137
- use tensor cores and the arithmetic stops being bandwidth-bound. Prefill and
138
- batched serving want a separate tile-and-`mma` path, which this kernel does
139
- not implement. Route by batch size.
140
- - Weights must already be binary. Nothing here trains or calibrates them;
141
- `from_dense` is a sign quantizer for testing, not a compression method.
142
 
143
  ## References
144
 
145
- Rastegari et al., "XNOR-Net" (2016), for binary-weight networks; Courbariaux et
146
- al., "BinaryConnect" (2015); Ma et al., "The Era of 1-bit LLMs" (2024) for the
147
- ternary sibling. Group-wise scaling with 1-bit weights as shipped in the Bonsai
148
- family (PrismML, 2026).
149
 
150
  ## License
151
 
 
6
  # binary-gemm
7
 
8
  W1A8 matrix products for binary-weight language models, loadable through
9
+ `kernels`. Each weight is one bit meaning `+1` or `-1`, with one fp16 scale
10
+ per group of `G` weights: at `G = 128` that is 1.125 effective bits per
11
+ weight, against 2 bits for a ternary layer and 16 for bf16. The reference
12
+ baseline is the dequantized product; the integer path is `torch.equal`-exact
13
+ where bf16 can represent the accumulator. Ternary siblings:
14
+ [bitnet-tc](https://huggingface.co/kernels/phanerozoic/bitnet-tc) (CUDA) and
15
+ [bitnet-cpu](https://huggingface.co/kernels/phanerozoic/bitnet-cpu) (CPU).
16
+
17
+ At one bit per weight a model's memory is its scales plus raw sign bits, and
18
+ decode, which reads every weight per token, becomes almost pure bit traffic.
19
+ This kernel multiplies the packed bits directly: a 16-entry constant table
20
+ expands a nibble of the pattern into four packed `±1` int8 values feeding
21
+ `__dp4a`, so the weight is never materialized as a number and the decode wall
22
+ drops with the bytes.
23
+
24
+ ![A real sign-bit field under a scanline next to per-layer decode bars where the binary path runs a fraction of bf16](https://huggingface.co/kernels/phanerozoic/binary-gemm/resolve/main/media/hero.gif)
25
+
26
+ *Actual packed sign bits of a layer (cyan +1, magenta -1), and decode
27
+ measured with weights rotated past L2 in both paths: 2.4x on attention QKV,
28
+ 2.8x on the MLP gate, 3.8x on the 248,320-row LM head.*
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
 
30
  ## Usage
31
 
 
40
  ws = group_scales.to(torch.float16) # [N, K//128] fp16
41
 
42
  y = bg.binary_linear(x, wq, ws, group_size=128) # bf16 in, bf16 out
43
+
44
+ layer = bg.BinaryLinear.from_dense(nn.Linear(K, N)) # sign + group mean-abs scale
45
  ```
46
 
47
  `version` selects the release branch; `trust_remote_code` is required by
48
  `kernels` for publishers without the trusted-publisher mark.
49
 
 
 
 
 
 
 
 
50
  ## API
51
 
52
  | Symbol | Purpose |
 
58
  | `binary_linear(x, wq, wscale, bias, group_size)` | one-shot forward |
59
  | `BinaryLinear(in, out, bias, group_size)` | `nn.Module`; `from_dense` converts a dense layer |
60
 
61
+ ## Method
62
+
63
+ Binary weights collapse the inner product: writing the stored bit as
64
+ `b = (w + 1) / 2`,
65
+
66
+ ```
67
+ dot(w, a) = 2 * sum_{i : b_i = 1} a_i - sum_i a_i
68
+ ```
69
+
70
+ so the weight never has to exist as a number. The kernel keeps the int8
71
+ activation path and resolves four weights at a time through a 16-entry
72
+ constant table that expands a nibble directly into four packed int8 `±1`
73
+ values for `__dp4a`; the table is the whole decode. One warp owns each output
74
+ column: lanes stride the weight groups, accumulate an int32 dot per group,
75
+ apply that group's fp16 scale in fp32, and the warp reduces. The batch
76
+ dimension is tiled at compile time so a weight word is fetched once per tile.
77
+
78
+ ## Measured
79
+
80
+ Bonsai 27B shapes, hidden 5120, `G = 128`, against cuBLAS bf16, weights
81
+ rotated over enough copies to exceed twice the card's L2 on both paths (an
82
+ 84 MB bf16 matrix inside a 96 MB L2 otherwise measures cache bandwidth):
83
+
84
+ | layer | N | cuBLAS bf16 | binary | speedup |
85
+ |---|---|---|---|---|
86
+ | attention QKV | 8,192 | 0.1124 ms | 0.0404 ms | 2.78x |
87
+ | attention out | 5,120 | 0.0733 ms | 0.0276 ms | 2.65x |
88
+ | MLP gate | 17,408 | 0.2425 ms | 0.0756 ms | 3.21x |
89
+ | MLP down | 5,120 | 0.0738 ms | 0.0272 ms | 2.71x |
90
+ | LM head | 248,320 | 3.3622 ms | 0.9637 ms | 3.49x |
91
+
92
+ Weight footprint at hidden 5120:
93
+
94
+ | layer | bf16 | ternary (2-bit) | binary (1.125-bit) | vs ternary |
95
+ |---|---|---|---|---|
96
+ | attention QKV | 80.0 MB | 10.6 MB | 5.6 MB | 1.89x |
97
+ | attention out | 50.0 MB | 6.6 MB | 3.5 MB | 1.89x |
98
+ | LM head | 2,425.0 MB | 322.1 MB | 170.5 MB | 1.89x |
99
+
100
+ 1.89x rather than the nominal 1.78x because the fp16 group scales are a
101
+ fixed overhead both formats pay.
102
+
103
+ ## Correctness
104
+
105
+ - Packing is lossless: `unpack(pack(W))` is `torch.equal` to `W` for every
106
+ shape tested. Zeros are treated as `-1`, so a ternary tensor deliberately
107
+ does not round-trip.
108
+ - The integer path is exact: with unit group scales the output is
109
+ `torch.equal` to an exact int32 reference wherever bf16 represents the
110
+ accumulator (`|acc| < 256`).
111
+ - The full path matches a dequantize-and-fp32 reference to below `2^-8`
112
+ normwise, the bf16 output rounding step.
113
+ - Activation quantization is per-token absmax with reconstruction error
114
+ under 1% of the row maximum; group size verified at 32, 64, 128, 256.
115
+ - Deterministic: repeated products are bitwise identical; fixed reduction
116
+ tree, no atomics.
117
+
118
  ## Requirements and limits
119
 
120
  - NVIDIA GPU with compute capability 8.0+ (`__dp4a`).
121
  - `K` a multiple of 32 and of `group_size`; bf16 activations and output.
122
+ - Decode only: a warp-per-column GEMV built for `M = 1`. Above that it loses
123
+ to cuBLAS (0.40x at `M = 2`, 0.02x at `M = 256`); route prefill and batch
124
+ elsewhere.
125
+ - Weights must already be binary; `from_dense` is a sign quantizer for
126
+ testing, not a compression method.
 
 
 
127
 
128
  ## References
129
 
130
+ Rastegari et al., "XNOR-Net" (2016); Courbariaux et al., "BinaryConnect"
131
+ (2015); Ma et al., "The Era of 1-bit LLMs" (2024); group-wise scaling with
132
+ 1-bit weights as shipped in the Bonsai family (PrismML, 2026).
 
133
 
134
  ## License
135
 
README.md CHANGED
@@ -6,90 +6,26 @@ license: apache-2.0
6
  # binary-gemm
7
 
8
  W1A8 matrix products for binary-weight language models, loadable through
9
- `kernels`. Each weight is one bit meaning `+1` or `-1`, with one fp16 scale per
10
- group of `G` weights. At `G = 128` that is `1 + 16/128 = 1.125` effective bits
11
- per weight, against 2 bits for a ternary `{-1, 0, +1}` layer and 16 for bf16.
12
- Activations are quantized per token to int8.
13
-
14
- The ternary members of this family on the Hub are
15
- [phanerozoic/bitnet-tc](https://huggingface.co/kernels/phanerozoic/bitnet-tc)
16
- (CUDA) and
17
- [phanerozoic/bitnet-cpu](https://huggingface.co/kernels/phanerozoic/bitnet-cpu)
18
- (CPU). This is the binary member, and it is a different kernel rather than a
19
- narrower case of those.
20
-
21
- ## How it works
22
-
23
- Binary weights collapse the inner product. Writing the stored bit as
24
- `b = (w + 1) / 2`,
25
-
26
- ```
27
- dot(w, a) = 2 * sum_{i : b_i = 1} a_i - sum_i a_i
28
- ```
29
-
30
- so the weight never has to be materialized as a number. This kernel keeps the
31
- int8 activation path and resolves four weights at a time through a 16-entry
32
- constant table that expands a nibble of the bit pattern directly into four
33
- packed int8 values of `±1`, which feeds `__dp4a`. The table is the whole decode:
34
- no shifts, no selects, no unpack buffer, no scratch.
35
-
36
- One warp owns each output column. Lanes stride the weight groups, accumulate an
37
- int32 dot per group, apply that group's fp16 scale in fp32, and the warp
38
- reduces. The batch dimension is tiled at compile time so a weight word is
39
- fetched once and reused across the tile.
40
-
41
- ## Measured
42
-
43
- RTX 6000 Ada (48 GB, 96 MB L2), torch 2.10 + CUDA 12.6. Bonsai 27B shapes,
44
- hidden size 5120, `G = 128`, against cuBLAS bf16 on the same shapes.
45
-
46
- Decode, one token:
47
-
48
- | layer | N | cuBLAS bf16 | binary | speedup |
49
- |---|---|---|---|---|
50
- | attention QKV | 8,192 | 0.1124 ms | 0.0404 ms | **2.78x** |
51
- | attention out | 5,120 | 0.0733 ms | 0.0276 ms | **2.65x** |
52
- | MLP gate | 17,408 | 0.2425 ms | 0.0756 ms | **3.21x** |
53
- | MLP down | 5,120 | 0.0738 ms | 0.0272 ms | **2.71x** |
54
- | LM head | 248,320 | 3.3622 ms | 0.9637 ms | **3.49x** |
55
-
56
- These numbers rotate over enough distinct weight copies to exceed twice the
57
- card's L2 on both paths. That matters: an 84 MB bf16 weight matrix fits inside
58
- 96 MB of L2, so a naive loop over one matrix measures cache bandwidth and
59
- reports the bf16 baseline as roughly three times faster than it is in a setting
60
- where a 5.9 GB model streams a different layer every step. Measured
61
- cache-resident, the small-`N` rows above invert.
62
-
63
- Weight footprint at hidden 5120:
64
-
65
- | layer | bf16 | ternary (2-bit) | binary (1.125-bit) | vs ternary |
66
- |---|---|---|---|---|
67
- | attention QKV | 80.0 MB | 10.6 MB | 5.6 MB | 1.89x |
68
- | attention out | 50.0 MB | 6.6 MB | 3.5 MB | 1.89x |
69
- | LM head | 2,425.0 MB | 322.1 MB | 170.5 MB | 1.89x |
70
-
71
- 1.89x rather than a nominal 1.78x because the fp16 group scales are a fixed
72
- overhead both formats pay, and they are a larger share of the ternary total.
73
-
74
- ## Correctness
75
-
76
- Verified on RTX 6000 Ada.
77
-
78
- - **Packing is lossless.** `unpack(pack(W))` is `torch.equal` to `W` for every
79
- shape tested, including 6144 x 5120. Zeros are treated as `-1`, so a ternary
80
- tensor deliberately does not round-trip.
81
- - **The integer path is exact.** With unit group scales the product is integer
82
- arithmetic, and the output is `torch.equal` to an exact int32 reference
83
- wherever bf16 can represent the accumulator, which is `|acc| < 256`. This is
84
- the strongest available statement: not "close", equal.
85
- - **The full path** matches a reference that dequantizes the weights and does
86
- the product in fp32 to below `2^-8` normwise, which is the bf16 output
87
- rounding step.
88
- - **Activation quantization** is per-token absmax: the largest magnitude in each
89
- row maps to `±127`, and reconstruction error is under 1% of the row maximum.
90
- - **Group size is a parameter**, verified at 32, 64, 128 and 256.
91
- - **Deterministic.** Repeated products are bitwise identical; the reduction tree
92
- is fixed and there are no atomics.
93
 
94
  ## Usage
95
 
@@ -104,18 +40,13 @@ wq = bg.pack_weights(W) # [N, K//32] int32
104
  ws = group_scales.to(torch.float16) # [N, K//128] fp16
105
 
106
  y = bg.binary_linear(x, wq, ws, group_size=128) # bf16 in, bf16 out
 
 
107
  ```
108
 
109
  `version` selects the release branch; `trust_remote_code` is required by
110
  `kernels` for publishers without the trusted-publisher mark.
111
 
112
- Module form, and conversion from an existing dense layer:
113
-
114
- ```python
115
- layer = bg.BinaryLinear(K, N, group_size=128).cuda()
116
- layer = bg.BinaryLinear.from_dense(nn.Linear(K, N)) # sign + group mean-abs scale
117
- ```
118
-
119
  ## API
120
 
121
  | Symbol | Purpose |
@@ -127,25 +58,78 @@ layer = bg.BinaryLinear.from_dense(nn.Linear(K, N)) # sign + group mean-abs sc
127
  | `binary_linear(x, wq, wscale, bias, group_size)` | one-shot forward |
128
  | `BinaryLinear(in, out, bias, group_size)` | `nn.Module`; `from_dense` converts a dense layer |
129
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  ## Requirements and limits
131
 
132
  - NVIDIA GPU with compute capability 8.0+ (`__dp4a`).
133
  - `K` a multiple of 32 and of `group_size`; bf16 activations and output.
134
- - **Decode only.** This is a warp-per-output-column GEMV structure and it is
135
- built for `M = 1`. Above that it loses to cuBLAS and keeps losing: measured
136
- 0.40x at `M = 2` and 0.02x at `M = 256`, because one warp per column cannot
137
- use tensor cores and the arithmetic stops being bandwidth-bound. Prefill and
138
- batched serving want a separate tile-and-`mma` path, which this kernel does
139
- not implement. Route by batch size.
140
- - Weights must already be binary. Nothing here trains or calibrates them;
141
- `from_dense` is a sign quantizer for testing, not a compression method.
142
 
143
  ## References
144
 
145
- Rastegari et al., "XNOR-Net" (2016), for binary-weight networks; Courbariaux et
146
- al., "BinaryConnect" (2015); Ma et al., "The Era of 1-bit LLMs" (2024) for the
147
- ternary sibling. Group-wise scaling with 1-bit weights as shipped in the Bonsai
148
- family (PrismML, 2026).
149
 
150
  ## License
151
 
 
6
  # binary-gemm
7
 
8
  W1A8 matrix products for binary-weight language models, loadable through
9
+ `kernels`. Each weight is one bit meaning `+1` or `-1`, with one fp16 scale
10
+ per group of `G` weights: at `G = 128` that is 1.125 effective bits per
11
+ weight, against 2 bits for a ternary layer and 16 for bf16. The reference
12
+ baseline is the dequantized product; the integer path is `torch.equal`-exact
13
+ where bf16 can represent the accumulator. Ternary siblings:
14
+ [bitnet-tc](https://huggingface.co/kernels/phanerozoic/bitnet-tc) (CUDA) and
15
+ [bitnet-cpu](https://huggingface.co/kernels/phanerozoic/bitnet-cpu) (CPU).
16
+
17
+ At one bit per weight a model's memory is its scales plus raw sign bits, and
18
+ decode, which reads every weight per token, becomes almost pure bit traffic.
19
+ This kernel multiplies the packed bits directly: a 16-entry constant table
20
+ expands a nibble of the pattern into four packed `±1` int8 values feeding
21
+ `__dp4a`, so the weight is never materialized as a number and the decode wall
22
+ drops with the bytes.
23
+
24
+ ![A real sign-bit field under a scanline next to per-layer decode bars where the binary path runs a fraction of bf16](https://huggingface.co/kernels/phanerozoic/binary-gemm/resolve/main/media/hero.gif)
25
+
26
+ *Actual packed sign bits of a layer (cyan +1, magenta -1), and decode
27
+ measured with weights rotated past L2 in both paths: 2.4x on attention QKV,
28
+ 2.8x on the MLP gate, 3.8x on the 248,320-row LM head.*
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
 
30
  ## Usage
31
 
 
40
  ws = group_scales.to(torch.float16) # [N, K//128] fp16
41
 
42
  y = bg.binary_linear(x, wq, ws, group_size=128) # bf16 in, bf16 out
43
+
44
+ layer = bg.BinaryLinear.from_dense(nn.Linear(K, N)) # sign + group mean-abs scale
45
  ```
46
 
47
  `version` selects the release branch; `trust_remote_code` is required by
48
  `kernels` for publishers without the trusted-publisher mark.
49
 
 
 
 
 
 
 
 
50
  ## API
51
 
52
  | Symbol | Purpose |
 
58
  | `binary_linear(x, wq, wscale, bias, group_size)` | one-shot forward |
59
  | `BinaryLinear(in, out, bias, group_size)` | `nn.Module`; `from_dense` converts a dense layer |
60
 
61
+ ## Method
62
+
63
+ Binary weights collapse the inner product: writing the stored bit as
64
+ `b = (w + 1) / 2`,
65
+
66
+ ```
67
+ dot(w, a) = 2 * sum_{i : b_i = 1} a_i - sum_i a_i
68
+ ```
69
+
70
+ so the weight never has to exist as a number. The kernel keeps the int8
71
+ activation path and resolves four weights at a time through a 16-entry
72
+ constant table that expands a nibble directly into four packed int8 `±1`
73
+ values for `__dp4a`; the table is the whole decode. One warp owns each output
74
+ column: lanes stride the weight groups, accumulate an int32 dot per group,
75
+ apply that group's fp16 scale in fp32, and the warp reduces. The batch
76
+ dimension is tiled at compile time so a weight word is fetched once per tile.
77
+
78
+ ## Measured
79
+
80
+ Bonsai 27B shapes, hidden 5120, `G = 128`, against cuBLAS bf16, weights
81
+ rotated over enough copies to exceed twice the card's L2 on both paths (an
82
+ 84 MB bf16 matrix inside a 96 MB L2 otherwise measures cache bandwidth):
83
+
84
+ | layer | N | cuBLAS bf16 | binary | speedup |
85
+ |---|---|---|---|---|
86
+ | attention QKV | 8,192 | 0.1124 ms | 0.0404 ms | 2.78x |
87
+ | attention out | 5,120 | 0.0733 ms | 0.0276 ms | 2.65x |
88
+ | MLP gate | 17,408 | 0.2425 ms | 0.0756 ms | 3.21x |
89
+ | MLP down | 5,120 | 0.0738 ms | 0.0272 ms | 2.71x |
90
+ | LM head | 248,320 | 3.3622 ms | 0.9637 ms | 3.49x |
91
+
92
+ Weight footprint at hidden 5120:
93
+
94
+ | layer | bf16 | ternary (2-bit) | binary (1.125-bit) | vs ternary |
95
+ |---|---|---|---|---|
96
+ | attention QKV | 80.0 MB | 10.6 MB | 5.6 MB | 1.89x |
97
+ | attention out | 50.0 MB | 6.6 MB | 3.5 MB | 1.89x |
98
+ | LM head | 2,425.0 MB | 322.1 MB | 170.5 MB | 1.89x |
99
+
100
+ 1.89x rather than the nominal 1.78x because the fp16 group scales are a
101
+ fixed overhead both formats pay.
102
+
103
+ ## Correctness
104
+
105
+ - Packing is lossless: `unpack(pack(W))` is `torch.equal` to `W` for every
106
+ shape tested. Zeros are treated as `-1`, so a ternary tensor deliberately
107
+ does not round-trip.
108
+ - The integer path is exact: with unit group scales the output is
109
+ `torch.equal` to an exact int32 reference wherever bf16 represents the
110
+ accumulator (`|acc| < 256`).
111
+ - The full path matches a dequantize-and-fp32 reference to below `2^-8`
112
+ normwise, the bf16 output rounding step.
113
+ - Activation quantization is per-token absmax with reconstruction error
114
+ under 1% of the row maximum; group size verified at 32, 64, 128, 256.
115
+ - Deterministic: repeated products are bitwise identical; fixed reduction
116
+ tree, no atomics.
117
+
118
  ## Requirements and limits
119
 
120
  - NVIDIA GPU with compute capability 8.0+ (`__dp4a`).
121
  - `K` a multiple of 32 and of `group_size`; bf16 activations and output.
122
+ - Decode only: a warp-per-column GEMV built for `M = 1`. Above that it loses
123
+ to cuBLAS (0.40x at `M = 2`, 0.02x at `M = 256`); route prefill and batch
124
+ elsewhere.
125
+ - Weights must already be binary; `from_dense` is a sign quantizer for
126
+ testing, not a compression method.
 
 
 
127
 
128
  ## References
129
 
130
+ Rastegari et al., "XNOR-Net" (2016); Courbariaux et al., "BinaryConnect"
131
+ (2015); Ma et al., "The Era of 1-bit LLMs" (2024); group-wise scaling with
132
+ 1-bit weights as shipped in the Bonsai family (PrismML, 2026).
 
133
 
134
  ## License
135