web: TEST_RESULTS with QKV fusion
Browse files- web/TEST_RESULTS.md +23 -0
web/TEST_RESULTS.md
CHANGED
|
@@ -85,6 +85,29 @@ gradient FNV hash compared old vs new on-device):
|
|
| 85 |
rate moved. At equal wall clock float still wins (~1.5% at the 200-step
|
| 86 |
horizon), so `unitBackward` stays off by default.
|
| 87 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
One bug was caught during the rework, by the bit-identity check itself: the
|
| 89 |
fused q+k+v sum initially ran in f64 and rounded once, where the old code
|
| 90 |
rounded to f32 after each add — a last-ulp fork that would have split
|
|
|
|
| 85 |
rate moved. At equal wall clock float still wins (~1.5% at the 200-step
|
| 86 |
horizon), so `unitBackward` stays off by default.
|
| 87 |
|
| 88 |
+
## QKV dual-GEMM fusion (CUTLASS ex. 45)
|
| 89 |
+
|
| 90 |
+
The q/k/v projections share the same left operand (`ln1.y`), so the forward
|
| 91 |
+
now quantizes it ONCE and runs all three as one batched (batch=3) dispatch —
|
| 92 |
+
2 fewer dispatches and 2 fewer full quantize passes per layer per step.
|
| 93 |
+
|
| 94 |
+
Bit-identity (old three-GEMM forward vs fused), 5 full training steps with
|
| 95 |
+
weight updates in between so a single-ulp divergence anywhere compounds:
|
| 96 |
+
|
| 97 |
+
```
|
| 98 |
+
char-96, float backward: 5 losses + 5×20480 grads + final weights bit-identical
|
| 99 |
+
char-96, unit backward: 5 losses + 5×20480 grads + final weights bit-identical
|
| 100 |
+
Spikewhale 16k, float backward: 5 losses + 5×545792 grads + final weights bit-identical
|
| 101 |
+
Spikewhale 16k, unit backward: 5 losses + 5×545792 grads + final weights bit-identical
|
| 102 |
+
generate() output identical (the fused output's subarray views feed attention)
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
On GPU (DP4A): gradient FNV hashes match the pre-fusion values exactly in
|
| 106 |
+
both modes (`7ff11308` float / `62596547` units); ~2% wall-clock gain at
|
| 107 |
+
width 32 (the shared operand is only 32 KB there — the saved quantize work
|
| 108 |
+
scales quadratically with model width). Two-device live run: both replicas
|
| 109 |
+
at step 71/300 with identical loss to the last digit, no sync-guard trips.
|
| 110 |
+
|
| 111 |
One bug was caught during the rework, by the bit-identity check itself: the
|
| 112 |
fused q+k+v sum initially ran in f64 and rounded once, where the old code
|
| 113 |
rounded to f32 after each add — a last-ulp fork that would have split
|