Quazim0t0 commited on
Commit
12e3142
·
verified ·
1 Parent(s): 30eb3fa

web: TEST_RESULTS with QKV fusion

Browse files
Files changed (1) hide show
  1. web/TEST_RESULTS.md +23 -0
web/TEST_RESULTS.md CHANGED
@@ -85,6 +85,29 @@ gradient FNV hash compared old vs new on-device):
85
  rate moved. At equal wall clock float still wins (~1.5% at the 200-step
86
  horizon), so `unitBackward` stays off by default.
87
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
  One bug was caught during the rework, by the bit-identity check itself: the
89
  fused q+k+v sum initially ran in f64 and rounded once, where the old code
90
  rounded to f32 after each add — a last-ulp fork that would have split
 
85
  rate moved. At equal wall clock float still wins (~1.5% at the 200-step
86
  horizon), so `unitBackward` stays off by default.
87
 
88
+ ## QKV dual-GEMM fusion (CUTLASS ex. 45)
89
+
90
+ The q/k/v projections share the same left operand (`ln1.y`), so the forward
91
+ now quantizes it ONCE and runs all three as one batched (batch=3) dispatch —
92
+ 2 fewer dispatches and 2 fewer full quantize passes per layer per step.
93
+
94
+ Bit-identity (old three-GEMM forward vs fused), 5 full training steps with
95
+ weight updates in between so a single-ulp divergence anywhere compounds:
96
+
97
+ ```
98
+ char-96, float backward: 5 losses + 5×20480 grads + final weights bit-identical
99
+ char-96, unit backward: 5 losses + 5×20480 grads + final weights bit-identical
100
+ Spikewhale 16k, float backward: 5 losses + 5×545792 grads + final weights bit-identical
101
+ Spikewhale 16k, unit backward: 5 losses + 5×545792 grads + final weights bit-identical
102
+ generate() output identical (the fused output's subarray views feed attention)
103
+ ```
104
+
105
+ On GPU (DP4A): gradient FNV hashes match the pre-fusion values exactly in
106
+ both modes (`7ff11308` float / `62596547` units); ~2% wall-clock gain at
107
+ width 32 (the shared operand is only 32 KB there — the saved quantize work
108
+ scales quadratically with model width). Two-device live run: both replicas
109
+ at step 71/300 with identical loss to the last digit, no sync-guard trips.
110
+
111
  One bug was caught during the rework, by the bit-identity check itself: the
112
  fused q+k+v sum initially ran in f64 and rounded once, where the old code
113
  rounded to f32 after each add — a last-ulp fork that would have split