Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -38,6 +38,108 @@ it is **not** a substitute for a real GPU on large models. Full envelope in
|
|
| 38 |
|
| 39 |
---
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
## Quick start
|
| 42 |
|
| 43 |
### Docker (most reliable β one command)
|
|
|
|
| 38 |
|
| 39 |
---
|
| 40 |
|
| 41 |
+
## Feature list
|
| 42 |
+
|
| 43 |
+
### Python cluster trainer (`daisychain/`)
|
| 44 |
+
- **Data-parallel training across mixed machines** β each node trains its own
|
| 45 |
+
shard; gradients combine into the exact full-batch gradient, replicas stay
|
| 46 |
+
bit-identical.
|
| 47 |
+
- **Capacity-weighted sharding** β faster machines automatically take a
|
| 48 |
+
bigger share of the batch.
|
| 49 |
+
- **Emulated GPU compute (verified INT8 units)** β `VerifiedLinear` layers
|
| 50 |
+
run every forward multiply / requantize / ReLU through the bundled trained
|
| 51 |
+
units; cluster-wide unit-invocation counts printed by rank 0.
|
| 52 |
+
- **Bring your own model** β any `Task` (`build_model` / `sample` / `loss`)
|
| 53 |
+
via `DAISY_TASK`; template in `examples/my_task_template.py`.
|
| 54 |
+
- **Plain-float alternative task** β same cluster and pooling with ordinary
|
| 55 |
+
float math.
|
| 56 |
+
- **Live dashboard** (`daisychain-dashboard`) β readiness banner, P2P
|
| 57 |
+
connectivity scan, pooled cores/RAM, per-node capacity plan, live loss.
|
| 58 |
+
- **SpikeWhale control panel** (`spikewhale_panel`, `localhost:8899`) β
|
| 59 |
+
sliders for model size / training settings, any HF dataset you can access
|
| 60 |
+
(default streamed FineWeb-Edu), start/stop/re-adjust, live loss.
|
| 61 |
+
- **Docker demo cluster** β 3 nodes + dashboard in one command.
|
| 62 |
+
- **Windows helper** (`scripts\setup.bat`) and Tailscale mesh guide.
|
| 63 |
+
|
| 64 |
+
### DaisyChain-Web (`web/`) β browser P2P training
|
| 65 |
+
- **Zero-install nodes** β opening the page IS joining; devices on one
|
| 66 |
+
network auto-group (Snapdrop-style, by public IP).
|
| 67 |
+
- **Private cross-network rooms** β `?room=CODE` with **host approval** for
|
| 68 |
+
every join.
|
| 69 |
+
- **Full WebRTC mesh** β gradients travel peer-to-peer; the server only
|
| 70 |
+
signals and serves static files, it never sees weights or gradients.
|
| 71 |
+
- **Leader-follower runs** β whoever presses Start sets width / sequence /
|
| 72 |
+
batch-per-device / steps / learning rate for the whole group; config
|
| 73 |
+
broadcast on the wire.
|
| 74 |
+
- **Mid-run join** β late devices are synced in (weights + step) and
|
| 75 |
+
contribute from the next step.
|
| 76 |
+
- **Bit-identical replicas** β same seeded init, strict roster-order gradient
|
| 77 |
+
averaging, deterministic Adam with identical state on every peer; verified
|
| 78 |
+
live by per-step weight hashes.
|
| 79 |
+
- **Sync guard** β any weight-hash mismatch stops the run instead of
|
| 80 |
+
training past a fork; the step roster forbids silent partial averages.
|
| 81 |
+
- **Gradient repair** β a follower missing a roster gradient re-requests it
|
| 82 |
+
from the leader (8 steps retained), bit-exact, and the run continues.
|
| 83 |
+
- **Cross-device kernel probe** β every step, every device re-hashes a fixed
|
| 84 |
+
seeded int8 GEMM through its live kernel; catches broken arithmetic that
|
| 85 |
+
weight hashes cannot see.
|
| 86 |
+
- **Hardcoded FineWeb-Edu streaming** β the server reads random slices of
|
| 87 |
+
the 10BT parquet shards straight off the HF CDN via HTTP range requests
|
| 88 |
+
(pure-JS `hyparquet`); built-in corpus fallback offline.
|
| 89 |
+
- **Checkpoints** β download `.pt`, upload β **broadcast to the whole
|
| 90 |
+
group**; validated (magic, dims, tokenizer vocab) before accepting.
|
| 91 |
+
- **Inference kit** β one self-contained HTML file with the trained weights
|
| 92 |
+
baked in; generations offline, anywhere.
|
| 93 |
+
- **In-page generation** β prompt box on the trained model.
|
| 94 |
+
- **Old-hardware tier** β no WebGPU? The identical units run on CPU (same
|
| 95 |
+
bits, so CPU and GPU devices co-train in one group). There is no
|
| 96 |
+
plain-float path.
|
| 97 |
+
- **Large-message fragmentation** β multi-MB gradients/checkpoints chunked
|
| 98 |
+
at 48 KB over the data channels.
|
| 99 |
+
|
| 100 |
+
### Verified compute & kernels (web)
|
| 101 |
+
- **Verified INT8 units everywhere** β block-scaled int8 GEMM: exact LUT
|
| 102 |
+
products, exact int32 accumulation, bit-exact f32 epilogue with a pinned
|
| 103 |
+
rounding schedule; scales derived in JS f64 (division never runs on GPU).
|
| 104 |
+
- **Backends, best-first** β DP4A hardware int8 dot β LUT compute shader β
|
| 105 |
+
CPU mirror; every kernel **exact-gated at init** (bit-level compares) and
|
| 106 |
+
demoted to the mirror on any mismatch.
|
| 107 |
+
- **Continuous random-cell audit** at live training shapes.
|
| 108 |
+
- **Fused attention kernels** β gather/scatter head-strided qΒ·kα΅ and aΒ·v
|
| 109 |
+
straight from BTΓC layout (CUTLASS ex. 36/52 style).
|
| 110 |
+
- **QKV dual-GEMM fusion** β shared left operand quantized once, one batch-3
|
| 111 |
+
dispatch (ex. 45); bit-identical.
|
| 112 |
+
- **B2B MLP chain** β both MLP GEMMs back-to-back on GPU with fused per-row
|
| 113 |
+
absmax reduction and on-device quantize (ex. 13 + 23); WGSL-exact respec
|
| 114 |
+
with a fround-stepped JS mirror; fma-contraction-immune by construction.
|
| 115 |
+
- **Dispatch-optimized backward** β overlapped independent GEMMs, batch-3
|
| 116 |
+
sibling fusions (ex. 05/24); bit-identical gradients; optional int8 STE
|
| 117 |
+
backward path (dormant, 1.21Γ vs float).
|
| 118 |
+
|
| 119 |
+
### Verification stack (web)
|
| 120 |
+
- **Exact init gates** on every kernel, every device, every boot β including
|
| 121 |
+
gates that "gate the gate" with discriminating boundary inputs.
|
| 122 |
+
- **IEEE-754 binary32 oracle** in exact BigInt arithmetic β proves the JS
|
| 123 |
+
epilogue mirror is spec-correct (rejects the old mirror on 34% of inputs).
|
| 124 |
+
- **Metamorphic property suite** β reference-free relations + definitional
|
| 125 |
+
absolutes; **4/4** on an externally-authored bug corpus, matching the
|
| 126 |
+
differential gate.
|
| 127 |
+
- **RDNA2 ISA audit hardenings** β bit-level (β0-aware) gate comparisons;
|
| 128 |
+
proof that FMA contraction cannot change the quantize.
|
| 129 |
+
- **Ten-suite test chain** (`cd web && npm test`) β convergence, replicas,
|
| 130 |
+
oracle, gates, properties, corpus, B2B, optimizer, transformer LM, int8
|
| 131 |
+
backward; results in [web/TEST_RESULTS.md](web/TEST_RESULTS.md).
|
| 132 |
+
|
| 133 |
+
### Documentation
|
| 134 |
+
- Python: [QUICKSTART](docs/QUICKSTART.md), [LIMITS](docs/LIMITS.md),
|
| 135 |
+
[CUSTOM_TASK](docs/CUSTOM_TASK.md), [TAILSCALE](docs/TAILSCALE.md).
|
| 136 |
+
- Web: [Getting started](web/docs/GETTING_STARTED.md),
|
| 137 |
+
[Architecture](web/docs/ARCHITECTURE.md),
|
| 138 |
+
[Verification](web/docs/VERIFICATION.md),
|
| 139 |
+
[Troubleshooting](web/docs/TROUBLESHOOTING.md).
|
| 140 |
+
|
| 141 |
+
---
|
| 142 |
+
|
| 143 |
## Quick start
|
| 144 |
|
| 145 |
### Docker (most reliable β one command)
|