| --- |
| license: mit |
| tags: |
| - distributed-training |
| - old-hardware |
| - int8 |
| - webgpu |
| - webrtc |
| --- |
| |
| # 🌼 DaisyChain-Train — Old Hardware Training Pipeline |
|
|
| **Part of DaisyChain on 🤗 Hugging Face → https://huggingface.co/DaisyChainAI** |
| Model page (weights + card): https://huggingface.co/DaisyChainAI/DaisyChain-Train |
|
|
| --- |
|
|
| > **In plain terms:** DaisyChain-Train lets you use **old / spare machines** to |
| > train neural networks. The training runs through **emulated GPU logic** — |
| > verified INT8 units (GUDA-style) that stand in for a GPU's math — so machines |
| > *without* a modern GPU can still do the work. Chain several together and they |
| > train one shared model as a cluster. |
| > Before you rely on it, see what it **can't** do → [Limitations](docs/LIMITS.md). |
|
|
| **Use the hardware you already have to train.** Each machine runs the emulated |
| GPU logic (verified INT8 units — multiply / requantize / ReLU) to compute the |
| model, and DaisyChain pools the machines data-parallel: device selection, |
| capacity-weighted sharding, gradient sync, a P2P setup, and a live dashboard. |
| Two ways to run — **Docker** or **Python**. |
|
|
| --- |
|
|
| ## ⚠️ Read this first |
| DaisyChain-Train is for **small models on spare hardware**. It **pools compute, |
| not memory** (the model must fit on one machine), scaling is **sublinear**, and |
| it is **not** a substitute for a real GPU on large models. Full envelope in |
| **[docs/LIMITS.md](docs/LIMITS.md)** — please read it before relying on it. |
|
|
| --- |
|
|
| ## Feature list |
|
|
| ### Python cluster trainer (`daisychain/`) |
| - **Data-parallel training across mixed machines** — each node trains its own |
| shard; gradients combine into the exact full-batch gradient, replicas stay |
| bit-identical. |
| - **Capacity-weighted sharding** — faster machines automatically take a |
| bigger share of the batch. |
| - **Emulated GPU compute (verified INT8 units)** — `VerifiedLinear` layers |
| run every forward multiply / requantize / ReLU through the bundled trained |
| units; cluster-wide unit-invocation counts printed by rank 0. |
| - **Bring your own model** — any `Task` (`build_model` / `sample` / `loss`) |
| via `DAISY_TASK`; template in `examples/my_task_template.py`. |
| - **Plain-float alternative task** — same cluster and pooling with ordinary |
| float math. |
| - **Live dashboard** (`daisychain-dashboard`) — readiness banner, P2P |
| connectivity scan, pooled cores/RAM, per-node capacity plan, live loss. |
| - **SpikeWhale control panel** (`spikewhale_panel`, `localhost:8899`) — |
| sliders for model size / training settings, any HF dataset you can access |
| (default streamed FineWeb-Edu), start/stop/re-adjust, live loss. |
| - **Docker demo cluster** — 3 nodes + dashboard in one command. |
| - **Windows helper** (`scripts\setup.bat`) and Tailscale mesh guide. |
|
|
| ### DaisyChain-Web (`web/`) — browser P2P training |
| - **Zero-install nodes** — opening the page IS joining; devices on one |
| network auto-group (Snapdrop-style, by public IP). |
| - **Private cross-network rooms** — `?room=CODE` with **host approval** for |
| every join. |
| - **Full WebRTC mesh** — gradients travel peer-to-peer; the server only |
| signals and serves static files, it never sees weights or gradients. |
| - **Leader-follower runs** — whoever presses Start sets width / sequence / |
| batch-per-device / steps / learning rate for the whole group; config |
| broadcast on the wire. |
| - **Mid-run join** — late devices are synced in (weights + step) and |
| contribute from the next step. |
| - **Bit-identical replicas** — same seeded init, strict roster-order gradient |
| averaging, deterministic Adam with identical state on every peer; verified |
| live by per-step weight hashes. |
| - **Sync guard** — any weight-hash mismatch stops the run instead of |
| training past a fork; the step roster forbids silent partial averages. |
| - **Gradient repair** — a follower missing a roster gradient re-requests it |
| from the leader (8 steps retained), bit-exact, and the run continues. |
| - **Cross-device kernel probe** — every step, every device re-hashes a fixed |
| seeded int8 GEMM through its live kernel; catches broken arithmetic that |
| weight hashes cannot see. |
| - **Hardcoded FineWeb-Edu streaming** — the server reads random slices of |
| the 10BT parquet shards straight off the HF CDN via HTTP range requests |
| (pure-JS `hyparquet`); built-in corpus fallback offline. |
| - **Checkpoints** — download `.pt`, upload → **broadcast to the whole |
| group**; validated (magic, dims, tokenizer vocab) before accepting. |
| - **Inference kit** — one self-contained HTML file with the trained weights |
| baked in; generations offline, anywhere. |
| - **In-page generation** — prompt box on the trained model. |
| - **Old-hardware tier** — no WebGPU? The identical units run on CPU (same |
| bits, so CPU and GPU devices co-train in one group). There is no |
| plain-float path. |
| - **Large-message fragmentation** — multi-MB gradients/checkpoints chunked |
| at 48 KB over the data channels. |
|
|
| ### Verified compute & kernels (web) |
| - **Verified INT8 units everywhere** — block-scaled int8 GEMM: exact LUT |
| products, exact int32 accumulation, bit-exact f32 epilogue with a pinned |
| rounding schedule; scales derived in JS f64 (division never runs on GPU). |
| - **Backends, best-first** — DP4A hardware int8 dot → LUT compute shader → |
| CPU mirror; every kernel **exact-gated at init** (bit-level compares) and |
| demoted to the mirror on any mismatch. |
| - **Continuous random-cell audit** at live training shapes. |
| - **Fused attention kernels** — gather/scatter head-strided q·kᵀ and a·v |
| straight from BT×C layout (CUTLASS ex. 36/52 style). |
| - **QKV dual-GEMM fusion** — shared left operand quantized once, one batch-3 |
| dispatch (ex. 45); bit-identical. |
| - **B2B MLP chain** — both MLP GEMMs back-to-back on GPU with fused per-row |
| absmax reduction and on-device quantize (ex. 13 + 23); WGSL-exact respec |
| with a fround-stepped JS mirror; fma-contraction-immune by construction. |
| - **Dispatch-optimized backward** — overlapped independent GEMMs, batch-3 |
| sibling fusions (ex. 05/24); bit-identical gradients; optional int8 STE |
| backward path (dormant, 1.21× vs float). |
|
|
| ### Verification stack (web) |
| - **Exact init gates** on every kernel, every device, every boot — including |
| gates that "gate the gate" with discriminating boundary inputs. |
| - **IEEE-754 binary32 oracle** in exact BigInt arithmetic — proves the JS |
| epilogue mirror is spec-correct (rejects the old mirror on 34% of inputs). |
| - **Metamorphic property suite** — reference-free relations + definitional |
| absolutes; **4/4** on an externally-authored bug corpus, matching the |
| differential gate. |
| - **RDNA2 ISA audit hardenings** — bit-level (−0-aware) gate comparisons; |
| proof that FMA contraction cannot change the quantize. |
| - **Eleven-suite test chain** (`cd web && npm test`) — convergence, replicas, |
| oracle, gates, properties, external corpus, **self-corpus** (the instruments |
| scored against my own bugs), B2B, optimizer, transformer LM, int8 backward; |
| results in [web/TEST_RESULTS.md](web/TEST_RESULTS.md). |
| - **Dirty-buffer gate** — the pool is poisoned before a re-sweep so state bugs |
| (a kernel assuming zeroed memory) are caught deterministically rather than |
| by ordering luck. |
|
|
| ### Documentation |
| - Python: [QUICKSTART](docs/QUICKSTART.md), [LIMITS](docs/LIMITS.md), |
| [CUSTOM_TASK](docs/CUSTOM_TASK.md), [TAILSCALE](docs/TAILSCALE.md). |
| - Web: [Getting started](web/docs/GETTING_STARTED.md), |
| [Architecture](web/docs/ARCHITECTURE.md), |
| [Verification](web/docs/VERIFICATION.md), |
| [Troubleshooting](web/docs/TROUBLESHOOTING.md). |
|
|
| --- |
|
|
| ## Quick start |
|
|
| ### Docker (most reliable — one command) |
| ```bash |
| docker compose -f docker/docker-compose.yml up --build |
| # open http://localhost:8080 |
| ``` |
| Brings up a 3-node demo cluster + dashboard on one machine. |
|
|
| ### Python (real machines) |
| On every machine (`pip install -e .`): |
| ```bash |
| export MASTER_ADDR=100.101.102.10 # coordinator IP (Tailscale 100.x recommended) |
| export MASTER_PORT=29560 |
| export WORLD_SIZE=3 |
| export RANK=0 # 1, 2, ... on the others |
| export GLOO_SOCKET_IFNAME=tailscale0 # your mesh / LAN NIC |
| daisychain-train |
| ``` |
|
|
| ### Windows helper |
| ```bat |
| scripts\setup.bat |
| ``` |
| An interactive menu: Docker, Python node, or just install deps. |
|
|
| Full walkthrough: **[docs/QUICKSTART.md](docs/QUICKSTART.md)**. |
|
|
| ### 🐋 SpikeWhale control panel (sliders → real training) |
| ```bash |
| python -m daisychain.spikewhale_panel |
| # open http://localhost:8899 |
| ``` |
| A web control panel: pick model size / training settings with sliders, choose any |
| HuggingFace dataset you have access to (default: streamed FineWeb-Edu), hit |
| Start, and watch the live loss. Stop and re-adjust any time with |
| **← Back to settings**. Launches the real DaisyChain training underneath. |
|
|
| ### 🌐 DaisyChain-Web (train by opening a browser tab) |
| ```bash |
| cd web && npm install && node server.js |
| # open http://localhost:8787 on every device |
| ``` |
| Zero-install browser training: devices on the same network auto-group |
| (Snapdrop-style) and train a shared model **peer-to-peer over WebRTC**, computing |
| through the same verified INT8 units (WebGPU, with the identical units on CPU |
| for machines without it — there is no plain-float path). Private cross-network |
| rooms via `?room=CODE` with **host approval** — the room creator accepts each |
| device before it can join. Includes gradient averaging with a deterministic |
| Adam optimizer (identical state on every peer, nothing extra over the wire), |
| checkpoint **download** (.pt) and **upload → broadcast** so one device can |
| restore the whole group after a failure. |
|
|
| **Live demo:** https://huggingface.co/spaces/Quazim0t0/DaisyChain-Web |
|
|
| ### Recent updates (July 2026) — DaisyChain-Web |
|
|
| **Verification stack** — the browser trainer's correctness is now *checked by |
| things that run*, not argued ([full results](web/TEST_RESULTS.md)): |
| - **IEEE-754 oracle** (`web/test_ieee.js`): a binary32 oracle built from the |
| standard in exact BigInt arithmetic proves the JS epilogue mirror is |
| spec-correct — and rejects the old round-once mirror on 34% of inputs. |
| - **Metamorphic properties + oracle mutation scoring** (`test_metamorphic.js`, |
| `test_corpus.js`): properties needing no reference implementation, scored |
| against an externally-authored bug taxonomy — **4/4**, matching the exact |
| differential gate's 4/4. Relations own the loop bugs; two definitional |
| absolutes (ReLU output range, a unit-scale integer anchor) own the value |
| bugs no relation can see. |
| - **Exact kernel gates on every live kernel**, a continuous audit at live |
| shapes, and a cross-device kernel probe (same seeded int8 GEMM, same hash on |
| every honest device, any backend). The audit's sampling was rebuilt against |
| a named bug class: its old constants (6 cells, 2% of GEMMs) bounded an |
| overhead that had never been measured (auditing *every* GEMM costs <0.01% of |
| a step), and uniform random cells cannot see a last-row/column bug at a |
| 16512-wide output. Sampling is now **stratified** — the first cells are the |
| structural danger points, chosen deliberately. On a last-column bug, same |
| cell budget: uniform caught **5/300** audits, stratified **300/300**, with |
| zero false positives. |
| - **RDNA2 ISA audit**: reading a real GPU's shader ISA against our |
| determinism assumptions confirmed three of them on silicon (exact packed |
| int8 dot; correctly-rounded f32 add/mul; 1-ULP reciprocal — division stays |
| off the GPU) and produced two hardenings. (1) Real ISAs have non-IEEE |
| variants that **flush −0 to +0**; JS `!==` can't see that (`-0 !== 0` is |
| false), so all gates and audits now compare **bit patterns** — exactly what |
| the replica hash sees. (2) FMA contraction of the quantize's `x·inv + 0.5` |
| (one rounding instead of two) turned out to be **floor-invisible by |
| construction** — proven in `test_b2b.js` with 175k+ last-ulp anomalies at |
| binade edges, zero surviving `floor()`. Rounding mode and denorm flushing |
| are runtime driver state on real hardware, which is why every device |
| re-runs the exact gates at every init. |
|
|
| **Training data** — FineWeb-Edu (10BT sample) is the hardcoded dataset. The |
| Space reads random slices of the parquet shards straight off the HF CDN with |
| range requests (pure-JS `hyparquet`, SNAPPY) and serves plain text at `/data` |
| — no dependency on the datasets-server rows API and its 503s. |
|
|
| **Resilience** — the sync guard now *repairs* instead of halting: a roster |
| gradient that reached the leader but not some follower (asymmetric WebRTC |
| mesh) is re-requested from the leader, bit-exact, and the run continues. The |
| guard still stops anything that would fork the weights. |
|
|
| **CUTLASS-style kernel work**, each step proven bit-identical or exact-gated: |
| - **Dispatch-optimized backward** (ex. 05/24): independent GEMMs overlapped, |
| sibling trios fused into batch-3 dispatches — bit-identical gradients, |
| dormant int8-backward path down from 1.63× to 1.21× vs float. |
| - **QKV dual-GEMM fusion** (ex. 45): q/k/v share one left operand — quantized |
| once, one batched dispatch, zero changed bits. |
| - **B2B MLP chain** (ex. 13 + 23): both MLP GEMMs back-to-back on the GPU with |
| a fused per-row absmax reduction; the intermediate is quantized on-device |
| via a WGSL-exact respec (`floor(f32(x·invScale)+0.5)` — no GPU division) |
| whose fround-stepped JS mirror keeps mixed GPU/CPU fleets bit-identical. |
|
|
| **Profile-driven speed work** — every change below is bit-identical (gradient |
| and loss hashes unchanged), so none of it trades correctness for wall clock: |
| - **Buffer pooling**: GPU buffers are recycled by size bucket instead of being |
| created and destroyed per dispatch (~19 per MLP call, per layer, per step). |
| **6–10% faster**, every hash unchanged. |
| - **Shared-operand embedding GEMMs**: profiling put two f32 backward GEMMs at |
| **55% of the entire step**, and both consumed the same `dlogits` operand — |
| ~17 MB at the 16512-token vocab, uploaded *twice*. One upload, one encoder, |
| one submit: that pair went 205 → 90 ms and the step **12% faster**. The |
| fusion is gated bit-for-bit against the two calls it replaces. |
| - **A negative result, kept on purpose**: the remaining hot kernel looked |
| cache-hostile (adjacent lanes wrote 66 KB apart), but making the writes |
| contiguous changed nothing. Two probes explain why — holding the output at |
| 17 MB while cutting compute 32× barely moved the time. The logits GEMM is |
| **transfer-bound, not compute-bound**, and the readback cannot be removed |
| because softmax must stay in JS (WGSL's `exp` is not correctly rounded, and |
| a per-vendor `exp` would fork replicas). The real lever there is the |
| vocabulary, not the kernel. |
| - **An init backend race was tried and removed**: it tied on the shipped path, |
| cost ~430 ms of init, and made the backend vary between page loads, which |
| silently invalidated three A/B comparisons before it was caught. A knob that |
| changes what you are measuring is worse than a fixed choice. |
|
|
| **Dirty-buffer gate — and the assumption it falsified.** Pooling introduced a |
| bug class the gates predate: a pooled buffer is *not* zero-initialized, so a |
| kernel that assumes zeros is right on step one and wrong on step two. That is |
| a **state** bug, where no single call is wrong and the *sequence* is, which is |
| the family no oracle can reach. The assumption was that the gates were blind |
| to it. Mutation-testing the gate proved otherwise: deleting the zeroing made |
| the plain gate fail at its *second* shape, because the sweep's own shapes |
| recycle each other's buffers. The suite had **incidental** coverage nobody |
| designed, which is coverage nobody can rely on — shorten the shape list and it |
| evaporates with the gate still green. It is now deliberate: the pool is |
| poisoned with 1e4-magnitude residue before a re-sweep, so detection no longer |
| depends on ordering luck. ~90 ms one-time. |
|
|
| **Scoring the oracles against my OWN bugs** (`web/test_selfcorpus.js`) — the |
| external corpus measures kernel bugs someone else wrote down, so this suite |
| asks the harder question: what do the instruments score against the four real |
| bugs of the month? Properties **0/2** on the data-plane pair (the `c·out` |
| theorem again), differential **2/2** — but half the bugs were not in the |
| kernels at all. A dead gate is a bug in a *checker*, caught only by mutating |
| the gate; a stalled roster gradient is a bug in the *protocol*, where every |
| computed value on every peer was correct, so no data oracle could fire. Those |
| needed different instruments, not better oracles. |
|
|
| All eleven test suites (`cd web && npm test`) pass; results with methodology in |
| [`web/TEST_RESULTS.md`](web/TEST_RESULTS.md). |
|
|
| --- |
|
|
| ## How it works |
|
|
| Each machine runs the **same** command; they form a cluster and train one shared |
| model. Two things happen: |
|
|
| 1. **The compute runs through the emulated GPU logic.** By default the model is |
| built from `VerifiedLinear` layers, so every forward multiply / requantize / |
| ReLU is done by the **bundled verified INT8 units** (`daisychain/verified/`) — |
| the emulated GPU math. Rank 0 prints **cluster-wide unit-invocation counts** |
| so you can see the emulated logic doing the work. |
| 2. **The machines are pooled data-parallel.** Each node trains on its own shard; |
| gradients are capacity-weighted and combined into the exact full-batch |
| gradient, so replicas stay **bit-identical**. Faster machines automatically |
| take a bigger share. |
|
|
| ``` |
| old machine A ─┐ |
| old machine B ─┼─► each runs the emulated GPU logic on its shard ─► one model |
| old machine C ─┘ (gradients combined across the cluster) |
| ``` |
|
|
| ## Bring your own model |
| DaisyChain-Train trains any **Task** (`build_model` / `sample` / `loss`). Copy |
| `examples/my_task_template.py`, set `DAISY_TASK=your_module:YourTask`. Use |
| `VerifiedLinear` (see `daisychain/verified_task.py`) to run your model's compute |
| through the emulated units. See **[docs/CUSTOM_TASK.md](docs/CUSTOM_TASK.md)**. |
|
|
| ## Plain-float alternative |
| To skip the emulated units and train with normal float math on each machine, set |
| `DAISY_TASK=daisychain.example_task:ExampleTask`. Same cluster, same pooling — |
| the model math just runs as ordinary float instead of through the verified units. |
|
|
| ## The dashboard |
| `daisychain-dashboard` (or the Docker service) serves a Tailwind page at `:8080` |
| — readiness banner, P2P connectivity scan, pooled cores/RAM + capacity plan |
| (per-node device, weight, batch), and live training loss. |
|
|
| ## Networking |
| Use **Tailscale** for a P2P mesh so machines on different networks get stable IPs |
| on one interface — **[docs/TAILSCALE.md](docs/TAILSCALE.md)**. |
|
|
| --- |
|
|
| ## Layout |
| ``` |
| daisychain/cluster.py capacity-weighted CPU/GPU data-parallel trainer |
| daisychain/train.py entry point (daisychain-train) |
| daisychain/verified/ bundled trained N/N units + VerifiedLinear (train through them) |
| daisychain/verified_task.py default task: forward runs on the verified units |
| daisychain/example_task.py plain-float alternative task |
| daisychain/task.py the Task interface + loader |
| daisychain/dashboard/ agent + P2P scanner + Tailwind server |
| docker/ Dockerfile, dashboard image, compose (demo cluster) |
| scripts/setup.bat / setup.sh interactive setup helpers |
| config/ nodes + cluster env examples |
| examples/my_task_template.py starting point for your own model |
| docs/ QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE |
| daisychain/spikewhale_task.py trains the real SpikeWhale on streamed HF datasets |
| daisychain/spikewhale_panel.py slider control panel (localhost:8899) |
| web/ DaisyChain-Web: P2P browser training (WebRTC + WebGPU) |
| export_luts_web.py regenerates web/public LUTs from the trained units |
| ``` |
|
|
| ## Install |
| ```bash |
| pip install torch numpy psutil |
| pip install -e . # exposes: daisychain-train, daisychain-agent, daisychain-dashboard |
| ``` |
| Requires Python ≥ 3.9, PyTorch ≥ 2.0. Multi-node is reliable on **Linux/macOS**; |
| on **Windows use Docker/WSL** (see [Limitations](docs/LIMITS.md)). |
|
|
| --- |
|
|
| ## Links |
| - **DaisyChain on Hugging Face:** https://huggingface.co/DaisyChainAI |
| - **This model:** https://huggingface.co/DaisyChainAI/DaisyChain-Train |
|
|
| **License:** MIT · **Author:** Dean Byrne (Quazim0t0) · **Org:** DaisyChainAI |
|
|
| ## Citation |
| ```bibtex |
| @misc{byrne2026daisychain, |
| title = {DaisyChain-Train: An Old Hardware Training Pipeline}, |
| author = {Byrne, Dean (Quazim0t0)}, |
| year = {2026}, |
| howpublished = {\url{https://huggingface.co/DaisyChainAI/DaisyChain-Train}}, |
| note = {Chain spare/old machines into a data-parallel training cluster} |
| } |
| ``` |
|
|
| **Dean Byrne (Quazim0t0)** · 2026 |
|
|
