--- license: mit tags: - distributed-training - old-hardware - int8 - webgpu - webrtc --- # 🌼 DaisyChain-Train β€” Old Hardware Training Pipeline **Part of DaisyChain on πŸ€— Hugging Face β†’ https://huggingface.co/DaisyChainAI** Model page (weights + card): https://huggingface.co/DaisyChainAI/DaisyChain-Train --- > **In plain terms:** DaisyChain-Train lets you use **old / spare machines** to > train neural networks. The training runs through **emulated GPU logic** β€” > verified INT8 units (GUDA-style) that stand in for a GPU's math β€” so machines > *without* a modern GPU can still do the work. Chain several together and they > train one shared model as a cluster. > Before you rely on it, see what it **can't** do β†’ [Limitations](docs/LIMITS.md). **Use the hardware you already have to train.** Each machine runs the emulated GPU logic (verified INT8 units β€” multiply / requantize / ReLU) to compute the model, and DaisyChain pools the machines data-parallel: device selection, capacity-weighted sharding, gradient sync, a P2P setup, and a live dashboard. Two ways to run β€” **Docker** or **Python**. --- ## ⚠️ Read this first DaisyChain-Train is for **small models on spare hardware**. It **pools compute, not memory** (the model must fit on one machine), scaling is **sublinear**, and it is **not** a substitute for a real GPU on large models. Full envelope in **[docs/LIMITS.md](docs/LIMITS.md)** β€” please read it before relying on it. --- ## Feature list ### Python cluster trainer (`daisychain/`) - **Data-parallel training across mixed machines** β€” each node trains its own shard; gradients combine into the exact full-batch gradient, replicas stay bit-identical. - **Capacity-weighted sharding** β€” faster machines automatically take a bigger share of the batch. - **Emulated GPU compute (verified INT8 units)** β€” `VerifiedLinear` layers run every forward multiply / requantize / ReLU through the bundled trained units; cluster-wide unit-invocation counts printed by rank 0. - **Bring your own model** β€” any `Task` (`build_model` / `sample` / `loss`) via `DAISY_TASK`; template in `examples/my_task_template.py`. - **Plain-float alternative task** β€” same cluster and pooling with ordinary float math. - **Live dashboard** (`daisychain-dashboard`) β€” readiness banner, P2P connectivity scan, pooled cores/RAM, per-node capacity plan, live loss. - **SpikeWhale control panel** (`spikewhale_panel`, `localhost:8899`) β€” sliders for model size / training settings, any HF dataset you can access (default streamed FineWeb-Edu), start/stop/re-adjust, live loss. - **Docker demo cluster** β€” 3 nodes + dashboard in one command. - **Windows helper** (`scripts\setup.bat`) and Tailscale mesh guide. ### DaisyChain-Web (`web/`) β€” browser P2P training - **Zero-install nodes** β€” opening the page IS joining; devices on one network auto-group (Snapdrop-style, by public IP). - **Private cross-network rooms** β€” `?room=CODE` with **host approval** for every join. - **Full WebRTC mesh** β€” gradients travel peer-to-peer; the server only signals and serves static files, it never sees weights or gradients. - **Leader-follower runs** β€” whoever presses Start sets width / sequence / batch-per-device / steps / learning rate for the whole group; config broadcast on the wire. - **Mid-run join** β€” late devices are synced in (weights + step) and contribute from the next step. - **Bit-identical replicas** β€” same seeded init, strict roster-order gradient averaging, deterministic Adam with identical state on every peer; verified live by per-step weight hashes. - **Sync guard** β€” any weight-hash mismatch stops the run instead of training past a fork; the step roster forbids silent partial averages. - **Gradient repair** β€” a follower missing a roster gradient re-requests it from the leader (8 steps retained), bit-exact, and the run continues. - **Cross-device kernel probe** β€” every step, every device re-hashes a fixed seeded int8 GEMM through its live kernel; catches broken arithmetic that weight hashes cannot see. - **Hardcoded FineWeb-Edu streaming** β€” the server reads random slices of the 10BT parquet shards straight off the HF CDN via HTTP range requests (pure-JS `hyparquet`); built-in corpus fallback offline. - **Checkpoints** β€” download `.pt`, upload β†’ **broadcast to the whole group**; validated (magic, dims, tokenizer vocab) before accepting. - **Inference kit** β€” one self-contained HTML file with the trained weights baked in; generations offline, anywhere. - **In-page generation** β€” prompt box on the trained model. - **Old-hardware tier** β€” no WebGPU? The identical units run on CPU (same bits, so CPU and GPU devices co-train in one group). There is no plain-float path. - **Large-message fragmentation** β€” multi-MB gradients/checkpoints chunked at 48 KB over the data channels. ### Verified compute & kernels (web) - **Verified INT8 units everywhere** β€” block-scaled int8 GEMM: exact LUT products, exact int32 accumulation, bit-exact f32 epilogue with a pinned rounding schedule; scales derived in JS f64 (division never runs on GPU). - **Backends, best-first** β€” DP4A hardware int8 dot β†’ LUT compute shader β†’ CPU mirror; every kernel **exact-gated at init** (bit-level compares) and demoted to the mirror on any mismatch. - **Continuous random-cell audit** at live training shapes. - **Fused attention kernels** β€” gather/scatter head-strided qΒ·kα΅€ and aΒ·v straight from BTΓ—C layout (CUTLASS ex. 36/52 style). - **QKV dual-GEMM fusion** β€” shared left operand quantized once, one batch-3 dispatch (ex. 45); bit-identical. - **B2B MLP chain** β€” both MLP GEMMs back-to-back on GPU with fused per-row absmax reduction and on-device quantize (ex. 13 + 23); WGSL-exact respec with a fround-stepped JS mirror; fma-contraction-immune by construction. - **Dispatch-optimized backward** β€” overlapped independent GEMMs, batch-3 sibling fusions (ex. 05/24); bit-identical gradients; optional int8 STE backward path (dormant, 1.21Γ— vs float). ### Verification stack (web) - **Exact init gates** on every kernel, every device, every boot β€” including gates that "gate the gate" with discriminating boundary inputs. - **IEEE-754 binary32 oracle** in exact BigInt arithmetic β€” proves the JS epilogue mirror is spec-correct (rejects the old mirror on 34% of inputs). - **Metamorphic property suite** β€” reference-free relations + definitional absolutes; **4/4** on an externally-authored bug corpus, matching the differential gate. - **RDNA2 ISA audit hardenings** β€” bit-level (βˆ’0-aware) gate comparisons; proof that FMA contraction cannot change the quantize. - **Eleven-suite test chain** (`cd web && npm test`) β€” convergence, replicas, oracle, gates, properties, external corpus, **self-corpus** (the instruments scored against my own bugs), B2B, optimizer, transformer LM, int8 backward; results in [web/TEST_RESULTS.md](web/TEST_RESULTS.md). - **Dirty-buffer gate** β€” the pool is poisoned before a re-sweep so state bugs (a kernel assuming zeroed memory) are caught deterministically rather than by ordering luck. ### Documentation - Python: [QUICKSTART](docs/QUICKSTART.md), [LIMITS](docs/LIMITS.md), [CUSTOM_TASK](docs/CUSTOM_TASK.md), [TAILSCALE](docs/TAILSCALE.md). - Web: [Getting started](web/docs/GETTING_STARTED.md), [Architecture](web/docs/ARCHITECTURE.md), [Verification](web/docs/VERIFICATION.md), [Troubleshooting](web/docs/TROUBLESHOOTING.md). --- ## Quick start ### Docker (most reliable β€” one command) ```bash docker compose -f docker/docker-compose.yml up --build # open http://localhost:8080 ``` Brings up a 3-node demo cluster + dashboard on one machine. ### Python (real machines) On every machine (`pip install -e .`): ```bash export MASTER_ADDR=100.101.102.10 # coordinator IP (Tailscale 100.x recommended) export MASTER_PORT=29560 export WORLD_SIZE=3 export RANK=0 # 1, 2, ... on the others export GLOO_SOCKET_IFNAME=tailscale0 # your mesh / LAN NIC daisychain-train ``` ### Windows helper ```bat scripts\setup.bat ``` An interactive menu: Docker, Python node, or just install deps. Full walkthrough: **[docs/QUICKSTART.md](docs/QUICKSTART.md)**. ### πŸ‹ SpikeWhale control panel (sliders β†’ real training) ```bash python -m daisychain.spikewhale_panel # open http://localhost:8899 ``` A web control panel: pick model size / training settings with sliders, choose any HuggingFace dataset you have access to (default: streamed FineWeb-Edu), hit Start, and watch the live loss. Stop and re-adjust any time with **← Back to settings**. Launches the real DaisyChain training underneath. ### 🌐 DaisyChain-Web (train by opening a browser tab) ```bash cd web && npm install && node server.js # open http://localhost:8787 on every device ``` Zero-install browser training: devices on the same network auto-group (Snapdrop-style) and train a shared model **peer-to-peer over WebRTC**, computing through the same verified INT8 units (WebGPU, with the identical units on CPU for machines without it β€” there is no plain-float path). Private cross-network rooms via `?room=CODE` with **host approval** β€” the room creator accepts each device before it can join. Includes gradient averaging with a deterministic Adam optimizer (identical state on every peer, nothing extra over the wire), checkpoint **download** (.pt) and **upload β†’ broadcast** so one device can restore the whole group after a failure. **Live demo:** https://huggingface.co/spaces/Quazim0t0/DaisyChain-Web ### Recent updates (July 2026) β€” DaisyChain-Web **Verification stack** β€” the browser trainer's correctness is now *checked by things that run*, not argued ([full results](web/TEST_RESULTS.md)): - **IEEE-754 oracle** (`web/test_ieee.js`): a binary32 oracle built from the standard in exact BigInt arithmetic proves the JS epilogue mirror is spec-correct β€” and rejects the old round-once mirror on 34% of inputs. - **Metamorphic properties + oracle mutation scoring** (`test_metamorphic.js`, `test_corpus.js`): properties needing no reference implementation, scored against an externally-authored bug taxonomy β€” **4/4**, matching the exact differential gate's 4/4. Relations own the loop bugs; two definitional absolutes (ReLU output range, a unit-scale integer anchor) own the value bugs no relation can see. - **Exact kernel gates on every live kernel**, a continuous audit at live shapes, and a cross-device kernel probe (same seeded int8 GEMM, same hash on every honest device, any backend). The audit's sampling was rebuilt against a named bug class: its old constants (6 cells, 2% of GEMMs) bounded an overhead that had never been measured (auditing *every* GEMM costs <0.01% of a step), and uniform random cells cannot see a last-row/column bug at a 16512-wide output. Sampling is now **stratified** β€” the first cells are the structural danger points, chosen deliberately. On a last-column bug, same cell budget: uniform caught **5/300** audits, stratified **300/300**, with zero false positives. - **RDNA2 ISA audit**: reading a real GPU's shader ISA against our determinism assumptions confirmed three of them on silicon (exact packed int8 dot; correctly-rounded f32 add/mul; 1-ULP reciprocal β€” division stays off the GPU) and produced two hardenings. (1) Real ISAs have non-IEEE variants that **flush βˆ’0 to +0**; JS `!==` can't see that (`-0 !== 0` is false), so all gates and audits now compare **bit patterns** β€” exactly what the replica hash sees. (2) FMA contraction of the quantize's `xΒ·inv + 0.5` (one rounding instead of two) turned out to be **floor-invisible by construction** β€” proven in `test_b2b.js` with 175k+ last-ulp anomalies at binade edges, zero surviving `floor()`. Rounding mode and denorm flushing are runtime driver state on real hardware, which is why every device re-runs the exact gates at every init. **Training data** β€” FineWeb-Edu (10BT sample) is the hardcoded dataset. The Space reads random slices of the parquet shards straight off the HF CDN with range requests (pure-JS `hyparquet`, SNAPPY) and serves plain text at `/data` β€” no dependency on the datasets-server rows API and its 503s. **Resilience** β€” the sync guard now *repairs* instead of halting: a roster gradient that reached the leader but not some follower (asymmetric WebRTC mesh) is re-requested from the leader, bit-exact, and the run continues. The guard still stops anything that would fork the weights. **CUTLASS-style kernel work**, each step proven bit-identical or exact-gated: - **Dispatch-optimized backward** (ex. 05/24): independent GEMMs overlapped, sibling trios fused into batch-3 dispatches β€” bit-identical gradients, dormant int8-backward path down from 1.63Γ— to 1.21Γ— vs float. - **QKV dual-GEMM fusion** (ex. 45): q/k/v share one left operand β€” quantized once, one batched dispatch, zero changed bits. - **B2B MLP chain** (ex. 13 + 23): both MLP GEMMs back-to-back on the GPU with a fused per-row absmax reduction; the intermediate is quantized on-device via a WGSL-exact respec (`floor(f32(xΒ·invScale)+0.5)` β€” no GPU division) whose fround-stepped JS mirror keeps mixed GPU/CPU fleets bit-identical. **Profile-driven speed work** β€” every change below is bit-identical (gradient and loss hashes unchanged), so none of it trades correctness for wall clock: - **Buffer pooling**: GPU buffers are recycled by size bucket instead of being created and destroyed per dispatch (~19 per MLP call, per layer, per step). **6–10% faster**, every hash unchanged. - **Shared-operand embedding GEMMs**: profiling put two f32 backward GEMMs at **55% of the entire step**, and both consumed the same `dlogits` operand β€” ~17 MB at the 16512-token vocab, uploaded *twice*. One upload, one encoder, one submit: that pair went 205 β†’ 90 ms and the step **12% faster**. The fusion is gated bit-for-bit against the two calls it replaces. - **A negative result, kept on purpose**: the remaining hot kernel looked cache-hostile (adjacent lanes wrote 66 KB apart), but making the writes contiguous changed nothing. Two probes explain why β€” holding the output at 17 MB while cutting compute 32Γ— barely moved the time. The logits GEMM is **transfer-bound, not compute-bound**, and the readback cannot be removed because softmax must stay in JS (WGSL's `exp` is not correctly rounded, and a per-vendor `exp` would fork replicas). The real lever there is the vocabulary, not the kernel. - **An init backend race was tried and removed**: it tied on the shipped path, cost ~430 ms of init, and made the backend vary between page loads, which silently invalidated three A/B comparisons before it was caught. A knob that changes what you are measuring is worse than a fixed choice. **Dirty-buffer gate β€” and the assumption it falsified.** Pooling introduced a bug class the gates predate: a pooled buffer is *not* zero-initialized, so a kernel that assumes zeros is right on step one and wrong on step two. That is a **state** bug, where no single call is wrong and the *sequence* is, which is the family no oracle can reach. The assumption was that the gates were blind to it. Mutation-testing the gate proved otherwise: deleting the zeroing made the plain gate fail at its *second* shape, because the sweep's own shapes recycle each other's buffers. The suite had **incidental** coverage nobody designed, which is coverage nobody can rely on β€” shorten the shape list and it evaporates with the gate still green. It is now deliberate: the pool is poisoned with 1e4-magnitude residue before a re-sweep, so detection no longer depends on ordering luck. ~90 ms one-time. **Scoring the oracles against my OWN bugs** (`web/test_selfcorpus.js`) β€” the external corpus measures kernel bugs someone else wrote down, so this suite asks the harder question: what do the instruments score against the four real bugs of the month? Properties **0/2** on the data-plane pair (the `cΒ·out` theorem again), differential **2/2** β€” but half the bugs were not in the kernels at all. A dead gate is a bug in a *checker*, caught only by mutating the gate; a stalled roster gradient is a bug in the *protocol*, where every computed value on every peer was correct, so no data oracle could fire. Those needed different instruments, not better oracles. All eleven test suites (`cd web && npm test`) pass; results with methodology in [`web/TEST_RESULTS.md`](web/TEST_RESULTS.md). --- ## How it works Each machine runs the **same** command; they form a cluster and train one shared model. Two things happen: 1. **The compute runs through the emulated GPU logic.** By default the model is built from `VerifiedLinear` layers, so every forward multiply / requantize / ReLU is done by the **bundled verified INT8 units** (`daisychain/verified/`) β€” the emulated GPU math. Rank 0 prints **cluster-wide unit-invocation counts** so you can see the emulated logic doing the work. 2. **The machines are pooled data-parallel.** Each node trains on its own shard; gradients are capacity-weighted and combined into the exact full-batch gradient, so replicas stay **bit-identical**. Faster machines automatically take a bigger share. ``` old machine A ─┐ old machine B ─┼─► each runs the emulated GPU logic on its shard ─► one model old machine C β”€β”˜ (gradients combined across the cluster) ``` ## Bring your own model DaisyChain-Train trains any **Task** (`build_model` / `sample` / `loss`). Copy `examples/my_task_template.py`, set `DAISY_TASK=your_module:YourTask`. Use `VerifiedLinear` (see `daisychain/verified_task.py`) to run your model's compute through the emulated units. See **[docs/CUSTOM_TASK.md](docs/CUSTOM_TASK.md)**. ## Plain-float alternative To skip the emulated units and train with normal float math on each machine, set `DAISY_TASK=daisychain.example_task:ExampleTask`. Same cluster, same pooling β€” the model math just runs as ordinary float instead of through the verified units. ## The dashboard `daisychain-dashboard` (or the Docker service) serves a Tailwind page at `:8080` β€” readiness banner, P2P connectivity scan, pooled cores/RAM + capacity plan (per-node device, weight, batch), and live training loss. ## Networking Use **Tailscale** for a P2P mesh so machines on different networks get stable IPs on one interface β€” **[docs/TAILSCALE.md](docs/TAILSCALE.md)**. --- ## Layout ``` daisychain/cluster.py capacity-weighted CPU/GPU data-parallel trainer daisychain/train.py entry point (daisychain-train) daisychain/verified/ bundled trained N/N units + VerifiedLinear (train through them) daisychain/verified_task.py default task: forward runs on the verified units daisychain/example_task.py plain-float alternative task daisychain/task.py the Task interface + loader daisychain/dashboard/ agent + P2P scanner + Tailwind server docker/ Dockerfile, dashboard image, compose (demo cluster) scripts/setup.bat / setup.sh interactive setup helpers config/ nodes + cluster env examples examples/my_task_template.py starting point for your own model docs/ QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE daisychain/spikewhale_task.py trains the real SpikeWhale on streamed HF datasets daisychain/spikewhale_panel.py slider control panel (localhost:8899) web/ DaisyChain-Web: P2P browser training (WebRTC + WebGPU) export_luts_web.py regenerates web/public LUTs from the trained units ``` ## Install ```bash pip install torch numpy psutil pip install -e . # exposes: daisychain-train, daisychain-agent, daisychain-dashboard ``` Requires Python β‰₯ 3.9, PyTorch β‰₯ 2.0. Multi-node is reliable on **Linux/macOS**; on **Windows use Docker/WSL** (see [Limitations](docs/LIMITS.md)). --- ## Links - **DaisyChain on Hugging Face:** https://huggingface.co/DaisyChainAI - **This model:** https://huggingface.co/DaisyChainAI/DaisyChain-Train **License:** MIT Β· **Author:** Dean Byrne (Quazim0t0) Β· **Org:** DaisyChainAI ## Citation ```bibtex @misc{byrne2026daisychain, title = {DaisyChain-Train: An Old Hardware Training Pipeline}, author = {Byrne, Dean (Quazim0t0)}, year = {2026}, howpublished = {\url{https://huggingface.co/DaisyChainAI/DaisyChain-Train}}, note = {Chain spare/old machines into a data-parallel training cluster} } ``` **Dean Byrne (Quazim0t0)** Β· 2026