Troubleshooting

Common failure modes, what the log messages mean, and what to do.

Connection problems

Devices don't see each other.

Same network? Auto-grouping is by public IP — devices on different networks (e.g. phone on cellular) will never auto-group. Use ?room=CODE on every device instead.
Room joins need host approval — the first device in the room must click Accept for each newcomer.
Corporate/CGNAT networks can block WebRTC (only STUN is configured, no TURN relay). Try a room over the hosted Space, or another network.

"WebGPU not available" / backend shows CPU.

WebGPU needs a secure context: localhost or HTTPS. Plain http://192.168.x.x never gets WebGPU — tunnel it or use the Space.
Old hardware/drivers without a modern GPU backend fall back to CPU. That's supported — CPU devices compute the identical bits, just slower.

A kernel warning at startup (e.g. "failed verification — using CPU LUT mirrors"). The device's GPU produced values that don't match the verified units bit-for-bit, so that kernel was refused and a slower-but-exact path is used. This is the system working as designed; the run stays correct.

Training problems

"SYNC GUARD: stopped — weight hash mismatch." Replicas forked — the run is stopped rather than trained past a divergence. The most common cause is mixed builds: devices running different versions of the page whose math specs differ (the quantize respec is a real, bounded spec change — old and new builds must not co-train). Make sure every device has the same version (hard-refresh: Ctrl+Shift+R), then restart the run or restore from a checkpoint.

"missing a roster gradient … requesting repair from the leader." A gradient reached some peers but not others (asymmetric mesh). The follower re-requests it from the leader, who retains the last 8 steps — normally the run continues by itself. If repair fails repeatedly, the sync guard stops the run; check whether a device went offline.

"probe hash mismatch." A device's kernel is producing wrong values at runtime (the seeded probe GEMM no longer matches the fleet). That device should rejoin — its gates will re-run at init and demote it to CPU if its GPU misbehaves.

Loss looks stuck / falls very slowly. Small widths and short runs are modest models — try more steps, a wider model, or more devices (each adds its batch to the effective batch). Remember the envelope: this pools compute, not memory; it is a proof of concept for small models.

A slow device paces everyone. Steps are synchronous: the group advances when every roster gradient arrives. Removing (or upgrading) the slowest device speeds up everyone.

Data problems

"dataset unreachable — using built-in corpus." The /data endpoint couldn't read the FineWeb-Edu shards (offline, or the CDN hiccuped — the server reads parquet slices straight off the HF CDN with range requests). Training continues on the built-in fallback text; generation quality will reflect the smaller corpus.

Checkpoint problems

"tokenizer mismatch." The checkpoint was trained with a different vocab than the tokenizer this page loaded — it cannot be resumed here.

"bad dims in checkpoint." Width/sequence outside the supported 16–128 range — likely a corrupted or foreign file.

Loading a valid checkpoint on one device broadcasts it to the whole group; you don't need to load it everywhere.

Verifying your install

npm test

All ten suites should pass on a clean clone (Node only, no browser/GPU required). If a suite fails on unmodified code, please report it — the tests are deterministic apart from bounded random sweeps that are designed not to flake. See TEST_RESULTS.md for what each suite proves.