docs/adrs/ADR-005-serverless-diloco.md

ADR-005 — Decoupled DiLoCo over serverless training systems

Status: Accepted Date: 2026-05-26 Wave: 13

Context

The brief's V2 clause says:

take that and combine it with diloco (decoupled, open, any variant of diloco)

The user expanded 2026-05-26: "Decoupled DiLoCo (so that we can leverage modal or huggingface-jobs or other serverless training systems). we need this both on the dataset generation and the RL orchestration side of things."

Spike 008 wrote composer_replication.diloco.make_diloco_outer_loop (wraps torchft.local_sgd.DiLoCo) but that's a single-process API. To realize "Decoupled DiLoCo across serverless executors" we need:

1. An abstraction layer that lets the framework launch N replicas on different serverless backends (Modal, HF Jobs, SageMaker, etc.) without per-backend code in the trainer.
2. A communication primitive that doesn't require inter-job NCCL/RDMA (most serverless executors don't expose that, and DiLoCo doesn't need it — sync happens once per ~500-1000 inner steps).

Options considered

docs/research/DILOCO_SERVERLESS_RECONNAISSANCE.md audited 6 executors:

Executor	Inter-job network	Cold start	$/A100·hr	$/H100·hr
Modal	yes (cluster mode)	~30s	$1.95	$5.50
HuggingFace Jobs	no	~60s	$4.18	$9.50
AWS SageMaker training	yes (warm pools)	~3-5min	~$3.06	~$8.50
GCP Vertex AI	yes (cluster)	~5-10min	~$3.67	~$10
Azure ML	yes (cluster)	~5-10min	~$3.67	~$10
k8s + Volcano/KubeRay	yes (cluster IP)	~30-90s	(BYO)	(BYO)

Most expose a "spin up a job, run a script" interface. Few expose inter-job networking; the ones that do require explicit cluster mode (extra cost + config).

Decision

Adopt object-store rendezvous as the default DiLoCo communication primitive across all serverless executors. Specifically:

• composer_replication.diloco.serverless package
• class ServerlessExecutor(Protocol) — uniform interface with launch_replicas / poll / stream_logs / cancel / collect / backend_name / supports_inter_replica_network
• class ObjectStoreAllReduce — fsspec-backed pseudo-gradient exchange using s3:// / gs:// / az:// / hf:// / file:// — single code path, swappable bucket
• v0 concrete adapters: ModalExecutor and HFJobsExecutor
• v0.1+ adapters: RunPodExecutor, SageMakerExecutor, K8sExecutor

Why object-store rendezvous (not NCCL across jobs)

DiLoCo paper (arXiv:2311.08105) shows the outer-loop sync is once per H = 500-1000 inner steps, equivalent to ~10-30 minutes of wall-clock at typical post-training step rates. For a 1B-param model in bf16:

• Pseudo-gradient size: ~2 GB per replica per outer round
• Sync frequency: ~once per 30 minutes
• Therefore: ~2 GB × N_replicas, every ~30 min, durably written to object storage with a single PutObject per replica + GetObject per other replica

Even with N=8 replicas, that's 16 GB write + 14 GB × 8 reads = 128 GB read spread over 30 minutes = ~70 MB/s aggregate. S3 free-tier handles this without breaking a sweat, and S3 cross-job reads cost ~$0.0001 per GET. Total inter-replica communication cost: ~$0.05 per outer round. Negligible compared to GPU spend.

By contrast, cross-job NCCL would require:

• Inter-job networking (mostly unavailable on serverless)
• Sustained low-latency connections (vs. burst-IO once per 30min)
• Backend-specific cluster mode (Modal-only on some platforms)

Object-store rendezvous decouples the algorithm from the executor and matches DiLoCo's actual communication profile.

Why Modal + HF Jobs as the v0 executors

• Modal: best dev velocity, sub-minute cold start, mature Python SDK, user already has CLI configured. Gives us a fast iteration loop for the serverless layer.
• HuggingFace Jobs: zero acquisition cost (HF token already wired up), brand-aligned with the framework's HF-native posture, ~$4.18/A100·hr. Not the cheapest, but the right "default executor for HF users."

These two cover the spectrum of "fast for development" + "natural HF integration." Other executors are documented and stubbed but not implemented in v0.

Consequences

Accepted

• New package composer_replication.diloco.serverless:
  • executor.py — ServerlessExecutor Protocol + base class
  • allreduce.py — ObjectStoreAllReduce mockManager that drops into make_diloco_outer_loop with no changes to the existing wrapper
  • modal.py — ModalExecutor (~150 LOC)
  • hf_jobs.py — HFJobsExecutor (~150 LOC)
  • replica_entrypoint.py — the script each replica runs (loaded from HF Datasets / object store)
• New optional dependency [serverless] extra: pip install -e .[serverless] pulls fsspec, s3fs, huggingface_hub (already a transitive dep), and modal-client (only if user opts in to Modal).
• Smoke test in spikes/009-decoupled-diloco/ (new, deferred — not part of this wave's commit) — local-only file:// rendezvous between two Python processes in tests/test_serverless_local.py. Multi-cloud test is post-replication.

Open / deferred

• Real serverless smoke: spinning up 2 Modal containers + S3 rendezvous
  • verifying both converge. Deferred to a small-budget post-Wave-13 spike ($2-5 estimated). Not blocking for the v0 packaging.
• HF Jobs API stability: HF Jobs is a relatively new product. The recon flagged "API may evolve through 2026"; we pin to a specific huggingface_hub minor and bump deliberately.

Trade-offs explicitly accepted

• We do NOT use Modal's cluster/RDMA mode in v0. That gives sub-second cross-job NCCL but costs more and is Modal-only. Object-store rendezvous is the right default; users on Modal who want faster sync can override.
• We do NOT support job-internal multi-GPU training in this layer. The serverless layer is for inter-replica sync; intra-replica training uses the existing make_diloco_outer_loop (which itself can wrap multi-GPU FSDP via torchft).

Source

docs/research/DILOCO_SERVERLESS_RECONNAISSANCE.md (2026-05-26 subagent recon, primary-sourced from each provider's official docs + pricing pages).