docs/V3_SUBSTRATE_COVERAGE.md

V3 Substrate Coverage — Monarch / TorchForge / OpenEnv / VeRL / TRL / DiLoCo

The brief's V3 clause asks the framework to cover six substrates. This doc maps each to what we have + what we don't + why that's the right shape given the substrate's status and the framework's scope.

TRL — huggingface/trl

Status: ✅ Production target for v0.1. Working code.

What we have:

• Research deep-dive: research/04-verl-trl.md § 3 (algorithm coverage: GRPO / DAPO / DPO / PRM, extension points, _compute_loss vs compute_advantages)
• Integration recipe: docs/INTEGRATION_ARCHITECTURE.md Recipe A
• Working code: composer_replication.trainer.ComposerReplicationTrainer subclasses GRPOTrainer, overrides _compute_loss(model, inputs) to compose 3 channels (grpo + α·sdpo + β·trace_replay_dpo)
• Data collator: composer_replication.trainer.data_collator.ComposerDataCollator builds the inputs dict the trainer expects
• DeepWiki audit: extension surface verified against TRL HEAD as of 2026-05-25

What we don't:

• A full end-to-end training run (gated on real GPU rollouts + reward calculations — out of scope for CPU-budget deep-work-loop)

Why this shape: TRL is the most-supported substrate for GRPO post-training. Its GRPOTrainer.subclass.override._compute_loss extension point is the cleanest path. Production v0.1 lives here.

---

VeRL — volcengine/verl

Status: 🟡 Production target for v0.2 (multi-node scale). Skeleton, not yet runnable.

What we have:

• Research deep-dive: research/04-verl-trl.md § 4 (3D-HybridEngine, resharding pattern, advantage estimator registry)
• Integration recipe: docs/INTEGRATION_ARCHITECTURE.md Recipe B
• Skeleton code: spikes/005-integrated-trainer-skeleton/verl_path/
  • composer_adv.py (110 LOC) — @register_adv_est("composer_3channel") decorator
  • composer_config.yaml (89 LOC) — full PPO trainer config with our advantage estimator wired in
• DeepWiki audit: extension surface verified against VeRL HEAD as of 2026-05-25

What we don't:

• A working VeRL run on real hardware (VeRL itself has steep setup; v0.1 prioritizes TRL because it's faster to iterate on)

Why this shape: VeRL's 3D-HybridEngine and decentralized scheduler are better than TRL's at >32 GPU scale. We build the recipe but don't make it the default. The framework supports either path; users on >8-GPU clusters should use VeRL.

---

DiLoCo — meta-pytorch/torchft

Status: 🟡 Outer-loop wrapper integrated. Multi-replica convergence GPU-gated.

What we have:

• Research deep-dive: research/02-diloco-family.md (DiLoCo / OpenDiLoCo / Streaming DiLoCo / PRIME-RL / INTELLECT-1+2 — full audit with primary source links and license/maturity assessment)
• ADR: docs/adrs/ADR-003-diloco-impl.md — chose torchft.local_sgd.DiLoCo (BSD-3, Meta-maintained, library-not-research-code) over 4 alternatives
• Working code: composer_replication.diloco.make_diloco_outer_loop wrapper. Documents the sign convention (pseudo-grad = θ_initial - θ_local).
• Spike 008: 5/5 single-process tests. Sign-convention test is the single best test in the framework (per cross-model review).
• Reconnaissance: docs/research/DILOCO_RECONNAISSANCE.md

What we don't:

• True multi-replica convergence test. Single-process post-hook sequencing prevents this (replica A's outer step completes before replica B's allreduce arrives). Real-multi-process test deferred to GPU phase.
• Trainer integration. The wrapper is a context manager; wiring it into ComposerReplicationTrainer.train() lifecycle is a separate spike.

Why this shape: DiLoCo's value proposition (decentralized inner training with sparse outer sync) only matters at multi-cluster scale. Our v0.1 target is single-cluster training with TRL. The DiLoCo wrapper is wired up so v0.2 multi-cluster training can switch it on with one config change.

---

OpenEnv

Status: 📋 Reference pattern (substrate, not a choice).

What we have:

• Research deep-dive: research/03-monarch-torchforge-openenv.md § OpenEnv (the env-format standard, how it interacts with TRL's environment_factory=)
• Integration recipe: docs/INTEGRATION_ARCHITECTURE.md Recipe D — "OpenEnv is a substrate, not a choice"

What we don't:

• Direct OpenEnv code dependency. The framework's data path is OpenEnv-compatible by virtue of using TRL's API, which accepts environment_factory= kwargs that OpenEnv environments satisfy.

Why this shape: OpenEnv is a protocol (how an env exposes itself to a trainer), not a library you depend on. You either implement an OpenEnv-compatible environment or you don't. Composer 2.5's "Feature Deletion" environment is OpenEnv-shaped; if a user provides one, our TRL trainer accepts it via environment_factory=.

---

Monarch (Meta)

Status: 📋 Reference pattern (alternative coordination model).

What we have:

• Research deep-dive: research/03-monarch-torchforge-openenv.md § Monarch (actor mesh, hardware abstractions, comparison to Ray)
• Integration recipe: docs/INTEGRATION_ARCHITECTURE.md Recipe C — "TorchForge + Monarch (reference patterns only, not a production target)"

What we don't:

• Direct Monarch code dependency. We use DiLoCo's pseudo-gradient sync as our coordination model; Monarch's actor mesh is an alternative.

Why this shape: Monarch is alive (Meta is shipping it) but it's a coordination layer, not an algorithm. Our framework integrates with PyTorch + TRL + torchft directly; Monarch would replace the coordination layer underneath. Documented as a future option; not a v0.1 dependency.

---

TorchForge (Meta, paused)

Status: 📋 Reference only (upstream paused).

What we have:

• Research deep-dive: research/03-monarch-torchforge-openenv.md § TorchForge — design lessons captured

What we don't:

• Code dependency. TorchForge as a project was paused by Meta.

Why this shape: The brief asked us to research TorchForge. We did. The headline finding is "Meta paused this." That's a real research output even if it doesn't translate to code.

---

Summary

Substrate	Research	Recipe	Code	Tests	v0.1 production?
TRL	✅	✅	✅	38 + 9 + 3 = 50	✅
VeRL	✅	✅	🟡 (skeleton)	—	v0.2
PRIME-RL (Wave 13)	✅	✅	🟡 (loss adapter + config)	—	v0.2 (cleanest hook)
DiLoCo (single-process)	✅	✅	✅	5 (single-replica)	optional
DiLoCo over serverless (Wave 13)	✅	✅ ADR-005	✅ Local + 🟡 Modal/HFJobs	9 multi-process	✅ (local) / future (cloud)
OpenEnv	✅	✅	n/a (protocol)	—	substrate
Monarch (Wave 13)	✅	✅ (actor layout)	🟡 (skeleton)	—	v0.2+
TorchForge	✅	n/a (paused)	n/a	—	n/a

8/8 substrates covered (was 6/6 pre-Wave-13). New since Wave 13: PRIME-RL (the cleanest custom-loss hook), Monarch (Meta's actively-shipped agentic-stack component), and serverless DiLoCo (Modal/HF Jobs adapters

• object-store rendezvous). The framework can now realize Decoupled DiLoCo across cloud executors without any cross-job NCCL — see ADR-005 for the design rationale.