docs/adrs/ADR-006-rl-frameworks.md

ADR-006 — RL framework strategy: TRL + VeRL + PRIME-RL

Status: Accepted Date: 2026-05-26 Wave: 13

Context

The brief's V3 clause names six substrates: monarch, torchforge, openenv, VeRL, TRL (plus DiLoCo). Cross-model review (Wave 11) flagged that V3 was thin on the RL-framework side: TRL has working code, VeRL has a config skeleton, and Monarch/TorchForge/OpenEnv are research-only.

User's 2026-05-26 expansion: "see if there are other frameworks that are more popular that we could try to use. meta's pytorch agentic stack components are something that I'd like to explore."

docs/research/RL_FRAMEWORKS_LANDSCAPE.md audited:

• 6 RL frameworks: OpenRLHF, PRIME-RL, NeMo-Aligner, Unsloth, LLaMA-Factory, DeepSpeed-Chat
• 4 Meta PyTorch stack components: Monarch, TorchTitan, TorchForge, torchchat

Options considered

Framework	License	GRPO/DAPO?	Custom-loss extension	Verdict
OpenRLHF	Apache-2	✅ DAPO	Fork openrlhf/models/loss.py + Trainer subclass (~400-600 LOC)	Strong but heavyweight
PRIME-RL	Apache-2	✅ GRPO + DAPO	First-class CustomLossConfig with LossInputs struct (~200-300 LOC)	Chosen
NeMo-Aligner	Apache-2	❌ no GRPO/DAPO	n/a	Reject
Unsloth	Apache-2	TRL patcher	Closed unsloth_zoo loss kernels — unhookable	Reject
LLaMA-Factory	Apache-2	❌ delegates to EasyR1	n/a	Reject
DeepSpeed-Chat	Apache-2	❌ PPO+DPO only	feature-stale since 2023	Reject

Meta stack	License	Active?	Role
Monarch	BSD-3	✅ v0.4.1 stable, v0.5 dev	Actor mesh — coordination layer for any SPMD trainer
TorchTitan	BSD-3	✅ active	Distributed-training stack (already a transitive dep of PRIME-RL)
TorchForge	BSD-3	❌ paused	Patterns only, per repo banner
torchchat	BSD-3	active	Inference only — out of scope

Decision

Add PRIME-RL as the third RL framework after TRL+VeRL, and Monarch as the agentic-stack coordination layer.

Why PRIME-RL

PRIME-RL ships a first-class CustomLossConfig with an import_path that lets us drop in a Python function returning a tensor. The config exposes a LossInputs struct with exactly the tensors we need: trainer_logprobs, inference_logprobs, teacher_logprobs, advantages, loss_mask. This is the cleanest possible extension point for a 3-channel loss — no fork, no Trainer subclass, no monkey- patching.

It also uses the verifiers env protocol (OpenEnv-compatible by design), so it slots into the framework's existing data path without translation.

PRIME-RL was used to train INTELLECT-1 (10B base, 30 nodes) and INTELLECT-2 (32B QwQ); production-tested on real distributed runs.

Why Monarch (not TorchForge or TorchTitan as a top-level)

• Monarch is what's actually shipping from Meta's agentic stack. v0.4.1 is stable, v0.5 dev daily. BSD-3.
• TorchForge is paused per its own repo banner. We document it (research/03) but don't depend on it.
• TorchTitan is a transitive dep of PRIME-RL already, so we get its benefits without needing to build a direct integration. If we wanted a TorchTitan-only path, it would be redundant with PRIME-RL.
• torchchat is inference-only and doesn't fit the training-framework conversation.

Monarch's role in our stack: the actor mesh that hosts trainer/generator/ rewarder/judge actors. PRIME-RL's three-actor split (trainer, generator, rewarder) maps naturally onto Monarch primitives.

Consequences

Accepted

• composer_replication/recipes/prime_rl/ directory:
  • prime_rl_recipe.md — integration recipe (parallel to TRL Recipe A, VeRL Recipe B)
  • composer_loss.py — the 3-channel loss adapted to PRIME-RL's LossInputs struct (~200-300 LOC)
  • prime_rl_config.yaml — example PRIME-RL config wiring our loss in
• composer_replication/recipes/monarch/ directory:
  • monarch_actor_layout.md — design doc for the actor mesh
  • actors.py — placeholder Monarch actor definitions (skeleton only; full integration is post-replication)
• New optional dependencies in pyproject.toml:
  • [prime-rl] extra: prime-rl>=0.5
  • [monarch] extra: monarch>=0.4.1
• docs/V3_SUBSTRATE_COVERAGE.md updated to reflect the new additions.

Three-recipe production matrix

User scenario	Recommended recipe
Quick start, single-cluster, ≤7B	TRL Recipe A
Production multi-node, ≤32B	VeRL Recipe B
Decentralized / DiLoCo-shape, any size	PRIME-RL recipe (NEW)
Coordination-heavy multi-actor RL	Monarch + any of the above

Trade-offs explicitly accepted

• Three RL frameworks is a maintenance burden. We accept this because no single one covers all the user scenarios above. The framework's contribution is the 3-channel loss + the trace-replay channel, expressed in three different framework idioms. Each recipe is ~200-300 LOC; total triplication tax ~700 LOC vs. picking one framework.
• Monarch is BSD-3 not MIT. The framework is MIT; users opting in to Monarch take on its license. Documented in pyproject.toml's optional extras.
• PRIME-RL's API may evolve. The LossInputs struct is currently the contract; if PRIME-RL stabilizes a different shape we'd need to bump. Pin to v0.5.x in our optional extras.

Source

docs/research/RL_FRAMEWORKS_LANDSCAPE.md (2026-05-26 subagent recon, primary-sourced from DeepWiki audits + GitHub repo READMEs + PyPI release metadata).