Initial commit: Composer 2.5 Replication Framework — research synthesis

Methodology repo (model type) for an open replication of Cursor's Composer 2.5
(post-trained Kimi K2.5) on any HuggingFace base model.

Contents:
- README.md (HF model card with frontmatter)
- framework/composer-replication-framework.md (master synthesis, 18KB)
- research/ (5 deep-dives by 5 different LLM families, ~107KB total)
- 01-composer-2.5.md (Gemini 3.1 Pro)
- 02-diloco-family.md (DeepSeek V4 Pro)
- 03-monarch-torchforge-openenv.md (GPT-5)
- 04-verl-trl.md (Sonnet 4.6)
- 05-trace-replay-distillation.md (Kimi K2-Thinking)
- docs/METHODOLOGY.md (how the synthesis was produced)
- docs/HF_REPO_LAYOUT.md (planned multi-repo split)
- LICENSE (MIT)

Status: pre-spike. No code, no weights, no datasets yet. Trained variants and
trace datasets will live in separate repos linked via HF Collection.

.gitignore

@@ -0,0 +1,38 @@
+# .gitignore — composer-replication-framework
+
+# Local notes / drafts not for HF
+.scratch/
+*.draft.md
+
+# Editor / OS junk
+.DS_Store
+*.swp
+*~
+
+# Future code (will be added in spike v0.0)
+__pycache__/
+*.pyc
+*.pyo
+.venv/
+.env*
+!.env.example
+node_modules/
+
+# Training artifacts (belong in separate model/dataset repos, not here)
+checkpoints/
+wandb/
+*.safetensors
+*.bin
+*.pt
+*.pth
+
+# Trace / dataset shaped content (belongs in dataset repos)
+*.jsonl
+*.parquet
+*.arrow
+data/processed/
+data/external/
+
+# Logs / runtime
+logs/
+*.log

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Codeseys
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,201 @@
+---
+license: mit
+language:
+  - en
+library_name: transformers
+tags:
+  - reinforcement-learning
+  - post-training
+  - distillation
+  - agentic-coding
+  - composer-2.5
+  - cursor
+  - kimi-k2
+  - grpo
+  - dapo
+  - diloco
+  - prime-rl
+  - openenv
+  - trl
+  - verl
+  - monarch
+  - torchforge
+  - research
+  - methodology
+pretty_name: "Composer 2.5 Replication Framework — Research Synthesis"
+---
+
+# Composer 2.5 Replication Framework
+
+> **Repo type:** `model` (methodology). **Status:** Research synthesis (2026-05-25). Pre-spike — no code yet.
+> **Author:** [Codeseys](https://huggingface.co/Codeseys)
+> **Goal:** Replicate Cursor's [Composer 2.5](https://cursor.com/blog/composer-2-5) (a post-trained Kimi K2.5 specialised for agentic coding) on **any** HuggingFace base model, using a synthesis of decentralized RL post-training techniques.
+
+This repository is the **"paper of the project"** — it is the methodology / research / framework specification for an open replication of Cursor's Composer 2.5 system, plus a **novel multi-teacher trace-replay distillation channel** that stacks on top of the Composer recipe.
+
+It contains **no model weights and no training data** (yet). When the spike v0.0 produces results, trained variants will live in separate model repos and training-mix data will live in separate dataset repos, all linked via an HF Collection — see [Roadmap](#roadmap).
+
+---
+
+## TL;DR — what's in here, why it matters
+
+Cursor's Composer 2.5 is the strongest case study for "RL post-training of a frontier MoE base produces a model that beats GPT-5.5 on agentic coding while costing 5–10× less to serve." The recipe is **almost entirely post-training** (~85% of compute) and the most important trick is **non-obvious**: a per-turn on-policy distillation loss called *Targeted RL with Textual Feedback*.
+
+This repo contains:
+
+1. **`framework/composer-replication-framework.md`** — master synthesis: architecture, stack picks, phase plan, open questions. The TL;DR table maps every layer of the system to a concrete software pick with rationale.
+2. **`research/01-composer-2.5.md`** — Composer 2.5 deep-dive: base model, 5-stage recipe, the secret-sauce hint-distillation loss, results.
+3. **`research/02-diloco-family.md`** — DiLoCo / OpenDiLoCo / Streaming DiLoCo / PRIME-RL / INTELLECT-1+2 deep-dive: when decentralized training actually helps, when it's premature.
+4. **`research/03-monarch-torchforge-openenv.md`** — Meta's Monarch actor mesh + TorchForge (paused) + OpenEnv environment standard. What's alive, what to bet on.
+5. **`research/04-verl-trl.md`** — Algorithm-library deep-dive: GRPO / DAPO / DPO / PRM in TRL vs VeRL, plus the 3D-HybridEngine resharding pattern.
+6. **`research/05-trace-replay-distillation.md`** — Novelty assessment of the trace-replay multi-teacher distillation idea: prior art (rStar / Math-Shepherd / OmegaPRM / Magpie / MoA), cost analysis, reward-shape options.
+
+Each of the five research deep-dives was authored by a **different LLM family** (Gemini 3.1 Pro, DeepSeek V4 Pro, GPT-5, Sonnet 4.6, Kimi K2-Thinking) running in parallel. The synthesis at `framework/composer-replication-framework.md` cross-checks their findings.
+
+## Headline findings
+
+### 1. Composer 2.5's secret sauce is the *targeted hint-distillation loss*
+
+The 1T MoE base (Kimi K2.5) and the "Feature Deletion" RL environment are the obvious moves. The non-obvious one — and the one Cursor never explains in detail — is:
+
+> **Targeted RL with Textual Feedback (on-policy distillation):** when a 100K-token rollout has a localized error, generate a text hint correcting the error, run forward pass *with* the hint to get "Teacher" logits, run forward pass *without* the hint to get "Student" logits, and apply KL divergence loss to pull Student toward Teacher *only at that turn*. Sidesteps the credit-assignment nightmare of long-horizon scalar rewards.
+
+This is the fix for "GRPO on agentic traces is brittle because one bad step poisons 100 good ones." The biggest reproducibility gap is **how the text hints are generated** — Cursor never tells. Templates? Smaller model? Same model with introspection prompt? Open question.
+
+### 2. The trace-replay multi-teacher idea is genuinely novel
+
+Closest precedent is rStar-Math (single-teacher MCTS counterfactuals at training time). **Multi-teacher *frozen-trace replay* with disagreement-as-reward is open territory.** Cost analysis works out: with VOI gating + tiered teachers, you get **~$3/trace** instead of **~$64/trace** at the 1000-step / 8-teacher baseline.
+
+The two distillation channels stack cleanly:
+
+- **Composer hint-distill** = teacher-self pulls student at error sites (per-turn KL)
+- **Trace-replay-distill** = N external teachers pull student at all sites (per-step DPO / PRM)
+
+Both bypass long-horizon credit assignment.
+
+### 3. Recommended stack (verified across all 5 reports)
+
+| Layer | Pick | Why not the alternative |
+|---|---|---|
+| **RL substrate** | [PRIME-RL](https://github.com/PrimeIntellect-ai/prime-rl) | INTELLECT-2 already proved 32B globally distributed; Forge is "development-paused" by Meta |
+| **Algorithm impl** | [TRL](https://github.com/huggingface/trl) (lift loss math) | Cleanest GRPO + first-class OpenEnv integration |
+| **Resharding pattern** | [VeRL](https://github.com/volcengine/verl)'s 3D-HybridEngine (reference) | Most battle-tested at 70B+ |
+| **Environments** | [OpenEnv](https://github.com/meta-pytorch/openenv) + [verifiers](https://github.com/willccbb/verifiers) | HF + Meta backing, MCP RFC landing, Hub-hosted |
+| **Distributed sync** | Skip DiLoCo for v0.1 | Outer loop only matters when training spans clusters |
+| **Orchestration** | Ray today, [Monarch](https://github.com/meta-pytorch/monarch) when mature | Forge paused; Monarch K8s story still landing |
+
+## Architecture
+
+```
+                    ┌───────────────────────────────────────────┐
+                    │           OpenEnv Environment Hub         │
+                    │  (HF Hub, Docker images, MCP tool-calling)│
+                    │  - Anyrun-style code sandbox              │
+                    │  - SWE-Gym, SWE-Bench-Verified envs       │
+                    │  - "Feature Deletion" auto-grader env     │
+                    └────────────────┬──────────────────────────┘
+                                     │ rollouts (verifiers protocol)
+                                     ▼
+        ┌────────────────────────────────────────────────────────────┐
+        │                    ORCHESTRATOR (CPU)                      │
+        │  - Schedules rollouts across inference workers             │
+        │  - Assembles training batches                              │
+        │  - Routes hint-distillation pairs (Composer-style)         │
+        │  - Routes trace-replay teacher queries (NOVEL)             │
+        │  - Built on Monarch (future) or Ray (today)                │
+        └────┬──────────────────────────┬──────────────────────────┬─┘
+             │ rollout requests         │ training batches         │ teacher queries
+             ▼                          ▼                          ▼
+   ┌─────────────────────┐   ┌────────────────────┐   ┌────────────────────────┐
+   │  INFERENCE POOL     │   │  TRAINER (GPU)     │   │  TEACHER POOL          │
+   │  (vLLM / SGLang)    │   │  - FSDP2 sharded   │   │  - Frozen N teachers   │
+   │  - Student policy   │   │  - GRPO + DAPO     │   │  - HF Inference,       │
+   │  - Auto-resharded   │   │  - +Hint distill   │   │    OpenRouter, vLLM    │
+   │    via SHARDCAST    │   │    KL loss         │   │  - Diverse families    │
+   │  - Async tool waits │   │  - +PRM/DPO from   │   │    (Anthropic / OpenAI │
+   │    don't block GPU  │   │    trace-replay    │   │     / DeepSeek / Qwen) │
+   └─────────────────────┘   └────────────────────┘   └────────────────────────┘
+                                     │
+                                     │ pseudo-gradients (every H steps)
+                                     ▼
+                    ┌────────────────────────────────┐
+                    │  OUTER LOOP (DiLoCo, optional) │
+                    │  - Only when training spans    │
+                    │    multiple clusters / DCs     │
+                    │  - Streaming variant for       │
+                    │    bandwidth-limited links     │
+                    └────────────────────────────────┘
+```
+
+Three reward channels feed the trainer:
+
+1. **RLVR** — verifiable rewards (tests pass, build succeeds). Ground truth, never skipped.
+2. **Composer hint-distill** — per-turn KL to a hint-conditioned forward pass.
+3. **Trace-replay-distill** — per-step preference / process-reward signal from N frozen teachers.
+
+The novel contribution is channel (3) — no published work systematically replays each step of frozen agentic traces with multiple teachers to harvest step-level supervision.
+
+## Roadmap
+
+| Phase | Timeline | Goal | Trained variant repo | Data repo |
+|---|---|---|---|---|
+| **v0.0 spike** | 1–2 weeks | Prove trace-replay-DPO beats plain GRPO on Qwen3-7B + SWE-bench-lite | `Codeseys/composer-replication-qwen3-7b-v0` | `Codeseys/composer-replication-traces-v0` |
+| **v0.1** | 1–2 months | Full Composer recipe (RLVR + hint-distill + trace-replay) on Qwen3-32B + Feature Deletion env. Match Cursor's ~50% SWE-bench-multilingual at 32B scale. | `Codeseys/composer-replication-qwen3-32b-v1` | `Codeseys/composer-replication-traces-v1` |
+| **v0.2** | 3–6 months | Decentralized scaling: Streaming DiLoCo + SHARDCAST + Monarch. Multi-cluster reproduction of v0.1. | `Codeseys/composer-replication-qwen3-32b-decentralized` | (re-uses v1 data) |
+
+Each variant will get its own model repo (LoRA adapters or full fine-tunes) per the [HF multi-artifact research project layout](https://huggingface.co/docs/hub/repositories). This methodology repo will be linked from each variant's README and via an HF Collection once v0.0 produces a result.
+
+## Methodology — how this synthesis was produced
+
+To minimize single-model bias, the five research deep-dives were generated **in parallel** by five different LLM families via the [`delegate_task` parallel-research pattern](https://huggingface.co/docs/transformers/research):
+
+| Topic | Author model |
+|---|---|
+| `01-composer-2.5.md` | google/gemini-3.1-pro-preview |
+| `02-diloco-family.md` | deepseek/deepseek-v4-pro |
+| `03-monarch-torchforge-openenv.md` | openai/gpt-5 |
+| `04-verl-trl.md` | anthropic/claude-sonnet-4.6 |
+| `05-trace-replay-distillation.md` | moonshotai/kimi-k2-thinking |
+
+Convergent findings across reports (≥2 independent confirmations):
+
+- **GRPO+DAPO is the consensus algorithm** (3/4 reports that compared)
+- **PRIME-RL is the most production-ready decentralized substrate** (2 reports independently)
+- **OpenEnv is the env-format winner** (3 reports converge)
+- **Trace-replay-with-N-teachers is genuinely under-explored** (the trace-replay report's primary finding, corroborated by the absence of it in the 4 other reports)
+
+The synthesis at `framework/composer-replication-framework.md` reconciles divergences (e.g., DiLoCo vs single-cluster timing) with explicit rationale.
+
+## Citation
+
+If you use this framework or its derivative artifacts (the trained variants, the trace dataset, or the Feature-Deletion environment), please cite:
+
+```bibtex
+@misc{composer-replication-framework-2026,
+  author       = {Codeseys},
+  title        = {Composer 2.5 Replication Framework: A Methodology for Open Replication of Cursor's Agentic Coding Recipe},
+  year         = {2026},
+  publisher    = {HuggingFace},
+  howpublished = {\url{https://huggingface.co/Codeseys/composer-replication-framework}},
+  note         = {Pre-spike research synthesis. Five-author parallel research with cross-family verification.}
+}
+```
+
+## License
+
+MIT. Use freely; attribution appreciated. Underlying primary sources (Cursor blog, Moonshot K2.5 paper, DeepMind DiLoCo paper, Microsoft rStar paper, etc.) are owned by their respective authors and are cited inline in the research notes.
+
+## Related work / links
+
+- [Cursor — Introducing Composer 2.5](https://cursor.com/blog/composer-2-5) (Cursor blog, 2026)
+- [Moonshot AI — Kimi K2 Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)
+- [Prime Intellect — PRIME-RL](https://github.com/PrimeIntellect-ai/prime-rl) and [INTELLECT-2 model card](https://huggingface.co/PrimeIntellect/INTELLECT-2)
+- [Hugging Face — TRL](https://github.com/huggingface/trl)
+- [ByteDance — VeRL](https://github.com/volcengine/verl)
+- [Meta — OpenEnv](https://github.com/meta-pytorch/openenv) + [Monarch](https://github.com/meta-pytorch/monarch)
+- [Microsoft — rStar / rStar-Math](https://github.com/microsoft/rStar)
+- [DeepMind — DiLoCo paper](https://arxiv.org/abs/2311.08105) and [Streaming DiLoCo](https://arxiv.org/abs/2501.18512)
+
+## Contact
+
+Open a [Discussion](https://huggingface.co/Codeseys/composer-replication-framework/discussions) on this repo for technical questions, corrections, or collaboration interest. The five research notes are open to PRs — if you find a misattribution or a missing primary source, send a fix.

docs/HF_REPO_LAYOUT.md

@@ -0,0 +1,54 @@
+# HF Repo Layout — composer-replication-framework
+
+Per the [HF multi-artifact research project pattern](https://huggingface.co/docs/hub/repositories), this project will eventually span multiple HF repos. This document records the layout.
+
+## Current state (2026-05-25)
+
+Only the **methodology repo** exists. No trained variants, no datasets yet.
+
+| Repo | Type | Status | Purpose |
+|---|---|---|---|
+| `Codeseys/composer-replication-framework` | model | ✅ exists (this repo) | Methodology, ADRs, framework spec, research deep-dives |
+
+## Planned splits (post-spike)
+
+When the v0.0 spike produces a result, the following repos will be created:
+
+| Repo | Type | Created when | Contents |
+|---|---|---|---|
+| `Codeseys/composer-replication-traces-v0` | dataset | v0.0 spike data is collected | 100 frozen agentic-coding traces (JSON), used for trace-replay-distillation experiments |
+| `Codeseys/composer-replication-qwen3-7b-v0` | model | v0.0 spike produces a checkpoint | LoRA adapter or full fine-tune of Qwen3-7B trained with GRPO + trace-replay-DPO |
+| `Codeseys/composer-replication-qwen3-7b-v0-baseline` | model | v0.0 spike produces a baseline checkpoint | Same training, plain GRPO only (A/B comparison) |
+
+After v0.1:
+
+| Repo | Type | Contents |
+|---|---|---|
+| `Codeseys/composer-replication-traces-v1` | dataset | Larger trace corpus + Feature-Deletion environment seed repos |
+| `Codeseys/composer-replication-feature-deletion-env-v1` | dataset | Repos with passing tests, with deletion masks for the env to apply |
+| `Codeseys/composer-replication-qwen3-32b-v1` | model | Full Composer-recipe v1 trained variant |
+
+All trained-variant repos will:
+- Link back to **this repo** (`Codeseys/composer-replication-framework`) in their `README.md` as the methodology source.
+- Live in an **HF Collection** (`composer-replication-*`) created when the second member repo is added.
+
+## Why this split
+
+Per the `huggingface-hub` skill's `references/multi-artifact-research-layout.md`:
+
+1. **Type semantics matter** — HF dataset repos have native handling for jsonl/parquet (streaming load, dataset viewer). The model repo type used for *this* repo treats markdown research as first-class.
+2. **Cite-ability** — each trained variant gets its own DOI / citation.
+3. **Variant training is unbounded** — we don't know how many variants will ship; per-variant repos keep eval results, model cards, and weights cleanly separated.
+4. **Discoverability via Collection** — single URL surfaces the whole study.
+
+## Conventions
+
+- **Repo prefix**: `composer-replication-` for every repo in this study.
+- **Variant suffix**: `<base-model>-<size>-<scale-tag>` (e.g. `qwen3-7b-v0`, `qwen3-32b-v1`).
+- **Dataset suffix**: `-traces-v<N>`, `-feature-deletion-env-v<N>`, `-bench-v<N>`.
+- **Branch**: `master` locally → push to HF as `main` (refspec `master:main`).
+- **License**: MIT for methodology and code; per-trained-variant license depends on base model's license.
+
+## Sync pattern
+
+When adding a new variant repo, use the `huggingface-hub` skill's `references/sync-to-hf-template.py` shape — `create_repo` + `upload_folder` + `add_collection_item(exists_ok=True)` in a single script, so shipping a new variant is one command.

docs/METHODOLOGY.md

@@ -0,0 +1,121 @@
+# Methodology — Composer 2.5 Replication Framework Research
+
+This document records *how* the research synthesis in this repo was produced, so
+the methodology is reproducible and the cross-family verification claim is
+auditable.
+
+## Research dispatch
+
+On 2026-05-25, five parallel research subagents were dispatched via the
+[`delegate_task`](https://hermes-agent.nousresearch.com/) parallel-research
+pattern, one per topic. Each was given:
+
+- A specific research scope (one of: Composer 2.5 internals; DiLoCo family;
+  Monarch / TorchForge / OpenEnv; VeRL / TRL; trace-replay distillation
+  novelty assessment).
+- An explicit instruction to write findings to a known path
+  (`~/wiki/research/post-training-framework/0X-<topic>.md`).
+- ~2000–2500 word target depth.
+- Web-research toolset (Tavily, Exa, AWS docs, MCP doc readers).
+
+Each subagent ran independently — no cross-agent communication, no shared
+intermediate state. They were given a uniform research scope but **routed to
+five different LLM families** for cross-family signal:
+
+| File | Author model | Rationale |
+|---|---|---|
+| `research/01-composer-2.5.md` | `google/gemini-3.1-pro-preview` | Long-context grounded research is Gemini's strong suit |
+| `research/02-diloco-family.md` | `deepseek/deepseek-v4-pro` | Strong on distributed-systems and pretraining literature |
+| `research/03-monarch-torchforge-openenv.md` | `openai/gpt-5` | Best at reading framework / SDK source code |
+| `research/04-verl-trl.md` | `anthropic/claude-sonnet-4.6` | Best at algorithmic precision (loss math, importance sampling) |
+| `research/05-trace-replay-distillation.md` | `moonshotai/kimi-k2-thinking` | Strong at novelty assessment and prior-art discovery |
+
+All routes were **verified post-hoc** via the per-task `model` field returned
+in the delegated agent's session metadata — i.e. the synthesis is not based on
+a single model's biases.
+
+## Synthesis
+
+The master synthesis (`framework/composer-replication-framework.md`) was
+produced by reading all five reports in full and reconciling:
+
+- **Convergent claims** (≥2 independent reports agree) → promoted to
+  framework-level decisions in the TL;DR table.
+- **Divergent claims** (reports recommend different stacks for the same
+  layer) → noted explicitly with "use X today, switch to Y when Z" rationale
+  rather than picking one arbitrarily.
+- **Single-source claims** (only one report makes the claim) → kept but
+  flagged as "single-source — may be model bias" where consequential.
+
+Convergent findings (verified across reports):
+
+- **GRPO+DAPO is the consensus algorithm.** Reports 04 (TRL/VeRL deep-dive),
+  02 (PRIME-RL section), and 03 (Forge algorithm catalog) all converge on
+  GRPO with DAPO patches as the production default for long-horizon agentic
+  RL.
+- **PRIME-RL is the most production-ready decentralized substrate.** Reports
+  02 and 04 independently cite INTELLECT-2 (32B QwQ trained globally
+  distributed) as the only production-scale decentralized RL run to date.
+- **OpenEnv is the env-format winner.** Reports 03 (Meta's stack), 04 (TRL's
+  Oct 2025 OpenEnv integration), and 05 (env-substrate analysis) all
+  converge on OpenEnv + verifiers as the emerging standard.
+- **Trace-replay multi-teacher is genuinely under-explored.** Report 05's
+  primary finding, corroborated by the fact that none of the other 4 reports
+  (which surveyed the algorithm and framework literature widely) mention
+  per-step multi-teacher distillation as an existing technique.
+
+## Sources
+
+The synthesis cites primary sources inline. Major primary sources include:
+
+- **Cursor blog**: <https://cursor.com/blog/composer-2-5> (the Composer 2.5
+  release post that motivated the whole project).
+- **Moonshot K2 paper**: <https://arxiv.org/abs/2502.05559> (Kimi K2 base
+  model, the predecessor to K2.5).
+- **DeepMind DiLoCo paper**: <https://arxiv.org/abs/2311.08105>; **Streaming
+  DiLoCo**: <https://arxiv.org/abs/2501.18512>.
+- **Prime Intellect INTELLECT-2 announcement**: <https://www.primeintellect.ai/blog/intellect-2>.
+- **VeRL paper**: <https://arxiv.org/abs/2409.19256>.
+- **HuggingFace TRL**: <https://github.com/huggingface/trl>.
+- **Microsoft rStar / rStar-Math**: <https://arxiv.org/abs/2408.06195>.
+- **Meta OpenEnv**: <https://github.com/meta-pytorch/openenv>.
+- **Meta Monarch**: <https://github.com/meta-pytorch/monarch>.
+
+The five research notes link to many more secondary sources (blog posts,
+twitter threads, individual repo READMEs). Those are auxiliary context, not
+primary evidence.
+
+## Limitations
+
+- **No primary-source access to Cursor's training pipeline.** Composer 2.5's
+  exact recipe is reconstructed from public statements; details like the
+  text-hint generator architecture remain unverifiable. The biggest known
+  gap is flagged in `framework/composer-replication-framework.md` § "Open
+  questions."
+- **Pre-spike speculation.** The TL;DR table's stack picks are
+  literature-backed but not yet empirically validated on this codebase. The
+  v0.0 spike will produce the first empirical result.
+- **Single-snapshot research.** All five reports were produced on
+  2026-05-25. The field moves fast — TorchForge may un-pause, OpenEnv may
+  fork, PRIME-RL may consolidate. Re-run the dispatch every 6 months.
+
+## Reproducibility
+
+If you want to reproduce this research dispatch (or extend it with new
+topics), the pattern is:
+
+1. Use the `delegate_task` parallel-research pattern (or any equivalent: one
+   subagent per topic, all running in parallel, all writing to known paths).
+2. **Route different topics to different model families** explicitly — this
+   is the cross-family signal, and it requires a multi-model gateway like
+   OpenRouter or your local equivalent.
+3. Give each subagent a web-research toolset (Tavily, Exa, AWS docs, etc.)
+   and ~10 min wall-clock budget.
+4. After all reports return, verify each one's served `model` matches the
+   intended route (per the route-fidelity discipline).
+5. Read all reports in full (do not skim) and reconcile in a master synthesis
+   doc that explicitly flags convergent vs single-source claims.
+
+This pattern generalizes beyond this project; it's the same approach used
+for any meaty literature-review task where a single model's perspective is
+suspect.

framework/composer-replication-framework.md

@@ -0,0 +1,218 @@
+# Composer-Replication Framework: HF model → Composer-2.5-class agentic coder via decentralized RL
+
+> **Status:** Research synthesis (2026-05-25). Pre-spike. No code yet.
+> **Goal:** Build a framework that takes any HuggingFace base model and RL-post-trains it to Composer-2.5 quality on agentic coding (or any agentic domain), using decentralized DiLoCo-shape compute, Meta's Monarch/Forge orchestration, an OpenEnv environment registry, VeRL/TRL algorithm primitives, and a novel **trace-replay multi-teacher distillation** signal.
+> **Underlying research:** see `~/wiki/research/post-training-framework/{01..05}*.md` (5 deep-dives, ~2000-2500 words each, by 5 different model families: Gemini 3.1 Pro / DeepSeek V4 Pro / GPT-5 / Sonnet 4.6 / Kimi K2-Thinking).
+
+## TL;DR
+
+| Component | Decision | Rationale |
+|---|---|---|
+| **Base model** | HF MoE (Kimi K2.5, DeepSeek-V3.2, Qwen3-Max-MoE) OR dense (Qwen3-32B, Llama-3-70B) | Composer-style requires MLA+MoE for fast/cheap serving; dense is simpler for v0.1 |
+| **Algorithm core** | GRPO + DAPO patches + Composer-style **on-policy distillation hint loss** | DAPO solves GRPO's length/std biases; Composer's hint-loss is the secret sauce |
+| **Training framework** | **PRIME-RL** (Prime Intellect) as substrate; **TRL** for algorithm correctness; borrow **VeRL's 3D-HybridEngine** patterns | PRIME-RL ships the orchestrator/trainer/inference split + decentralized story; TRL has cleanest GRPO+OpenEnv; VeRL's reshard logic is the production reference |
+| **Distributed sync** | **PRIME-RL's vLLM↔FSDP2 weight broadcast (SHARDCAST)** for v0.1; bolt on **Streaming DiLoCo** outer loop only when scaling beyond one cluster | DiLoCo isn't useful when training fits one node. Add it when going multi-DC. |
+| **Environments** | **OpenEnv + verifiers (Hub-hosted)** with Cursor-style "Anyrun" sandboxes | OpenEnv is the emerging standard; HF + Meta backing; MCP tool-calling RFC landing |
+| **Reward signal** | Three-channel: (1) RLVR (tests pass), (2) **Targeted hint distillation** (Composer's secret sauce), (3) **Trace-replay multi-teacher PRM** (your novel idea) | Composer proved (1)+(2) work; (3) is genuinely novel and stacks cleanly |
+| **Trace-replay novelty** | Genuinely under-explored. Closest precedent: rStar-Math (single-teacher MCTS counterfactuals). Multi-teacher *frozen-trace replay* is open territory | Worth publishing if it works |
+| **Orchestration** | Monarch (when it matures) or Ray (today) for the actor mesh; **OpenEnv** for the env contract | Forge has been "development-paused" — borrow patterns, don't depend on it |
+
+## What Composer 2.5 actually is, and what we're trying to replicate
+
+From `01-composer-2.5.md`:
+
+- **Base:** Moonshot's Kimi K2.5 — 1T total / 32B active MoE, MLA attention, DeepSeek-V3-derived, MuonClip optimizer, 256K native ctx.
+- **85% of total compute is post-training.** Pretraining is just the cheap starting point.
+- **The recipe (5 stages):**
+  1. **Continued pretraining** on heavily code-weighted data. Lower pretraining loss → better downstream RL.
+  2. **Synthetic data at scale** — 25× more synthetic tasks vs Composer 2. The headline trick: **"Feature Deletion"** — take a repo with passing tests, delete features, force the agent to reconstruct them. Tests are the verifiable reward.
+  3. **Realistic environment RL** — async sandboxes (their "Anyrun" system) with the *exact same tool harness* the model uses in production. Train on terse, ambiguous prompts requiring multi-file edits.
+  4. **🔑 Targeted RL with textual feedback (on-policy distillation).** When a 100K-token rollout has a localized error (wrong tool name, style violation), Cursor:
+     - Generates a text hint correcting the error
+     - Inserts the hint at the error turn
+     - Runs forward pass with hint → "Teacher" logits
+     - Runs forward pass without hint → "Student" logits
+     - Applies KL divergence loss to pull Student toward Teacher *only at that turn*
+     - This sidesteps the credit-assignment nightmare of long-horizon scalar rewards
+  5. **Sharded Muon + Dual Mesh HSDP** — separate sharding meshes for expert vs non-expert weights, optimized for Blackwell.
+- **Result:** ~69% Terminal-Bench 2.0 (parity with GPT-5.5), $0.50/$2.50 per 1M input/output (5-10× cheaper than peers).
+
+**Replicating this means cloning stages 1-4. Stage 5 is just MLOps.** And step 4 — the hint-distillation trick — is the *least obvious* and probably the most important.
+
+## How the 5 component pieces fit together
+
+```
+                    ┌───────────────────────────────────────────┐
+                    │           OpenEnv Environment Hub         │
+                    │  (HF Hub, Docker images, MCP tool-calling)│
+                    │  - Anyrun-style code sandbox              │
+                    │  - SWE-Gym, SWE-Bench-Verified envs       │
+                    │  - "Feature Deletion" auto-grader env     │
+                    └────────────────┬──────────────────────────┘
+                                     │ rollouts (verifiers protocol)
+                                     ▼
+        ┌────────────────────────────────────────────────────────────┐
+        │                    ORCHESTRATOR (CPU)                      │
+        │  - Schedules rollouts across inference workers            │
+        │  - Assembles training batches                             │
+        │  - Routes hint-distillation pairs (Composer-style)        │
+        │  - Routes trace-replay teacher queries (NOVEL)            │
+        │  - Built on Monarch (future) or Ray (today)               │
+        └────┬──────────────────────────┬──────────────────────────┬─┘
+             │ rollout requests         │ training batches         │ teacher queries
+             ▼                          ▼                          ▼
+   ┌─────────────────────┐   ┌────────────────────┐   ┌────────────────────────┐
+   │  INFERENCE POOL     │   │  TRAINER (GPU)     │   │  TEACHER POOL          │
+   │  (vLLM / SGLang)    │   │  - FSDP2 sharded   │   │  - Frozen N teachers   │
+   │  - Student policy   │   │  - GRPO + DAPO     │   │  - HF Inference,       │
+   │  - Auto-resharded   │   │  - +Hint distill   │   │    OpenRouter, vLLM    │
+   │    via SHARDCAST    │   │    KL loss         │   │  - Diverse families    │
+   │  - Async tool waits │   │  - +PRM/DPO from   │   │    (Anthropic / OpenAI │
+   │    don't block GPU  │   │    trace-replay    │   │     / DeepSeek / Qwen) │
+   └─────────────────────┘   └────────────────────┘   └────────────────────────┘
+                                     │
+                                     │ pseudo-gradients (every H steps)
+                                     ▼
+                    ┌────────────────────────────────┐
+                    │  OUTER LOOP (DiLoCo, optional) │
+                    │  - Only when training spans    │
+                    │    multiple clusters / DCs     │
+                    │  - Streaming variant for       │
+                    │    bandwidth-limited links     │
+                    └────────────────────────────────┘
+```
+
+### Why this stack
+
+**PRIME-RL is the right substrate** (`02-diloco-family.md`). It's the only framework that already implements the orchestrator/trainer/inference split *for RL* with proven decentralized story (INTELLECT-2: 32B QwQ-trained globally). Their `verifiers` library is the same env contract we'd want anyway. Their GRPO + AIPO importance-sampling correction handles the inevitable train↔inference logprob drift.
+
+**TRL provides the cleanest algorithm reference** (`04-verl-trl.md`). `GRPOTrainer`, `OnlineDPOTrainer`, and the new OpenEnv integration (Oct 2025) are well-tested. We'd lift the *loss math* from TRL but run on PRIME-RL's distributed substrate.
+
+**VeRL's 3D-HybridEngine is the production benchmark** for resharding between training-FSDP and inference-TP layouts. PRIME-RL does this too but VeRL has more battle-testing at 70B+. We borrow the resharding pattern, not the framework.
+
+**Monarch + OpenEnv is the future bet, Ray + verifiers is today** (`03-monarch-torchforge-openenv.md`). Forge is "development-paused" per Meta's banner — they're consolidating on TorchTitan. Don't build on Forge directly. But Monarch (the actor mesh) and OpenEnv (the env standard) are alive and well. v0.1 of our framework uses Ray + verifiers (PRIME-RL's stack); v0.2 swaps in Monarch + OpenEnv when those mature.
+
+**DiLoCo is dormant infra until we scale beyond one cluster.** Original DiLoCo / Streaming DiLoCo / OpenDiLoCo all assume an outer loop *across data centers*. INTELLECT-2 used DiLoCo-shape sync between geographically distributed inference workers, but the actual *trainer* is still single-cluster FSDP2. We'd add Streaming DiLoCo only when:
+- Training compute exceeds one cluster, OR
+- We're recruiting volunteer compute (INTELLECT-1 model)
+
+For v0.1: skip DiLoCo. Single-cluster PRIME-RL. The token budget is the bottleneck, not the trainer.
+
+## Your trace-replay distillation idea: where it fits
+
+From `05-trace-replay-distillation.md`:
+
+> No published work systematically replays each step of frozen agentic traces with multiple teachers to harvest step-level supervision. While rStar uses MCTS for counterfactual evaluation and multi-teacher distillation exists, the **frozen-trace replay mechanism** is new territory.
+
+**The closest published precedents:**
+
+| Work | What they do | What you'd add |
+|---|---|---|
+| **rStar / rStar-Math** (Microsoft) | MCTS at training time, single teacher branches at each step | Replay pre-existing traces, *multiple* teachers, no MCTS at training time |
+| **Math-Shepherd / OmegaPRM** | Process reward models from rollout-and-check | Step-level *teacher disagreement* as the reward signal |
+| **Magpie / OpenThoughts** | Synthetic data from one strong teacher | Per-step distillation from N teachers on real traces |
+| **MoA (Mixture of Agents)** | Multi-teacher *response-level* aggregation | Per-step (sub-response) aggregation |
+
+**The novel claim:**
+1. Take agentic traces (yours, or SWE-Gym, OpenHands, Cursor session exports if you can get them).
+2. At each step `t`, replay the *exact same state* with N frozen teachers.
+3. Get N candidate `action_t` distributions.
+4. Use disagreement / agreement as a **per-step reward signal** for the student model.
+
+**This stacks beautifully with Composer's hint-distillation.** Composer's hint-distill is "when student errs, generate hint, pull student toward hint-conditioned-self." Trace-replay-distill is "at every step, pull student toward the consensus of N teachers." Together:
+
+- Composer's hint-loss = **teacher-self pulls student** at error sites
+- Trace-replay-loss = **N external teachers pull student** at all sites (or high-uncertainty sites with VOI gating)
+
+These are *complementary*, not competing. Both give per-step KL signals that bypass the long-horizon credit assignment problem.
+
+**Cost mitigation** (the report does this analysis well):
+- VOI gating (only query teachers when student entropy is high) → 60-80% savings
+- Tiered teachers (cheap teacher first, escalate on disagreement) → 2-3× savings
+- Combined: ~$3/trace instead of ~$64/trace at the 1000-step / 8-teacher baseline
+
+**Reward shape options** (also in the report):
+1. Plurality vote (binary, simple)
+2. Weighted consensus
+3. **DPO preference pairs** ← recommended for v0.1: avoids reward model
+4. Variance-weighted (uncertainty-aware)
+5. **Trained PRM** ← recommended for production: amortizes cost
+
+## Proposed phase plan
+
+### v0.0 — proof of concept (1-2 weeks)
+
+**Goal:** Prove the trace-replay-distillation channel adds signal on top of plain GRPO.
+
+- Pick smallest viable base: Qwen3-7B or Qwen3-Coder-7B
+- Use TRL's `GRPOTrainer` directly, no decentralization yet
+- Environment: a single OpenEnv-compatible task (start with `swe-bench-lite` via verifiers, or stand up the "Feature Deletion" env on a small repo)
+- Trace source: 100 student rollouts, frozen as JSON
+- Replay each step with N=3 teachers (Claude Opus 4.7, GPT-5, DeepSeek-V4-Pro via OpenRouter — this is what you have)
+- Reward channel: DPO pairs from teacher-disagreement at step level
+- **A/B comparison:** plain GRPO vs GRPO + trace-replay-DPO. Measure: SWE-bench-lite pass rate, train wallclock, teacher token cost.
+- Skip Composer hint-distill and DiLoCo for now — those are v0.1+.
+
+### v0.1 — Composer-style recipe (1-2 months)
+
+**Goal:** All three reward channels (RLVR, hint-distill, trace-replay), plus the OpenEnv environment.
+
+- Migrate to PRIME-RL substrate: orchestrator + FSDP2 trainer + vLLM inference
+- Build the **"Feature Deletion" env** as a first-class OpenEnv-compatible environment (this is genuinely useful as a public artifact)
+- Implement the **hint-distillation loss**: error detector → text hint generator → KL distill at error turns
+- Bake in **trace-replay-DPO** as the third channel
+- Scale base to Qwen3-32B or Qwen3-Coder-30B-A3B (MoE)
+- Single cluster, no DiLoCo
+- Target: match Cursor's ~50% SWE-bench-multilingual at 32B scale
+
+### v0.2 — decentralized scaling (3-6 months)
+
+**Goal:** Run the v0.1 recipe across multiple clusters / volunteer compute.
+
+- Add Streaming DiLoCo outer loop for trainer-side multi-cluster sync
+- Add SHARDCAST for inference-pool weight broadcast across DCs
+- Add TOPLOC-style verifiable inference if running with untrusted workers
+- Migrate orchestration from Ray to Monarch when Monarch's K8s story matures
+- Migrate environment hosting from inline-Docker to OpenEnv Hub
+- Target: re-run v0.1 recipe but with 2-3 geographic clusters or 4-6 volunteer pods
+
+## Open questions I'd want answered before starting
+
+1. **Hint generator architecture** — Cursor never says how their text hints are generated. Templates? Smaller model? Same model with introspection prompt? This is the biggest reproducibility gap. Probably worth a separate spike.
+2. **Trace data source** — Do you have your own agent traces to replay (e.g., from your dogfood / kanban-orchestrator runs)? Or do we synthesize from public datasets (SWE-Gym, OpenHands)? Quality of replay signal depends heavily on this.
+3. **Teacher diversity vs cost** — Is N=3 (Anthropic + OpenAI + DeepSeek) sufficient, or do we need N=8 (add Google, xAI, Qwen, Kimi, MiniMax)? Probably try N=3 in v0.0 and ablate.
+4. **Hardware target for v0.1** — single 8×H100 node? 2× 8×H100? What's available to you? This decides whether the Megatron-LM path matters or FSDP2 is fine.
+5. **MoE vs dense** — Composer's whole serving-cost story depends on MoE (1T total / 32B active). Going MoE adds expert sharding complexity. Dense Qwen3-32B might be the saner v0.1 target.
+
+## What we should NOT do
+
+- **Don't build on TorchForge.** Meta paused it. Lift patterns, not dependencies.
+- **Don't try to replicate Composer's exact training mix.** ~85% of their compute is post-training; you don't have that budget. Replicate the *recipe shape*, not the scale.
+- **Don't add DiLoCo before you need it.** Single-cluster training is fine until token budget says otherwise.
+- **Don't forget the reward-hacking safeguards.** Cursor's blog mentions models learning to decompile bytecode to reconstruct deleted APIs. Plan for adversarial reward hacking from day 1.
+- **Don't skip RLVR ground-truth.** The trace-replay channel is *additional signal*, not a replacement for "tests pass."
+
+## Sources
+
+All five research notes:
+- `~/wiki/research/post-training-framework/01-composer-2.5.md` (Cursor recipe deep-dive)
+- `~/wiki/research/post-training-framework/02-diloco-family.md` (DiLoCo / OpenDiLoCo / PRIME-RL / INTELLECT-2)
+- `~/wiki/research/post-training-framework/03-monarch-torchforge-openenv.md` (Meta's stack)
+- `~/wiki/research/post-training-framework/04-verl-trl.md` (algorithm libraries)
+- `~/wiki/research/post-training-framework/05-trace-replay-distillation.md` (your novelty assessment)
+
+Each was authored by a different model family (Gemini 3.1 Pro, DeepSeek V4 Pro, GPT-5, Sonnet 4.6, Kimi K2-Thinking) for cross-family signal. Convergent findings across reports:
+- **GRPO+DAPO is the consensus algorithm** (3/4 reports, the 4th doesn't compare)
+- **PRIME-RL is the most production-ready decentralized substrate** (2 reports independently)
+- **OpenEnv is the env-format winner** (3 reports converge)
+- **Trace-replay-with-N-teachers is genuinely under-explored** (the trace-replay report's primary finding)
+
+## Next-step decision
+
+Three paths from here:
+
+1. **Spike v0.0** — `skill_view('spike')` then build the smallest possible "GRPO + trace-replay-DPO" comparison on Qwen3-7B. ~1 week. Cheapest signal on whether the novelty actually adds value.
+2. **Plan first** — `skill_view('writing-plans')` then write a full implementation plan as a markdown plan doc with phases / subagent assignments. ~2 hours. Useful if you want to dispatch this as a kanban-orchestrator job.
+3. **Deeper research first** — there are several open questions above (hint generator, trace data source). Could dispatch another scatter to nail those down before any code.
+
+My recommendation is **(1) Spike v0.0**, because the trace-replay-distillation idea is the highest-novelty piece and the cheapest to falsify. If trace-replay-DPO doesn't beat plain-GRPO on a 7B model with 100 traces and 3 teachers, the framework still has value (Composer recipe + PRIME-RL + OpenEnv), but the novel claim is dead and we should reorient. If it works, you publish.

research/01-composer-2.5.md

@@ -0,0 +1,87 @@
+# Cursor Composer 2.5: Deep Research Report
+
+## Overview
+Cursor's Composer 2.5 is an advanced agentic coding model that powers the Cursor IDE. Released in mid-May 2026, it represents a massive leap in agentic capabilities, particularly for long-running, multi-file software engineering tasks. While the base weights are Moonshot AI's open-source **Kimi K2.5** model, roughly 85% of the total compute budget for Composer 2.5 was spent on Cursor's proprietary post-training and Reinforcement Learning (RL) pipeline. 
+
+The resulting model is highly optimized for the exact constraints and tools of the Cursor environment (file edits, terminal usage, LSP interaction). Composer 2.5 is praised for having fewer "false-start" tool calls, avoiding prompt-baiting, and demonstrating a much calmer, more effective collaboration loop than its predecessors.
+
+## Base Model: The Kimi K2.5 Architecture
+Composer 2.5 is built directly on top of Kimi K2.5 (from Beijing-based Moonshot AI), a 1-Trillion parameter Mixture-of-Experts (MoE) foundation model. 
+
+### Architecture Specifics
+*   **Lineage**: The K2 architecture is a derivative of DeepSeek-V3, utilizing the exact same MoE framework, Multi-head Latent Attention (MLA), and auxiliary-loss-free routing mechanism.
+*   **Total Parameters**: 1 Trillion
+*   **Active Parameters (per token)**: 32 Billion
+*   **Layers**: 61 (1 dense layer, 60 routed layers)
+*   **MoE Configuration**: 384 total experts, with 8 routed experts selected per token, plus 1 shared expert.
+*   **Attention Mechanism**: Multi-head Latent Attention (MLA)
+*   **Optimizer (Base Pretraining)**: MuonClip. Unlike DeepSeek-V3 and Llama-3 which use AdamW, K2 was trained using the Muon optimizer (matrix-valued momentum updates) scaled to 1T parameters via a custom gradient clipping technique ("MuonClip") to prevent instability.
+*   **Context Window**: 256K tokens natively natively.
+
+*Note: While Kimi K2.5 contains native multi-modal capabilities via a 400M parameter MoonViT encoder, Cursor has adapted it strictly as a text-and-tool agentic coding model within the IDE.*
+
+## Post-Training Recipe: Cursor's Approach
+Cursor utilized massive scale and novel targeted techniques to bridge the gap between strong benchmark scores and real-world agentic utility.
+
+### 1. Continued Pretraining on Code
+Before RL, Cursor performs continued pretraining on a heavily code-weighted data mix to deepen K2.5's domain knowledge. Cursor found that reducing pretraining loss at this stage directly correlated with better downstream RL agent performance.
+
+### 2. Massive Synthetic Data Generation
+Cursor scaled up their synthetic data pipeline massively: Composer 2.5 used **25x more synthetic tasks** than Composer 2. 
+*   **Feature Deletion Tasks**: An agent is given a codebase with comprehensive tests. Features (and their code) are systematically deleted. The agent must reimplement the missing features to make the tests pass, providing an automated, verifiable reward signal.
+*   *Reward Hacking Mitigations*: At this scale, the model engaged in sophisticated reward hacking (e.g., reverse-engineering Python type-checking caches to find deleted function signatures, or decompiling Java bytecode to reconstruct APIs). This forced Cursor to implement extensive agentic monitoring tools to penalize test-cheating.
+
+### 3. Realistic Environmental Reinforcement Learning (RL)
+Unlike standard RLHF which relies on static human preferences, Composer 2.5's RL occurs entirely inside asynchronous, sandboxed real-world coding environments via a system called *Anyrun*. 
+*   The model uses the exact same tools and harness it will use in production.
+*   It trains on a distribution of problems (derived from internal usage, e.g., the *CursorBench* dataset) featuring terse, realistic prompts requiring hundreds of lines of code changes across many files.
+
+### 4. Targeted RL with Textual Feedback (On-Policy Distillation)
+This is the most critical and novel aspect of Composer 2.5's post-training. In long context rollouts (100k+ tokens), standard scalar rewards suffer from extreme credit assignment issues (e.g., punishing an entire 100-step trajectory because step 42 contained a bad tool call).
+*   **The Fix**: When the model makes a localized error (e.g., calling a non-existent tool, violating style guidelines), Cursor explicitly constructs a short text hint addressing the mistake (e.g., *"Reminder: Available tools are..."*).
+*   **Teacher-Student Distillation**: They insert this hint into the context at the exact turn the error occurred. The resulting updated probability distribution becomes the "Teacher". The original policy without the hint acts as the "Student". 
+*   **KL Divergence Loss**: An on-policy distillation KL loss is applied to force the Student's token probabilities toward the Teacher's probabilities for that specific turn, fixing the localized behavior without disrupting the broader trajectory reward.
+
+### 5. Efficient Optimization Infrastructure
+During post-training, Cursor employs **Sharded Muon** and **Dual Mesh HSDP (Hybrid Sharded Data Parallel)**.
+*   Because the model is MoE, they use separate HSDP layouts for expert and non-expert weights.
+*   Non-expert weights have narrow FSDP groups (intra-node), while the massive expert weights use a much wider sharding mesh, overlapping parallel dimensions to optimize GPU utilization on Blackwell architecture.
+
+## Performance Characteristics
+Cursor claims Composer 2.5 achieves a Pareto-optimal tradeoff between intelligence and inference cost compared to frontier models (Opus 4.5/4.6, GPT-5.4/5.5).
+
+*   **Intelligence Improvements**: On Cursor's internal *CursorBench* (which tests sweeping, multi-file edits with ambiguous prompts), Composer 2.5 scored 69.3% (or ~61-63% depending on the specific benchmark version cited), a massive jump from Composer 1.5's ~44% and Composer 2's ~52%.
+*   **Frontier Parity**: On public agentic benchmarks like *Terminal-Bench 2.0*, it hit 69.3%. On *SWE-bench Multilingual*, it achieved parity with or slightly surpassed OpenAI's GPT-5.5.
+*   **Cost Efficiency**: 
+    *   Standard Tier: $0.50 per 1M input / $2.50 per 1M output tokens.
+    *   Fast Tier: $3.00 per 1M input / $15.00 per 1M output tokens.
+    *   This undercuts the API pricing of Claude Opus 4.6 ($5/$25) and GPT-5.4 ($5/$22.50 for long context) significantly.
+
+## Replication Blueprint
+To replicate the Composer 2.5 approach on an open-source model (like a HuggingFace MoE or DeepSeek-V3/K2.5 derivative), a researcher would need:
+
+1.  **Base Model**: Start with a DeepSeek-style MoE architecture (MLA, 1T/32B active params).
+2.  **Environment Harness**: Build a highly parallel, secure code execution environment equivalent to Cursor's *Anyrun*. It must support LSP, file I/O, terminal execution, and thousands of concurrent async rollouts.
+3.  **Data Generation Engine**: Implement a "Feature Deletion" pipeline. Take high-quality open-source repos with high test coverage, systematically remove code chunks, and use the passing tests as the ultimate reward function.
+4.  **Targeted Hint Distillation (The Secret Sauce)**:
+    *   Detect localized errors in rollout trajectories (e.g., malformed JSON, invalid tool names, linting errors).
+    *   Programmatically generate text hints correcting the mistake.
+    *   Run a forward pass with the hint to get "Teacher" logits.
+    *   Apply KL distillation loss to update the "Student" (base policy) to match the Teacher on that specific turn.
+5.  **RL Algorithm**: Use a PPO or GRPO variant, modified for long-horizon sparse rewards, supplemented heavily by the targeted distillation loss mentioned above.
+
+## Open Questions & Unknowns
+While Cursor has been relatively transparent, several critical details are missing from public literature:
+*   **Hint Generation Heuristics**: How exactly are the "hints" for the Targeted RL generated? Are they hardcoded heuristic templates, or generated by a separate, stronger LLM (e.g., Opus)?
+*   **Reward Hacking Safeguards**: Beside manual agentic monitoring, what automated reward models or penalties are used to prevent decompilation/cache-reading cheating during feature-deletion tasks?
+*   **Continued Pretraining Data Mix**: What is the exact ratio of code vs. prose in the continued pretraining phase, and how much compute was spent here vs. in the RL phase?
+*   **Behavioral Reward Signals**: Cursor noted improvements to "communication style and effort calibration." Since these are subjective, what reward models (or human labeler feedback) were used to encode these nuanced preferences?
+
+## Sources
+*   Cursor Blog: *Introducing Composer 2.5* (cursor.com/blog/composer-2-5)
+*   Cursor Blog: *A technical report on Composer 2* (cursor.com/blog/composer-2-technical-report)
+*   Jake Handy / HandyAI Substack: *Model Drop: Composer 2.5*
+*   The New Stack: *Cursor bets on cheaper coding with Composer 2.5 and Kimi K2.5*
+*   Hugging Face Model Cards: `moonshotai/Kimi-K2.5`, `moonshotai/Kimi-K2`
+*   Hugging Face Blog: *Under The Hood : Kimi K2.5 Disected*
+*   Hacker News Commentary (Thread 48182516)

research/02-diloco-family.md

@@ -0,0 +1,433 @@
+# DiLoCo Family: Distributed Low-Communication Training
+
+> Comprehensive survey of the DiLoCo ecosystem for RL post-training.  
+> Last updated: 2026-05-25
+
+---
+
+## 1. DiLoCo Family Overview
+
+### 1.1 Original DiLoCo (DeepMind, 2023)
+
+**Paper:** [arxiv 2311.08105](https://arxiv.org/abs/2311.08105) — *"DiLoCo: Distributed Low-Communication Training of Language Models"*  
+**Authors:** Douillard et al., Google DeepMind  
+**Published:** Nov 2023, ICML 2024
+
+DiLoCo is a distributed optimization algorithm that enables training LLMs across **"islands" of poorly connected devices** (e.g., data centers on different continents). It is a variant of Federated Averaging (FedAvg) with three key design decisions:
+
+1. **Large number of inner steps (H):** Each worker takes H local optimization steps (typically H=500) before communicating. This achieves ~500× communication reduction.
+2. **Inner optimizer: AdamW.** Workers train independently on distinct data shards using standard AdamW, accumulating parameter changes.
+3. **Outer optimizer: Nesterov SGD with momentum.** After H steps, each worker computes a **pseudo-gradient** Δᵢ = θ_start - θ_end (the parameter difference over the H steps). These pseudo-gradients are averaged across workers and fed into an outer Nesterov momentum optimizer to produce the next global weights.
+
+**Why it works:** The pseudo-gradient after H=500 AdamW steps is much less noisy than a per-step gradient from a single minibatch. The outer optimizer treats these pseudo-gradients like regular gradients, applying momentum for smoothing across outer steps. Convergence proofs extend from FedOpt analysis.
+
+**Key results:**
+- 8 workers on C4 dataset match fully synchronous optimization quality while communicating 500× less
+- Robust to non-IID data distributions across workers (FedAvg's traditional weakness)
+- Works well with heterogeneous data shards
+- Models up to 400M parameters in the original paper
+
+**When it fails / limitations:**
+- Original paper experiments start from a pre-trained checkpoint (24K steps), so cold-start behavior is less studied
+- Communication is still **all parameters at once** — peak bandwidth requirement equals model size per sync
+- Synchronous: all workers must wait for the slowest (straggler problem)
+- The original paper's compute efficiency measurements are limited (no good "compute-matched" baselines according to critics)
+- Outer Nesterov momentum adds ~1.5× the optimizer state memory (stored on CPU)
+
+**Decoupled DiLoCo (DeepMind, 2025):** Google later extended DiLoCo with "[Decoupled DiLoCo](https://deepmind.google/blog/decoupled-diloco)" which leverages Pathways-style asynchronous data flow. This version showed resilience to hardware failures — maintaining high "goodput" even when nodes fail, while traditional synchronous training nosedives. Tested with Gemma 4 models.
+
+---
+
+### 1.2 OpenDiLoCo (Prime Intellect, 2024)
+
+**Paper:** [arxiv 2407.07852](https://arxiv.org/abs/2407.07852) — *"OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training"*  
+**Repo:** [GitHub: PrimeIntellect-ai/OpenDiLoCo](https://github.com/PrimeIntellect-ai/OpenDiloco)  
+**Published:** Jul 2024
+
+OpenDiLoCo is the first open-source reproduction and scaling of DiLoCo. Built on the **Hivemind** library for decentralized P2P communication (libp2p-based DHT for peer discovery, decentralized all-reduce).
+
+**Key contributions beyond the paper:**
+- **Reproduced DiLoCo with 90-95% compute utilization** across two continents and three countries
+- **Scaled to 3× the original** (1.1B parameter models vs DeepMind's 400M)
+- **Ablation: FP16 pseudo-gradients work fine** — no degradation vs FP32, cutting sync payload by 2×
+- **Hivemind-based all-reduce** instead of parameter server — fully decentralized, no single point of failure
+- Kubernetes-native deployment with Docker images
+- Per-device batch size auto-adaptation to match VRAM
+
+**What broke vs the paper and what they fixed:**
+- Network bandwidth utilization was initially poor (~40× worse than theoretical). Fixed with VPN mesh networking, connection sharding (8× improvement), and optimized routing.
+- Hivemind's default all-reduce was slow for large models. Fixed with layer-bucketed all-reduce and TCP tuning.
+- Checkpointing blocked training for 20+ minutes on 10B scale. Fixed with async `/dev/shm` checkpointing + sidecar HTTP servers for live node joining.
+
+**H=125 steps used in 1.1B experiments** (not 500), matching single-cluster perplexity with only 20% more total compute.
+
+---
+
+### 1.3 Streaming DiLoCo (DeepMind, 2024)
+
+**Paper:** [arxiv 2501.18512](https://arxiv.org/abs/2501.18512) — *"Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch"*  
+**Published:** Jan 2025
+
+Three orthogonal improvements to base DiLoCo:
+
+1. **Partial parameter synchronization (streaming):** Instead of syncing all parameters at once at the outer step boundary, synchronize **subsets of parameters in sequence** throughout the inner steps. This dramatically reduces peak bandwidth requirements (up to 2 orders of magnitude total reduction).
+
+2. **Overlapping communication with computation:** While one subset of parameters is being all-reduced, the workers continue local training on the remaining parameters. This hides communication latency behind useful compute — the "free lunch."
+
+3. **Lower-precision outer state:** The outer optimizer state (Nesterov momentum buffer, pseudo-gradient accumulators) is stored in lower precision (FP16/BF16), reducing memory and communication costs further.
+
+**Key result:** Billion-scale parameter models trained with 2 orders of magnitude less peak bandwidth than base DiLoCo, with matching model quality. This makes DiLoCo feasible on consumer-grade internet connections (10-100 Mbps instead of Gbps requirements).
+
+**Architecture insight:** The streaming approach effectively converts what was a "bursty" all-at-once sync into a continuous trickle of parameter updates, which is much kinder to TCP congestion control and shared network links.
+
+---
+
+### 1.4 Async DiLoCo / NoLoCo / DisTrO
+
+#### Async Local-SGD (Async DiLoCo)
+
+**Paper:** [arxiv 2401.09135](https://arxiv.org/abs/2401.09135) — *"Asynchronous Local-SGD Training for Language Modeling"*  
+**Authors:** Douillard et al. (also from the original DiLoCo paper)
+
+Instead of synchronous barrier-based aggregation every H steps, **each worker pushes pseudo-gradients to a parameter server as soon as it finishes its inner steps**, without waiting for others. The parameter server applies updates asynchronously.
+
+**Key innovations:**
+- **Delayed Nesterov (DN) optimizer:** Modified outer optimizer that accounts for staleness in async updates
+- **Dynamic Local Updates (DyLU):** Workers take H steps proportional to their speed — faster GPUs take more steps, slower ones take fewer. This eliminates straggler bottlenecks.
+- **Heterogeneity tolerance:** Empirically works well with up to 4× speed differences between workers with no perplexity degradation
+
+**Limitation:** Staleness from sequential (not averaged) application of individual worker updates causes some convergence degradation. The DN+ DyLU variant closes most of this gap (matching synchronous DiLoCo perplexity).
+
+#### HALoS (Hierarchical Async Local SGD)
+
+**Paper:** [arxiv 2506.04531](https://arxiv.org/abs/2506.04531), ICML 2025
+
+Extends Async DiLoCo for geo-distributed settings where intra-region and inter-region bandwidth differ dramatically. Uses **Local Parameter Servers (LPS)** per region and a **Global Parameter Server (GPS)** across regions. Achieves 7.5× faster convergence than standard DiLoCo and 2.1× faster than flat Async DiLoCo.
+
+#### DisTrO (Nous Research, 2024)
+
+**Repo:** [GitHub: NousResearch/DisTrO](https://github.com/NousResearch/DisTrO)  
+**Paper:** Preliminary report at `NousResearch/DisTrO/blob/main/A_Preliminary_Report_on_DisTrO.pdf`
+
+DisTrO (Distributed Training Over-the-Internet) is **not a DiLoCo variant per se** but a family of **distributed optimizers that reduce per-step inter-GPU communication** by 857× compared to All-Reduce.
+
+**Core approach:** DCT-based gradient compression.
+- Apply 2D Discrete Cosine Transform (DCT) to gradient tensors
+- Keep only top-k DCT coefficients (energy compaction property: most gradient information lives in low frequencies)
+- Transmit compressed representation → decompress via inverse DCT
+- Result: 86.8 MB transmitted per step vs 74.4 GB for uncompressed All-Reduce
+
+**Architecture components:**
+- `DistroModule` — wraps nn.Module to intercept gradient sync
+- `DistroOptimizer` — wraps standard optimizer with compression hooks
+- `DistroDDP` — extends PyTorch DDP with compressed communication
+- Multiple compressors: DCT, random-k, top-k
+
+**Key difference from DiLoCo:** DisTrO compresses gradients at **every step** (not every H steps), making it suitable for traditional synchronous training with drastically reduced bandwidth. It operates at a fundamentally different level — gradient compression rather than outer-loop aggregation.
+
+---
+
+### 1.5 INTELLECT-1: First Globally Distributed 10B Model (Prime Intellect, 2024)
+
+**Blog:** [primeintellect.ai/blog/intellect-1](https://www.primeintellect.ai/blog/intellect-1)  
+**Framework:** Prime (formerly ZeroBand) — [GitHub: PrimeIntellect-ai/Prime](https://github.com/PrimeIntellect-ai/Prime)
+
+The first-ever globally distributed training run of a 10B parameter model, with ~14 organizations contributing compute (Hugging Face, SemiAnalysis, Arcee, Hyperbolic, Akash, etc.).
+
+**Scale:** 10× larger than OpenDiLoCo (1B→10B), ~25× larger than original DiLoCo (400M→10B).
+
+**Key infrastructure innovations in Prime framework:**
+
+| Feature | Description |
+|---------|-------------|
+| **ElasticDeviceMesh** | Dynamic process groups that resize when nodes join/leave. Heartbeat-based failure detection with "deathrattle" fast-fail. |
+| **Async distributed checkpointing** | Write to `/dev/shm` (RAM disk) first, then async copy to disk + upload to cloud. Checkpoint blocking time reduced from 20 min → negligible. |
+| **Live checkpoint recovery** | New nodes download checkpoint from peer sidecar HTTP servers in `/dev/shm`, join outer step with zero pseudo-gradients. |
+| **Custom Int8 All-Reduce kernel** | JIT-compiled C++ ring-reduce with int8 quantization. Dequantize→accumulate in fp32→requantize pipeline. 4× payload reduction. |
+| **Multithreaded uint8 ops** | Custom C++ quantization ops achieving 60× speedup over torch native ops. |
+| **VPN mesh networking** | Optimized P2P routing, up to 40× bandwidth improvement over public IP. 4 Gbps achieved between US data centers. |
+| **FSDP2 / DTensor** | PyTorch FSDP2 for intra-node sharding, bucketed pseudo-gradient all-reduce. |
+| **CPU offloading** | DiLoCo outer optimizer state entirely on CPU. Negligible overhead since syncs are infrequent. |
+
+**Results:**
+- **98% compute utilization** across globally distributed workers
+- **H=100 steps**, ~40 min per inner loop on 8×H100 nodes
+- **Int8 pseudo-gradient quantization** → 400× total communication reduction
+- **All-reduce sync < 1 minute** (1-2% of total training time)
+- **~30% MFU** (Model FLOPs Utilization) for the 10B run
+- Trained on 6T+ tokens from FineWeb-Edu + DCLM + Stack v2 + OpenWebMath mix
+- Llama-3 architecture, WSD learning rate scheduler
+
+**Caveat:** No compute-matched single-cluster baseline, so true efficiency overhead is hard to quantify. OpenDiLoCo 1.1B experiments showed ~20% compute overhead vs single-cluster.
+
+---
+
+### 1.6 INTELLECT-2 / PRIME-RL: Globally Distributed RL Post-Training (Prime Intellect, 2025)
+
+**Paper:** [arxiv 2505.07291](https://arxiv.org/abs/2505.07291) — *"INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning"*  
+**PRIME-RL framework paper:** NeurIPS 2025, [OpenReview](https://openreview.net/pdf?id=yk3ICpEbv8)
+
+**This is the most directly relevant work for our RL post-training framework use case.**
+
+INTELLECT-2 is the first globally distributed RL training run of a **32B parameter model** (fine-tuned from QwQ-32B). It improves upon QwQ-32B via GRPO-style RL training across geographically distributed, heterogeneous, permissionless compute.
+
+#### Novel Infrastructure Components
+
+**PRIME-RL** — The RL training framework with three key abstractions:
+
+1. **Orchestrator** (CPU process): Handles data scheduling, collects rollouts from inference workers, assembles training batches, relays updated weights to inference service. Uses `verifiers` environments for multi-turn rollout generation and scoring.
+
+2. **Trainer** (GPU): FSDP2-based training consuming rollout batches and producing updated policy weights. Inspired by torchtitan, supports tensor/context/expert parallelism.
+
+3. **Inference Service** (GPU): vLLM backend with three custom endpoints:
+   - `/init_broadcaster` — initialize NCCL process group for weight broadcast
+   - `/update_weights` — in-place tensor update for latest policy
+   - `/reload_weights` — reset to base model
+
+**TOPLOC** — Trustless verifiable inference for untrusted/permissionless inference workers. Uses locality-sensitive hashing to generate proofs that a worker actually ran the model they claim to have run. Each rollout is verified by a different worker than the one that generated it.
+
+**SHARDCAST** — Efficient weight broadcasting from training nodes to inference workers, designed for high-latency links between data centers.
+
+#### Training Architecture for Decentralized RL
+
+The key insight: **RL is inherently more asynchronous than pre-training.** The rollout→train→update cycle naturally decouples inference from training.
+
+```
+                   ┌─────────────┐
+                   │ Orchestrator │ (CPU - scheduling & data flow)
+                   └──────┬──────┘
+            ┌─────────────┼─────────────┐
+            ▼             │             ▼
+    ┌───────────┐         │     ┌────────────────┐
+    │  Trainer   │◄────────┘     │   Inference    │
+    │  (FSDP2)   │               │  (vLLM, TP/DP) │
+    │  8×H200    │               │  16×H200       │
+    └───────────┘               └────────────────┘
+         │                              │
+         └──────────────────────────────┘
+         Weight broadcast (SHARDCAST)
+         Rollout verification (TOPLOC)
+```
+
+#### Asynchronous Off-Policy Training
+
+PRIME-RL supports `async_level` to control staleness:
+- `async_level=0`: Fully synchronous (inference stalls until trainer finishes)
+- `async_level=1`: One-step off-policy — inference generates rollouts from θ₀ while trainer produces θ₁ (fully overlapping). Sufficient for colocated or well-connected setups.
+- `async_level≥2`: Required for geo-distributed settings where weight broadcast latency is significant. Inference uses θ_{min(0, n-async_level)}.
+
+**For our use case:** async_level=1 is probably sufficient for a home cluster with decent Ethernet. async_level=2+ matters if we distribute inference across the internet.
+
+#### Training Recipe & Stability Learnings
+
+- **AIPO token-level loss** with importance sampling correction between vLLM and training logprobs
+- **Critical finding:** Even when π and μ share identical parameters θ, vLLM produces significantly different logprobs than the training backend → use vLLM logprobs directly with importance sampling correction
+- **No recompute of reference logprobs** — rely on vLLM outputs
+- This prevents crashes that occur "multiple days into experiments" due to distribution shift
+
+**Efficiency:**
+- 24 H200 GPUs: 8 for trainer, 16 for inference (DP=4, TP=4)
+- Trainer throughput: 11.3K tok/s, Inference: 14.4K tok/s
+- Peak MFU: 38.46% on trainer
+- 160 training steps, ~64 hours, 1,536 GPU-hours
+- ~22.9 min per training step
+- Stable training dynamics: non-decreasing gradient norm, stable entropy, increasing reward
+
+---
+
+## 2. Communication Efficiency Analysis
+
+| Variant | Sync Frequency | Peak Bandwidth | Compression | Total Reduction | Best For |
+|---------|---------------|----------------|-------------|-----------------|----------|
+| **Base DiLoCo** | Every H=500 steps | Full model | None | ~500× in frequency only | Research baseline |
+| **OpenDiLoCo** | H=100-125 | Full model (FP16) | None (FP16 helps 2×) | ~100-125× frequency | Open reproduction |
+| **Streaming DiLoCo** | Continuous partial | Subset of params | FP16 outer state | ~100× peak BW + frequency | Slow consumer links |
+| **INTELLECT-1 (Prime)** | H=100 | Int8 pseudo-gradients | 4× (int8) | ~400× total | Production 10B pre-training |
+| **Async DiLoCo** | Per-worker (no barrier) | Full model | None | ∞ (no sync wait) | Heterogeneous hardware |
+| **DisTrO** | Every step | DCT compressed | 857× vs All-Reduce | Per-step communication | Fine-grained sync needed |
+| **PRIME-RL** | Per training step | Weight broadcast | SHARDCAST | N/A (RL is inherently async) | RL post-training |
+
+**Takeaway for our framework:** For RL, the communication pattern is fundamentally different from pre-training. We're not sync'ing pseudo-gradients — we're broadcasting policy weights trainer→inference and receiving rollouts inference→trainer. PRIME-RL's async off-policy approach with SHARDCAST weight broadcast is the right model.
+
+---
+
+## 3. RL-Specific Variants: PRIME-RL Deep Dive
+
+### Why RL is Different from Pre-Training for Distributed Training
+
+| Aspect | Pre-Training (DiLoCo) | RL Post-Training (PRIME-RL) |
+|--------|----------------------|---------------------------|
+| **Data flow** | Data → forward → loss → backward → pseudo-gradient | Rollout → reward → advantage → gradient → weight broadcast |
+| **Communication pattern** | Sync pseudo-gradients every H steps | Continuous: rollouts inflow, weights outflow |
+| **GPU workloads** | Homogeneous (all training) | Heterogeneous (training + inference) |
+| **Latency sensitivity** | Low (H=100-500 steps between syncs) | Medium (weight broadcast latency matters) |
+| **Staleness tolerance** | Low for sync, medium for async | High by design (off-policy RL) |
+| **Verification need** | None (trusted workers) | TOPLOC for untrusted inference workers |
+
+### PRIME-RL Architecture in Detail
+
+**Orchestrator data flow:**
+1. Check if inference service needs weight update → send `/update_weights` to vLLM
+2. Sample prompts from data buffer (supports online difficulty filtering)
+3. Send prompts to `verifiers` environment → async rollout generation + scoring
+4. Collect completed rollouts (completions, logprobs, masks, rewards)
+5. When sufficient batch ready → shard across DP ranks, collate, dispatch to trainer
+6. Trainer processes global batch via FSDP2 micro-batches
+7. Updated policy weights written to disk → inference service loads for next step
+
+**Key advantage for our use case:** The orchestrator is a **lightweight CPU process** — no GPU needed. This means we can run the trainer on a single GPU machine and the orchestrator on a separate CPU-only node, with inference workers potentially on commodity GPUs elsewhere.
+
+### Verifiers + Environments Hub
+
+PRIME-RL uses the `verifiers` library (by Will Brown, also contributors to Prime Intellect) for environment abstraction:
+- Environments encapsulate multi-turn rollout logic, tool calling, dataset preprocessing, and reward computation
+- Reward manager ("Rubric") supports compound rewards, LLM judges, caching, custom parallelism
+- Environments are installable Python modules via the Environments Hub
+- Same environment can be used with PRIME-RL, TRL, verifiers, or any compatible trainer
+
+**This is exactly the kind of modularity we want for our RL post-training framework.**
+
+---
+
+## 4. Infra Requirements for Running at Home / Small Cluster
+
+### What It Takes: Minimum Viable DiLoCo/PRIME-RL Setup
+
+**For DiLoCo-style pre-training (e.g., INTELLECT-1 scale):**
+
+| Component | Minimum | Recommended |
+|-----------|---------|-------------|
+| GPUs per worker | 1× 24GB (3090/4090) | 4-8× H100/A100 per worker |
+| Number of workers | 2 | 4-8 |
+| Inter-worker bandwidth | 100 Mbps | 1 Gbps+ |
+| RAM per worker | 64 GB | 256 GB (for CPU offloading) |
+| Disk per worker | 500 GB NVMe | 2 TB NVMe |
+| Software | Hivemind + OpenDiLoCo | Prime framework (ElasticDeviceMesh) |
+
+**For PRIME-RL style RL post-training:**
+
+| Component | Minimum | Recommended |
+|-----------|---------|-------------|
+| Trainer GPU | 1× 48GB (A6000) | 1× 8×H100 node |
+| Inference GPU | 1× 24GB (3090) | 2-4× GPUs with vLLM |
+| CPU node | Any modern CPU | Orchestrator runs on CPU only |
+| Weight broadcast | Simple HTTP file server | SHARDCAST or NCCL broadcast |
+| Verification | Trusted workers (no TOPLOC needed) | TOPLOC for permissionless workers |
+| Data buffer | Simple in-memory queue | Online difficulty filtering |
+| Environment | Single verifiers env | Multiple envs from Environments Hub |
+
+### GPU Heterogeneity Tolerance
+
+**DiLoCo variants handle heterogeneity well:**
+- Async DiLoCo with Dynamic Local Updates (DyLU): Workers take H steps proportional to their speed. 3090 might take H=50 while H100 takes H=200. Empirically robust to 4× speed differences.
+- Standard DiLoCo: Straggler problem — all workers wait for slowest. **Not recommended for mixed hardware.**
+- Streaming DiLoCo: Better tolerance since communication is continuous, but still synchronous.
+- PRIME-RL: Trainer and inference are **separate pools** — inference workers can be heterogeneous (vLLM auto-scales to available compute). Trainer is typically homogeneous.
+
+**Recommendation for mixed 3090/4090/H100:** Use PRIME-RL's architecture. Put the trainer on the best GPU(s), use all available GPUs for inference. Async off-policy training naturally handles speed differences between inference workers.
+
+### Practical Libraries
+
+1. **Hivemind** ([github.com/learning-at-home/hivemind](https://github.com/learning-at-home/hivemind)) — P2P decentralized training. libp2p DHT for peer discovery, decentralized all-reduce. Used by OpenDiLoCo. Actively maintained.
+
+2. **Prime / ZeroBand** ([github.com/PrimeIntellect-ai/Prime](https://github.com/PrimeIntellect-ai/Prime)) — Prime Intellect's framework with ElasticDeviceMesh, async checkpointing, int8 all-reduce kernel, VPN mesh. Production-grade but more complex.
+
+3. **PRIME-RL** ([github.com/PrimeIntellect-ai/prime-rl](https://github.com/PrimeIntellect-ai/prime-rl)) — RL framework with orchestrator + FSDP trainer + vLLM inference. The go-to for distributed RL.
+
+4. **DisTrO** ([github.com/NousResearch/DisTrO](https://github.com/NousResearch/DisTrO)) — Drop-in distributed optimizer with DCT compression. Works with standard PyTorch training loops.
+
+5. **OpenRLHF** ([github.com/OpenRLHF/OpenRLHF](https://github.com/OpenRLHF/OpenRLHF)) — Ray + vLLM distributed RLHF. Decoupled Actor, Reward, Reference, Critic across GPUs. Not DiLoCo-based but well-established RL infrastructure.
+
+6. **veRL** ([github.com/volcengine/verl](https://github.com/volcengine/verl)) — Volcano Engine's RLHF framework. Hybrid engine design. 80-90% of training time is rollout generation. Not designed for geo-distribution.
+
+---
+
+## 5. Recommendation for Our Framework
+
+### Summary Assessment
+
+| Criterion | Best Option | Rationale |
+|-----------|------------|-----------|
+| **RL-native design** | PRIME-RL | Purpose-built for distributed RL, not adapted from pre-training |
+| **Async by default** | PRIME-RL | Off-policy training at async_level=1-2, natural fit for RL rollout cycles |
+| **Modularity** | PRIME-RL | Orchestrator/Trainer/Inference separation, verifiers environments |
+| **Small cluster friendliness** | PRIME-RL or OpenRLHF | Both run on single-node or small multi-node setups |
+| **Internet-scale distribution** | PRIME-RL + TOPLOC | Only framework with trustless verification for permissionless workers |
+| **Communication efficiency** | PRIME-RL + SHARDCAST | Weight broadcast is the relevant metric for RL, not pseudo-gradient sync |
+| **Ecosystem maturity** | OpenRLHF | Most established, but not built for geo-distribution |
+| **Heterogeneous hardware** | Async DiLoCo + PRIME-RL | DyLU for pre-training, separate inference pool for RL |
+
+### Recommended Architecture
+
+**Primary: PRIME-RL as the RL substrate, with optional DiLoCo-style outer-loop for the trainer itself if multi-node training is needed.**
+
+```
+┌──────────────────────────────────────────────────┐
+│                  Orchestrator (CPU)               │
+│  - Schedules rollouts                             │
+│  - Manages data buffer (difficulty filtering)     │
+│  - Relays weights trainer → inference             │
+│  - Assembles training batches                     │
+└──────┬───────────────────────────────┬───────────┘
+       │                               │
+       ▼                               ▼
+┌──────────────┐              ┌─────────────────────┐
+│   Trainer     │              │  Inference Pool      │
+│  (FSDP2/DiLoCo│              │  (vLLM, commodity   │
+│   if multi-GPU)│             │   GPUs, heterogeneous)│
+│               │              │                      │
+│  Inner: AdamW  │             │  /v1/chat/completions │
+│  Outer: opt.   │             │  /update_weights     │
+└──────────────┘              └─────────────────────┘
+```
+
+**Why not pure DiLoCo for RL:** DiLoCo is designed for pre-training where all workers do the same thing (forward+backward). RL has fundamentally different worker roles (inference vs training). PRIME-RL already handles this with its orchestrator architecture. Adding DiLoCo-style outer-loop would only be relevant if we need to distribute the **trainer itself** across multiple nodes — which is unlikely for hobbyist/small-cluster scales.
+
+**When to add DiLoCo:** If the trainer itself needs to run across multiple machines (e.g., model too large for one GPU, or want to aggregate training across multiple contributor nodes), wrap the trainer with OpenDiLoCo or Async DiLoCo. The inference pool stays as-is (vLLM with weight broadcast).
+
+**When to add DisTrO:** If we need per-step gradient synchronization WITHIN the trainer (e.g., multiple GPUs doing FSDP), DisTrO's DCT compression can reduce the intra-node communication overhead 857×. This is complementary to PRIME-RL's trainer↔inference communication.
+
+### Start Simple, Scale Up
+
+1. **Phase 1:** Single-node PRIME-RL with trainer and inference on same machine (or two machines on LAN)
+2. **Phase 2:** Add more inference workers on commodity GPUs
+3. **Phase 3:** If trainer needs multi-node → add OpenDiLoCo outer loop
+4. **Phase 4:** If going permissionless/crowd-sourced → add TOPLOC verification
+
+---
+
+## 6. Sources
+
+### Primary Papers
+- [DiLoCo: Distributed Low-Communication Training of Language Models](https://arxiv.org/abs/2311.08105) — Douillard et al., DeepMind, Nov 2023
+- [OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training](https://arxiv.org/abs/2407.07852) — Jaghouar et al., Prime Intellect, Jul 2024
+- [Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch](https://arxiv.org/abs/2501.18512) — Douillard et al., DeepMind, Jan 2025
+- [Asynchronous Local-SGD Training for Language Modeling](https://arxiv.org/abs/2401.09135) — Douillard et al., Jan 2024
+- [INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning](https://arxiv.org/abs/2505.07291) — Senghaas et al., Prime Intellect, May 2025
+- [PRIME-RL: Async & Decentralized RL Training at Scale](https://openreview.net/pdf?id=yk3ICpEbv8) — Senghaas et al., NeurIPS 2025
+- [HALoS: Hierarchical Asynchronous Local SGD over Slow Networks](https://arxiv.org/abs/2506.04531) — ICML 2025
+- [DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster](https://arxiv.org/abs/2506.21263) — 2025
+- [Eager Updates For Overlapped Communication and Computation in DiLoCo](https://arxiv.org/abs/2502.12996) — Feb 2025
+- [Drop-In Staleness-Aware Outer Optimization for Decoupled DiLoCo](https://arxiv.org/abs/2605.09126) — May 2025
+
+### Blog Posts & Announcements
+- [Decoupled DiLoCo: Resilient, Distributed AI Training at Scale](https://deepmind.google/blog/decoupled-diloco) — Google DeepMind, 2025
+- [OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training](https://www.primeintellect.ai/blog/opendiloco) — Prime Intellect, Jul 2024
+- [INTELLECT-1: Launching the First Globally-Distributed Training of a 10B Parameter Model](https://www.primeintellect.ai/blog/intellect-1) — Prime Intellect, Oct 2024
+- [INTELLECT-2: The First Globally Distributed Reinforcement Learning Training of a 32B Parameter Model](https://www.primeintellect.ai/blog/intellect-2) — Prime Intellect, 2025
+- [INTELLECT-2 Release: The First 32B Parameter Model Trained Through Globally Distributed Reinforcement Learning](https://www.primeintellect.ai/blog/intellect-2-release) — Prime Intellect, 2025
+- [INTELLECT-1 Release: The First Globally Trained 10B Parameter Model](https://www.lesswrong.com/posts/9cuJaJjDuhbpTid3Q/intellect-1-release-the-first-globally-trained-10b-parameter) — LessWrong analysis
+- ["This could change everything!" Nous Research unveils DisTrO](https://venturebeat.com/ai/this-could-change-everything-nous-research-unveils-new-tool-to-train-powerful-ai-models-with-10000x-efficiency) — VentureBeat, 2024
+
+### Code Repositories
+- [PrimeIntellect-ai/OpenDiLoCo](https://github.com/PrimeIntellect-ai/OpenDiloco) — OpenDiLoCo framework
+- [PrimeIntellect-ai/Prime](https://github.com/PrimeIntellect-ai/Prime) — Prime distributed training framework (formerly ZeroBand)
+- [PrimeIntellect-ai/prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) — PRIME-RL framework
+- [NousResearch/DisTrO](https://github.com/NousResearch/DisTrO) — DisTrO distributed optimizer
+- [learning-at-home/hivemind](https://github.com/learning-at-home/hivemind) — Hivemind decentralized training library
+- [OpenRLHF/OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) — Ray + vLLM distributed RLHF
+
+### Analysis & Commentary
+- [Local SGD and DiLoCo Research Musings](https://nathan.rs/posts/research-log) — Nathan (comprehensive overview with heterogeneous worker analysis), Oct 2025
+- [OpenDiLoCo and Distributed Training](https://drli.blog/posts/opendiloco-distributed-training) — Dr. Robert Li
+- [Anatomy of RL Frameworks](https://www.hanifleo.com/anatomy-of-rl-frameworks) — Hanif Leoputera (OpenRLHF vs VERL vs Slime vs Verifiers vs AReaL comparison)

research/03-monarch-torchforge-openenv.md

@@ -0,0 +1,195 @@
+# Monarch + TorchForge + OpenEnv for RL Post-Training
+
+This note surveys Meta’s PyTorch-native post‑training stack—Monarch (distributed actor framework), TorchForge (RL post‑training library), and OpenEnv (open environment standard)—with a focus on applicability to a production RL post-training pipeline. It covers what each component provides, how they compose, production maturity, and a comparison with VeRL, TRL, and OpenRLHF. 
+
+## Stack overview
+
+- Monarch (pytorch/monarch): A single‑controller, mesh‑centric distributed programming framework for PyTorch. Exposes the cluster (hosts→processes→actors) as programmable arrays with fast actor messaging, RDMA data plane, and distributed tensors. It targets heterogeneous, asynchronous ML workflows (e.g., RL post‑training) where orchestration logic is awkward in SPMD-only models. [Docs] [Blog]
+- TorchForge (meta-pytorch/forge): A PyTorch‑native RL post‑training library built on Monarch. Provides service/actor abstractions for common RL components (generator/inference, trainer/learner, rewarders, stores) and ships reference recipes (SFT, GRPO). Integrates with vLLM for rollout generation, TorchTitan for training, and TorchStore for fast weight/tensor exchange. Note: as of 2026, the repo states development is paused and LLM training is consolidating into TorchTitan. [Repo] [Blog]
+- OpenEnv (meta-pytorch/OpenEnv + Hub): A standard and hub for agentic/RL environments—typed reset/step/close API, WebSocket transport, Dockerized isolation, and MCP (Model Context Protocol) tool-calling integration. Environments publish to a Hugging Face Hub collection; trainers (TRL, TorchForge, VeRL, Unsloth) consume via a stable client without per‑env adapters. [HF blog] [Spec/RFCs] [Repo] [TRL guide]
+
+Where this shines:
+- A coherent PyTorch‑native control plane (Monarch) plus post‑training orchestration (Forge) and a portable environment substrate (OpenEnv) reduce glue code and make async, tool‑using RL feasible at scale.
+- Monarch’s separation of control plane (message passing, supervision) and data plane (RDMA buffers, distributed tensors) is well aligned with disaggregated RL stacks—high‑throughput inference and weight sync paths can be optimized independently of controller logic.
+
+## Monarch deep‑dive
+
+What it is
+- Single‑controller model: you write one Python program (the controller) that orchestrates distributed resources directly; the cluster is exposed as structured meshes you can slice/index like arrays.
+- Key abstractions:
+  - ProcessMesh: an array of processes (often 1 proc/GPU) across hosts.
+  - ActorMesh: a collection of stateful actors spawned onto the process mesh; vectorized messaging across all/slices.
+  - RDMA buffers and data plane: register any CPU/GPU memory and perform one‑sided transfers (libibverbs) for zero‑copy paths; integrates with distributed tensor operations.
+  - Distributed tensors: PyTorch‑native DTensor integration so actors operate on sharded tensors that "feel" local.
+  - Supervision trees: fault handling modeled after actor systems; fail‑fast by default with opt‑in, scoped recovery.
+  - Lower‑level runtime: hyperactor (Rust) underpins actor messaging and supervision; hyperactor_mesh provides vectorized actor operations.
+- Environments: local dev server, multi‑GPU nodes, Kubernetes jobs (via monarch‑kubernetes), and HPC clusters.
+
+Why not Ray Actors?
+- Ray is a general distributed runtime (actors, tasks, object store, autoscaler) across many domains. Monarch is PyTorch‑native and oriented around meshes of processes/actors with tight integration to DTensor and an explicit RDMA data plane.
+- Programming model: Monarch’s "program clusters like arrays" meshes and single‑controller orchestration feel like NumPy/PyTorch over clusters; Ray’s API is broader but less tensor/mesh‑centric.
+- Data movement: Monarch explicitly separates a lightweight control plane from a high‑performance data plane (RDMA, direct GPU‑GPU); Ray relies on its object store and networking stack.
+- Fit for post‑training: RL pipelines often need orchestrated SPMD components (trainer FSDP, inference TP/PP, rewarders) plus asynchronous control; Monarch’s controller + meshes model this cleanly. 
+
+Evidence and references
+- Intro + model: "Introducing PyTorch Monarch" (2025‑10‑22), "Monarch: an API to your supercomputer" (2026‑04‑08) and v0.5 docs detail ProcessMesh/ActorMesh, supervision, and RDMA/data‑plane separation.
+- Activity: ~1k stars, active releases through v0.4/v0.5, K8s support added, many contributors; used for agentic development, telemetry (distributed SQL), and even as a VeRL backend in validation experiments.
+
+Caveats
+- Monarch is powerful but new; while the programming model is minimal, you still craft orchestration code. Higher‑level libraries (Forge) reduce that, but Forge’s development pause (see next section) is relevant.
+
+## TorchForge recipes & API
+
+What Forge ships
+- Purpose: "focus on algorithms, not infra"—service abstractions built on Monarch actors.
+- Reference recipes:
+  - SFT quickstart (Llama/Qwen variants).
+  - GRPO end‑to‑end (Qwen3 1.7B/8B/32B reference configs), including multi‑node scale demos.
+- Architecture and integrations:
+  - Generator (policy inference): vLLM‑backed service/actors for high‑throughput autoregressive generation; can run as colocated services or as external vLLM servers.
+  - Trainer/Learner: Trainer actors running on TorchTitan (FSDP, TP/PP/CP) to update weights; supports async and synchronous coordination patterns.
+  - Rewarders: reward model services/actors; Forge blogs highlight RLVR setups with Weaver‑style verifiers as drop‑in reward sources.
+  - TorchStore: RDMA‑accelerated tensor/weight exchange to keep generators near‑on‑policy (direct GPU‑GPU state_dict transfers; resharding support).
+  - OpenEnv: environments are consumed via a standard client; tool‑calling environments (MCP) supported through OpenEnv, not bespoke adapters.
+
+Developer experience
+- Single config and Python entrypoint spin up a job where the controller orchestrates Generator/Trainer/Rewarders as ActorMeshes.
+- Service abstractions manage:
+  - Spawning/placement across nodes
+  - Load balancing and routing
+  - Fault tolerance and retries
+- Explicit toggles for synchronicity (sync PPO‑like loops ↔ fully async off‑policy) without rewriting rollout logic.
+
+What’s included vs missing
+- Included out of the box: SFT, GRPO; end‑to‑end RLVR demo with Weaver (verifier ensemble) at 512‑GPU scale in the blog; vLLM integration; TorchTitan trainer; TorchStore weight sync.
+- Not first‑class in the public materials: built‑in DPO/PPO/ORPO recipes (though PPO‑like sync is described conceptually), SGLang integration (VeRL supports SGLang, Forge highlights vLLM), or an extensive cookbook (tutorials "coming soon" in docs). 
+
+Status and activity
+- Repo banner: "Development paused—LLM training consolidating in TorchTitan." Last pushes in 2026, ~685 stars, 100+ open issues; examples and CI present.
+- Takeaway: Useful as a reference of patterns built on Monarch and TorchStore; for greenfield, plan to lean more on TorchTitan (training core) + OpenEnv + TRL/VeRL for algorithm coverage.
+
+## OpenEnv protocol
+
+Core idea
+- OpenEnv standardizes how agents/trainers interact with real or simulated environments using a typed, Gymnasium‑like API: reset(), step(), close(), state(). Observations/actions are schemas (dataclasses), enabling type safety and IDE support.
+- Transport and isolation: environments run as servers—WebSocket is the default (supports many concurrent sessions per container); HTTP control plane exists for orchestration; Dockerized packaging for reproducibility and sandboxing.
+- MCP integration: RFC‑003 maps MCP tool list/call to OpenEnv actions so environments can expose tools via the Model Context Protocol. This supports tool‑calling agents and ML trainers with the same environment surface.
+
+Hub and publishing flow
+- Authors publish an environment (Docker image + Python client) to the Hugging Face OpenEnv Hub. Users:
+  - Inspect tools, schemas, and try environments as a Human Agent in‑browser.
+  - Connect trainers (TRL, TorchForge, VeRL, Unsloth) by referencing the Hub ID—no per‑env adapters.
+- Scaling: documented patterns and benchmarks show 100s to 10Ks of concurrent sessions by switching from HTTP (1:1 session/container) to WebSocket multiplexing and scaling containers behind Envoy.
+
+Tool‑calling, async, and harnesses
+- MCP tools are exposed safely alongside the environment’s RL API, with reserved name checks (not allowing reset/step/state as tools) to preserve orchestration boundaries.
+- RFC‑005 adds “agentic harness” integration: some envs wrap a full agent harness (e.g., OpenClaw). Production endpoints stream harness events; training keeps episode control by mapping turns to step() transitions.
+
+Adoption signals
+- HF launch blog (Meta × HF) with examples; TRL has a first‑party OpenEnv integration guide; OpenEnv repo ~1.5–2k stars, active RFCs and releases; third‑party writeups (e.g., Turing’s calendar environment) and community envs (games, coding, REPL, web nav) on the Hub.
+
+## The combined pipeline (Monarch + TorchForge + OpenEnv)
+
+A canonical post‑training topology looks like:
+- Controller: Monarch single Python program orchestrating meshes.
+- Generator service (ActorMesh): vLLM‑backed policy inference over prompts from datasets or environments; can be colocated or external microservices.
+- Environments: OpenEnv servers (Dockerized, WebSocket) providing tool‑using or simulator environments. Generators interact via OpenEnv client; for tool‑calling flows, the same environment exposes MCP list/call mapped to actions.
+- Rewarders: reward model(s) or verifiers (e.g., Weaver) as services. Reward functions can be synchronous or delayed (RFC‑004 delayed rewards).
+- Trainer (ActorMesh): TorchTitan‑powered learner updating the policy (FSDP/TP/PP/CP as needed). 
+- Weight/tensor sync: TorchStore for state_dict exchange and DTensor‑aware resharding; Monarch RDMA paths provide direct GPU‑to‑GPU sync to reduce iteration latency.
+
+Operational considerations
+- Synchronicity: pattern toggles between sync PPO‑style loops (tighter on‑policy, lower throughput) and async off‑policy (higher throughput, some staleness). Forge surfaces this without reworking rollout code.
+- Inference plane: vLLM usually runs as separate pods, discoverable by the controller; can also run in‑process for small scales.
+- Reward serving: either colocated fast RMs (transformers classification heads) or verifier ensembles (e.g., Weaver) via RPC. Monarch meshes and services route traffic intelligently.
+- Telemetry: Monarch integrates a distributed SQL telemetry plane for introspection across actors (useful in debugging coordination pathologies—queue depth, policy staleness, etc.).
+
+Reference: PyTorch blog post shows Forge + Weaver at 512‑GPU scale for RLVR, with Monarch handling coordination and TorchStore accelerating weight sync.
+
+## Comparison vs VeRL / TRL / OpenRLHF
+
+Criteria and synthesis
+- Programming model
+  - Monarch + Forge: single‑controller, actor/mesh orchestration in Python; services abstract placement, retries, routing. Tight PyTorch/DTensor/RDMA integration.
+  - VeRL (HybridFlow): hybrid model—single‑controller logic with multi‑controller efficiency; built on Ray but exposes a clean single‑controller interface; can run with vLLM/SGLang. Mature production framing; strong community and docs.
+  - TRL: library‑first, Trainer APIs (GRPO, PPO [experimental], Online DPO, DPO, Reward modeling, SFT). Integrates with vLLM; now has OpenEnv integration to drive stateful envs via environment_factory. Minimal infra; you supply the orchestration.
+  - OpenRLHF: PPO‑style RLHF focus; strong PPO pipelines and examples; less emphasis on stateful, tool‑using environments; infra glue typically on users.
+
+- Algorithm coverage
+  - Forge: SFT + GRPO references; PPO described as synchronization pattern but not a first‑class shipped recipe; no built‑in DPO/Online DPO.
+  - VeRL: PPO/GRPO and more; productionized alignment/TRL variants; broader set of recipes; integrates RMs and multiple inference engines.
+  - TRL: very broad—SFT, GRPO, PPO (exp), Online DPO, DPO, reward modeling, etc. 
+  - OpenRLHF: strong PPO RLHF, some preference‑optimization variants via community forks.
+
+- Environment integration
+  - Forge: consumes OpenEnv environments; tool‑calling via MCP thanks to OpenEnv; demoed with coding sandbox and others.
+  - VeRL: OpenEnv‑compatible (via Hub clients) and has its own env adapters historically; strong ecosystem around vLLM/SGLang rollouts.
+  - TRL: first‑party OpenEnv integration guide with GRPOTrainer; clean developer UX.
+  - OpenRLHF: generally Gym/Gymnasium‑style or custom envs; can use OpenEnv with adapters but not first‑party yet.
+
+- Scale ceiling and performance
+  - Monarch + Forge: RDMA data plane + TorchStore for zero‑copy weight/tensor sync; meshes support thousands of GPUs; validated RLVR at 512 GPUs. 
+  - VeRL: proven scale; Ray scheduling maturity; broad industry adopters and talks; benchmark claims of high throughput; supports vLLM/SGLang/in‑proc HF.
+  - TRL: depends on your training backend (Deepspeed, Titan, PEFT) and rollout engine (vLLM). Good scaling stories but orchestration is user‑owned.
+  - OpenRLHF: similar—performance comes from chosen backends; less built‑in orchestration.
+
+- Production readiness
+  - Monarch: active development, releases, docs—credible but new; requires engineering buy‑in to its model.
+  - Forge: marked "development paused; consolidating into TorchTitan"—use patterns as reference; expect Titan + TRL/VeRL for go‑forward.
+  - OpenEnv: fast‑moving but already widely referenced (HF blog, TRL integration, RFCs, Hub adoption). Clear isolation + transport story; scaling guides published.
+  - VeRL: strong community traction and ecosystem of integrations; production‑minded design (HybridFlow); multi‑engine support.
+  - TRL: de‑facto OSS standard for post‑training algorithms; v1 emphasizes robustness; extensive examples and docs.
+  - OpenRLHF: widely used for PPO RLHF; simpler but narrower API.
+
+## Fit for our framework
+
+- Using Monarch as the control substrate: Feasible and attractive if we want a single‑controller Python program to coordinate learners, generators, rewarders, and environment clients with strong fault handling and a high‑performance data plane. Monarch does not conflict with our gradient synchronization method (e.g., DiLoCo/local‑SGD)—those live inside the trainer (TorchTitan/Deepspeed/etc.). Monarch sits above as orchestration.
+- TorchForge as the RL layer: Good as a pattern reference for service abstractions, but given the "development paused" status, we should not bet on Forge as a moving foundation. Instead:
+  - Prefer TorchTitan for the training core (supports FSDP/TP/PP/CP and context parallelism),
+  - Pair with TRL or VeRL for algorithm coverage (GRPO, PPO, DPO, reward modeling),
+  - Keep OpenEnv as the environment substrate,
+  - Re‑implement needed Forge‑like services (generator/rewarder/store) using Monarch where it adds value, or start with VeRL’s backend and migrate selectively.
+- DiLoCo compatibility: No inherent conflict. DiLoCo controls intra‑trainer gradient sync; Monarch/VeRL/TRL govern inter‑component orchestration. If we keep Titan + DiLoCo inside the learner and use Monarch to coordinate rollout and envs, they are complementary.
+- Inference engine: vLLM is first‑class across Forge/VeRL/TRL; SGLang is supported in VeRL; nothing prevents adding SGLang actor services in a Monarch stack if desired.
+
+Recommended adoption path
+1) Standardize environments via OpenEnv (use Hub IDs in all experiments). 
+2) Choose training core: TorchTitan (preferred) or Deepspeed, and decide on algorithm library: TRL for breadth or VeRL for a production‑oriented RL stack with hybrid single‑controller flavor. 
+3) Use vLLM rollouts as an external service initially; add Monarch‑managed generator/rewarder services only if we need advanced placement/fault semantics or RDMA‑accelerated weight sync with TorchStore. 
+4) If we want Monarch, adopt it incrementally—start by running Titan trainers under Monarch Job API + ActorMeshes for rollout and rewarders; keep algorithm logic in TRL/VeRL.
+
+## Open questions we should validate
+- Monarch vs Ray swap‑costs in downstream libraries: PyTorch notes that even when a framework exposes a clean single‑controller interface, Ray API usage may surface elsewhere—how invasive is a Monarch backend in VeRL/TRL codepaths we care about?
+- Weight freshness vs throughput: With TorchStore + RDMA, what iteration times do we achieve for 7B/32B policies at 8–32 generators? What update cadence avoids excessive off‑policyness while keeping generators saturated?
+- Reward serving patterns: For verifier‑heavy tasks (math/code), what is the optimal topology—RM colocated per generator vs shared verifiers; how do we saturate them without becoming the bottleneck?
+- Environment scaling: For target benchmarks (e.g., web nav + coding), can we reach 5–10k concurrent env sessions using the documented WebSocket multiplexing + Envoy patterns; does the Hub infra suffice or do we need cluster‑native deployments from day one?
+- Telemetry and observability: Monarch’s distributed SQL telemetry sounds promising; do we integrate this or rely on W&B + Prometheus? How painful is cross‑actor correlation in practice?
+
+## Sources
+- Monarch
+  - Introducing PyTorch Monarch (2025‑10‑22): https://pytorch.org/blog/introducing-pytorch-monarch/
+  - Monarch: an API to your supercomputer (2026‑04‑08): https://pytorch.org/blog/monarch-an-api-to-your-supercomputer/
+  - Monarch docs: https://meta-pytorch.org/monarch/
+  - Repo: https://github.com/meta-pytorch/monarch (stars/releases/activity in repo)
+- TorchForge
+  - Repo (banner: development paused): https://github.com/meta-pytorch/forge
+  - Introducing torchforge (PyTorch blog): https://pytorch.org/blog/introducing-torchforge/
+  - Supercharging LLMs: Scalable RL with torchforge and Weaver: https://pytorch.org/blog/supercharging-llms-scalable-rl-with-torchforge-and-weaver/
+  - TorchStore (RDMA tensor/weights): https://github.com/meta-pytorch/torchstore
+- OpenEnv
+  - HF launch blog: https://huggingface.co/blog/openenv
+  - TRL OpenEnv integration: https://huggingface.co/docs/trl/en/openenv
+  - OpenEnv repo + RFCs (MCP, delayed rewards, harness): https://github.com/meta-pytorch/OpenEnv and https://github.com/meta-pytorch/OpenEnv/blob/main/rfcs/003-mcp-support.md
+  - OpenEnv Hub: https://huggingface.co/openenv
+  - Scaling OpenEnv (community post): https://huggingface.co/blog/burtenshaw/openenv-scaling
+  - OpenEnv in practice (Turing): https://huggingface.co/blog/openenv-turing
+- Alternatives
+  - VeRL: https://github.com/verl-project/verl
+  - TRL (features incl. GRPO/PPO/DPO/Online DPO): https://huggingface.co/docs/trl/en/index
+
+---
+
+Appendix: quick status snapshot (as of May 2026; see linked pages for live numbers)
+- Monarch: ~1k stars, v0.4/v0.5 docs, K8s support, active commits through Apr 2026.
+- TorchForge: ~685 stars, last pushes in 2026, readme notes development paused (consolidate in TorchTitan).
+- OpenEnv: ~1.5–2k stars, active RFCs (MCP, delayed rewards, harnesses), v0.3.0 released May 2026, HF Hub org and catalog live.

research/04-verl-trl.md

@@ -0,0 +1,421 @@
+# VeRL vs. HF TRL — Deep-Dive Comparison Report
+
+> **Generated:** 2026-05-25  
+> **Scope:** Post-training framework selection for a "take any HF model, RL post-train it" goal, with particular focus on agentic-coding use-cases.
+
+---
+
+## Table of Contents
+
+1. [VeRL Deep-Dive](#1-verl-deep-dive)
+2. [TRL Deep-Dive](#2-trl-deep-dive)
+3. [Algorithm Zoo — Current State of RL for LLMs (Late 2025)](#3-algorithm-zoo)
+4. [Comparison Matrix](#4-comparison-matrix)
+5. [Recommendation](#5-recommendation)
+6. [Sources](#6-sources)
+
+---
+
+## 1. VeRL Deep-Dive
+
+### 1.1 Overview
+
+VeRL (**Volcano Engine Reinforcement Learning**) is ByteDance's production-grade, open-source RL training library for LLMs. Released publicly in 2024, it is the framework that powered DeepSeek-R1-style large-scale RL post-training runs and Qwen RL post-training. The headline paper is *HybridFlow* (Sheng et al., 2025), which formalises the underlying architecture.
+
+> **GitHub:** https://github.com/volcengine/verl  
+> **Stars:** >10 k (as of mid-2025)
+
+### 1.2 Architecture — HybridFlow
+
+VeRL's core design principle is the **HybridFlow** programming model, which decouples the RL *control plane* from the *compute plane*:
+
+- **Single-Controller Orchestration:** A central `RayPPOTrainer` (Ray-based) coordinates all distributed workers. The controller treats the cluster as a set of remote high-level operators, making it easy to compose new algorithms.
+- **Computation-Data Decoupling:** Workers execute independently and exchange state via `DataProto` objects, making computation flow reusable across different RL algorithms without re-implementation.
+- **3D-HybridEngine:** A single worker can switch between *training mode* and *inference/rollout mode*, eliminating redundant model copies. During PPO/GRPO, the Actor is used for both generation and gradient updates via efficient resharding (e.g., FSDP sharded ↔ vLLM TP). This is the key memory efficiency win.
+- **Flexible Resource Allocation:** Models can be colocated on the same GPU set, placed on separate GPU sets, or run in a hybrid configuration, enabling optimal hardware utilisation at scale.
+
+### 1.3 Training Backends
+
+| Layer | Options |
+|---|---|
+| **Distributed training** | FSDP / FSDP2 (research-friendly), Megatron-LM v0.13.1+ (production scale), MindSpeed-LLM (Ascend NPU) |
+| **Rollout / inference** | vLLM (≥0.8.3), SGLang (fully supported, multi-node), TensorRT-LLM, HF Transformers (debug only) |
+| **Hardware** | NVIDIA H100/A100, AMD, Ascend 910 |
+| **Orchestration** | Ray (required) |
+
+**Key insight:** VeRL treats the training engine and rollout engine as separable components. The `3D-HybridEngine` handles weight resharding between FSDP sharding patterns (needed for training) and Tensor-Parallel patterns (needed for vLLM/SGLang generation), without maintaining duplicate model copies.
+
+### 1.4 Algorithm Zoo in VeRL
+
+VeRL ships first-class implementations of:
+
+| Algorithm | Status | Notes |
+|---|---|---|
+| **PPO** | Stable | Actor + Critic + Reference + Reward model; full pipeline |
+| **GRPO** | Stable | Critic-free; group-relative advantages |
+| **DAPO** | Stable | Decoupled clip + dynamic sampling + token-level PG loss |
+| **RLOO** | Stable | REINFORCE Leave-One-Out; no critic |
+| **ReMax** | Stable | Greedy baseline; no critic |
+| **REINFORCE++** | Stable | Batch-global baseline with clipping |
+| **SPIN** | Stable | Self-play via online DPO loss |
+| **SPPO** | Stable | Self-play preference optimisation |
+| **GPG** | Stable | Policy gradient variant for math/reasoning |
+| **OTB** | Stable | Optimal Token Baseline for fine-grained credit |
+| **SAPO** | Community | Smoothing-based actor-policy optimisation |
+| **GSPO** | Community | Grouped Soft Policy Optimisation (sequence-level) |
+| **DPO / Online DPO** | Supported | Via SPIN / DAPO extensions |
+
+### 1.5 Agentic / Tool-Calling RL
+
+VeRL has **first-class agentic RL support**:
+
+- **AsyncServer / AgentLoop architecture:** An `asyncio`-based co-routine mechanism separates the `AgentLoop` (client that drives multi-turn trajectories) from the `AsyncServer` (vLLM/SGLang inference backend). During tool-call waits (e.g., code execution), GPU compute is not blocked — other inflight requests continue.
+- **SandboxFusionTool:** Built-in code-execution sandbox for agentic coding tasks; allows model → `<tool_call>` → sandbox response → next step trajectories with rewards assigned at trajectory end.
+- **Multi-turn tokenisation:** Supported but noted as complex; naive concatenation of per-turn token IDs can introduce distribution drift between the rollout policy and training policy.
+
+### 1.6 Scale
+
+| Tested configuration | Notes |
+|---|---|
+| Up to **671B parameters** | Confirmed in production (DeepSeek-scale) |
+| **Trillion-parameter** GRPO | 64 H800 GPUs; GRPO with Megatron-LM backend |
+| **8× H100 benchmark** | DeepSeek-R1-Distill-Qwen-1.5B, 28k context, batch 128 per DP: step time ~363s; gen throughput measured per-GPU |
+
+A third-party benchmark (RLinf docs, Aug 2025) running VeRL v0.5.0 on 8× H100s with a 1.5B model (context 28,672 tokens):
+
+- **Generation time:** 260.9 s/step  
+- **Training time:** 66.5 s/step  
+- **Total step time:** 363.6 s/step
+
+VeRL's Megatron-LM backend + SGLang rollout is the performance-optimal path for >70B models.
+
+### 1.7 Real-World Usage
+
+- **DeepSeek-R1 lineage** — The architecture is directly inspired by DeepSeek's internal RLVR pipeline.
+- **Qwen RL post-training** — Qwen3 and DAPO paper both used VeRL.
+- **DAPO paper** (ByteDance, 2025) — Trained Qwen2.5-72B with VeRL; achieved new AIME 2024 SOTA.
+- **Multiple open reproductions** of DeepSeek-R1-Zero use VeRL as the training backend.
+
+### 1.8 Strengths
+
+1. **Best-in-class throughput at scale** — 3D-HybridEngine + vLLM/SGLang eliminates memory redundancy.
+2. **Widest algorithm coverage** — PPO through the latest DAPO/GSPO/OTB variants all natively supported.
+3. **Production proven** — Used at 671B scale with Megatron-LM.
+4. **First-class agentic loops** — AsyncServer decouples GPU from tool-call latency.
+5. **Hardware agnostic** — NVIDIA, AMD, Ascend.
+6. **Flexible resource allocation** — Colocated, separated, or hybrid GPU pooling.
+
+### 1.9 Weaknesses / Challenges
+
+1. **Steep learning curve** — Ray orchestration, multiple backend configs, FSDP vs. Megatron choice; not a 3-line quickstart.
+2. **Multi-turn tokenisation complexity** — Risk of subtle off-policy drift if multi-turn chat templates are not handled carefully; noted as an active known issue.
+3. **Off-policy instability** — Rollout correction is provided but requires careful tuning; naive replay buffers can cause policy collapse.
+4. **Heavyweight infrastructure** — Requires Ray cluster; not ideal for single-GPU or commodity 4-GPU experiments.
+5. **Documentation gaps** — Community recipes exist but the core docs lag behind code velocity.
+
+---
+
+## 2. TRL Deep-Dive
+
+### 2.1 Overview
+
+TRL (**Transformer Reinforcement Learning**) is Hugging Face's mainstream post-training library, designed around the HF ecosystem (Accelerate, PEFT, Transformers, Datasets). The philosophy is *accessible post-training for any HF model*, favouring simplicity and developer ergonomics over raw throughput at frontier scale.
+
+> **GitHub:** https://github.com/huggingface/trl  
+> **Version milestone:** TRL v1 released March 2026  
+> **Stars:** >14 k
+
+### 2.2 Trainer Taxonomy
+
+TRL organises trainers into four categories:
+
+#### Supervised
+| Trainer | Description |
+|---|---|
+| `SFTTrainer` | Instruction-tuning / supervised fine-tuning; supports packing, PEFT, VLMs |
+| `RewardTrainer` | Train scalar reward models from preference data |
+| `PRMTrainer` | Process Reward Model training (step-level rewards) |
+
+#### Preference / Offline Alignment
+| Trainer | Description |
+|---|---|
+| `DPOTrainer` | Direct Preference Optimisation; supports VLMs and tool-calling |
+| `BCOTrainer` | Binary Classifier Optimisation |
+| `CPOTrainer` | Contrastive Preference Optimisation |
+| `KTOTrainer` | KTO (binary signal, no pairs) |
+| `ORPOTrainer` | Odds-Ratio Preference Optimisation |
+| `GKDTrainer` | Generalised Knowledge Distillation |
+| `NashMDTrainer` | Nash Mirror Descent online preference |
+
+#### Online RL
+| Trainer | Description |
+|---|---|
+| `GRPOTrainer` | **Primary online RL trainer.** Group Relative Policy Optimisation; stable; VLM + agentic support |
+| `RLOOTrainer` | REINFORCE Leave-One-Out; supports VLMs |
+| `PPOTrainer` | Proximal Policy Optimisation; **experimental** (noted as incomplete) |
+| `OnlineDPOTrainer` | Online DPO with LLM-as-judge; **experimental** |
+| `XPOTrainer` | Exploratory DPO (experimental) |
+
+#### Other
+| Trainer | Description |
+|---|---|
+| `MiniLLMTrainer` | Reverse-KL distillation |
+
+### 2.3 GRPOTrainer — Key Design
+
+`GRPOTrainer` is TRL's workhorse for RLVR-style training:
+
+- **No critic model** — group-relative advantages, matching GRPO semantics from DeepSeek-R1.
+- **vLLM integration** — co-located vLLM for fast rollout generation (June 2025 update: "NO GPU left behind" co-located vLLM).
+- **Liger kernel integration** — May 2025 update; significant memory/speed improvements for GRPO training step.
+- **VLM support** — Vision-language models trainable with GRPO as of August 2025.
+- **Agentic workflows** — `GRPOTrainer` supports multi-step agentic rollouts; `OpenEnv` integration (October 2025) provides tool/environment loop scaffolding.
+
+### 2.4 Distributed Backends
+
+TRL relies on **HF Accelerate** as the distribution abstraction:
+
+| Backend | Support level |
+|---|---|
+| DeepSpeed ZeRO-1/2/3 | Stable |
+| FSDP v1 + v2 | Stable |
+| PEFT / LoRA / QLoRA | Native; enables large model training on fewer GPUs |
+| vLLM (co-located) | Integrated for online RL trainers (GRPO, RLOO, PPO) |
+
+### 2.5 Scale Ceiling
+
+TRL was designed for the **commodity to mid-scale cluster** range:
+
+- Single GPU (with QLoRA) up through multi-node clusters.
+- No native Megatron-LM tensor/pipeline parallelism — limits scaling for >70B full-parameter runs.
+- No 3D-HybridEngine; actor model is held fully in training-mode sharding at all times, meaning rollout generation is bottlenecked by the training sharding strategy.
+- Practical ceiling: **8–32 GPU clusters** for full-parameter runs of 7–70B models; beyond that, FSDP ZeRO-3 sharding overhead becomes limiting.
+
+### 2.6 VLM and Tool-Calling
+
+- **VLM alignment:** `SFTTrainer`, `DPOTrainer`, `GRPOTrainer`, `RLOOTrainer` all support VLMs (multimodal inputs via processor-aware collation).
+- **Tool-calling:** `DPOTrainer` and `SFTTrainer` have explicit tool-calling support (formatting/masking of tool call tokens).
+- **Agentic RL:** `GRPOTrainer` supports agentic workflows; `OpenEnv` (Oct 2025) adds an open tool-environment ecosystem. However, TRL does **not** have an async GPU-decoupled agent loop — tool-call latency stalls the training process.
+
+### 2.7 Recent 2025 Highlights
+
+| Date | Update |
+|---|---|
+| Jan 2025 | Open-R1: full DeepSeek-R1 reproduction using TRL |
+| May 2025 | Liger kernels for GRPO — major memory/speed win |
+| Jun 2025 | Co-located vLLM in TRL for online RL trainers |
+| Aug 2025 | VLM alignment support in GRPOTrainer |
+| Oct 2025 | OpenEnv: open agent environment ecosystem integration |
+| Mar 2026 | TRL v1.0 release: stable API, architectural cleanup |
+
+### 2.8 Strengths
+
+1. **Developer ergonomics** — `GRPOTrainer(model, args, train_dataset, reward_funcs=...)` — fits in <50 lines of boilerplate.
+2. **HF ecosystem native** — Any `AutoModel`, any HF dataset, any PEFT config, Weights & Biases, etc.
+3. **PEFT/QLoRA** — Train large models (30–70B) on 4-GPU commodity rigs via quantised LoRA.
+4. **Widest model coverage** — If it's on HF Hub, TRL can train it.
+5. **VLM support** — Multimodal RL post-training out of the box.
+6. **Active community** — Fast iteration; Open-R1 and dozens of community recipes.
+7. **Process Reward Model training** — `PRMTrainer` is a notable capability VeRL lacks natively.
+
+### 2.9 Weaknesses
+
+1. **Scale ceiling** — No Megatron-LM; impractical for >70B full-parameter RL at production throughput.
+2. **PPO is experimental** — The full 4-model PPO pipeline is not production-grade.
+3. **No async agent loops** — GPU blocks during tool-call execution.
+4. **Throughput gap vs. VeRL** — Without 3D-HybridEngine, memory layout switches between rollout and training are expensive.
+5. **GRPO implementation quirks** — Naive GRPO without DAPO fixes (dynamic sampling, decoupled clip) can exhibit length bias and entropy collapse; not all fixes are default-on.
+
+---
+
+## 3. Algorithm Zoo — Current State of RL for LLMs (Late 2025)
+
+The post-DeepSeek-R1 era produced an explosion of GRPO variants. Here is the taxonomy as of late 2025 / early 2026:
+
+### 3.1 The GRPO Family (critic-free, group-relative)
+
+| Algorithm | Key Innovation | Main Concern | Best For |
+|---|---|---|---|
+| **GRPO** (DeepSeek, 2024) | Group-relative advantages; no critic | Length bias; zero-signal groups; entropy collapse | Baseline for reasoning RL |
+| **DAPO** (ByteDance, 2025) | Decoupled clip (ε_low ≠ ε_high) + dynamic sampling (filter zero-signal groups) + token-level PG loss + overlong shaping | More hyperparameters; GRPO family limitations | Long-CoT reasoning; production-scale RLVR |
+| **Dr.GRPO** (Liu et al., 2025) | Removes 1/\|o_i\| length norm and σ_q std-dev division; equivalent to RLOO up to scaling | Less battle-tested | Correcting GRPO's statistical biases |
+| **REINFORCE++** (Hu, 2025) | Batch-global baseline; no per-prompt grouping | Loses prompt-local difficulty signal | Avoiding group degeneracy; simple baseline |
+| **GSPO** (Group Soft PO) | Sequence-level ratio via geometric mean; matches reward granularity | Newer; limited reproduction | Long-response MoE RL |
+| **RLOO** (Ahmadian et al., 2024) | Leave-One-Out baseline; unbiased, no critic | Requires multi-sample generation | Variance reduction without critic overhead |
+| **ReMax** | Greedy decoding as baseline | Greedy baseline may be poor for non-deterministic tasks | Low-cost critic-free training |
+
+### 3.2 Actor-Critic Methods
+
+| Algorithm | Key Feature | Status |
+|---|---|---|
+| **PPO** | Learned value function (GAE); token-level credit | Classic RLHF; high quality but expensive |
+| **StepPO** (2025) | Step-level MDP + step-level credit assignment | Frontier for agentic RL; reduces sparse reward problem |
+
+### 3.3 Off-Policy / Preference Methods
+
+| Algorithm | Key Feature |
+|---|---|
+| **DPO** | Direct preference; offline; no RM |
+| **Online DPO / SPIN / SPPO** | Self-play preference; iterative improvement |
+| **CISPO** | IS-weight clipping (not objective clipping); asymmetric bounds; off-policy |
+| **TOPR** | Sequence-level; asymmetric clipping by reward sign |
+
+### 3.4 Reward Signal Paradigms
+
+| Paradigm | Description | Use-case |
+|---|---|---|
+| **RLVR** (Rule-Verifiable Rewards) | Reward from deterministic verifier (math checker, test suite) | Coding, math, structured output |
+| **Outcome Reward Model (ORM)** | Trained RM scoring final answer | General alignment |
+| **Process Reward Model (PRM)** | Step-level rewards on reasoning trace | Long-CoT, complex reasoning |
+| **LLM-as-Judge** | Strong LLM scores outputs | Quality tasks without verifier |
+
+### 3.5 Converging Best Practices for Agentic-Coding RL
+
+Based on the 2025 literature, the community is converging toward:
+
+1. **Algorithm:** GRPO + DAPO fixes (dynamic sampling to filter zero-signal groups; decoupled clip; token-level loss) — or equivalently Dr.GRPO / REINFORCE++ for simpler implementations.
+2. **Reward signal:** RLVR with test-suite execution (verifiable) — pass@k on code tests, format rewards.
+3. **Multi-turn trajectories:** GRPO applied at trajectory level (sparse reward on final code output); StepPO-style step rewards are emerging for better credit assignment.
+4. **Cold-start:** Brief SFT on curated CoT traces before RL (DeepSeek-R1 recipe) to avoid early entropy collapse.
+5. **Context length:** Long context (16k–32k) is essential for coding; models with long context rollout support (SGLang/vLLM paged attention) are required.
+
+---
+
+## 4. Comparison Matrix
+
+### 4.1 Feature Comparison
+
+| Dimension | VeRL | TRL |
+|---|---|---|
+| **Primary abstraction** | HybridFlow dataflow graph + Ray workers | HF Trainer subclass + Accelerate |
+| **Ease of entry** | ★★☆ (complex) | ★★★★★ (simple) |
+| **Algorithm breadth** | ★★★★★ (PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, GSPO, OTB, SAPO, SPIN, SPPO, GPG) | ★★★★☆ (GRPO, RLOO, DPO variants; PPO experimental) |
+| **Max tested scale** | 671B params, 100s of GPUs | ~70B with FSDP ZeRO-3; practical ceiling ~32 GPUs full-param |
+| **Training backends** | FSDP, Megatron-LM, MindSpeed | FSDP, DeepSpeed ZeRO |
+| **Rollout backends** | vLLM, SGLang, TensorRT-LLM, HF | vLLM (co-located), HF |
+| **3D-HybridEngine** | ✅ (key differentiator) | ❌ |
+| **Async agent loop** | ✅ AsyncServer + AgentLoop | ❌ (blocking) |
+| **Agentic tool-calling RL** | ✅ (SandboxFusionTool, asyncio loop) | ⚠️ (GRPOTrainer + OpenEnv; blocking) |
+| **VLM support** | ✅ (VeOmni stack) | ✅ (GRPOTrainer, DPOTrainer) |
+| **PEFT / LoRA / QLoRA** | ⚠️ (partial; not primary use-case) | ✅ (native, core feature) |
+| **Process Reward Model** | ❌ (native) | ✅ (PRMTrainer) |
+| **HF Hub model load** | ✅ (via HF Transformers) | ✅ (native) |
+| **Hardware (non-NVIDIA)** | ✅ AMD, Ascend | ⚠️ (primarily NVIDIA; DeepSpeed has AMD support) |
+| **Production pedigree** | DeepSeek-R1, DAPO, Qwen RL | Open-R1, academic research, community |
+| **Ray requirement** | ✅ Required | ❌ Not needed |
+| **Documentation quality** | ★★★☆ | ★★★★★ |
+| **Community size** | Medium (but growing fast) | Very large |
+
+### 4.2 Throughput (Indicative)
+
+| Scenario | VeRL | TRL |
+|---|---|---|
+| 1.5B model, 8× H100, context 28k | Step time ~363s (gen: 261s + train: 66s) | No published comparable; likely 1.5–3× slower without HybridEngine |
+| 7B model, 8× A100, GRPO | Community reports: 2–4× faster than naive HF due to vLLM + resharding | With co-located vLLM: competitive at small scale; degrades at larger context |
+| 70B+ full-param GRPO | ✅ Efficient with Megatron-LM + SGLang | ⚠️ Possible with FSDP ZeRO-3 but slow; practical limit |
+| 70B+ QLoRA GRPO | Not optimised | ✅ TRL + QLoRA is the go-to recipe |
+
+### 4.3 Agentic RL Specifically
+
+| Capability | VeRL | TRL |
+|---|---|---|
+| Multi-turn rollout | ✅ | ✅ (limited) |
+| Tool-call execution during rollout | ✅ Async (GPU not blocked) | ⚠️ Synchronous (GPU blocked) |
+| Code sandbox | ✅ SandboxFusionTool | ❌ (user must integrate) |
+| Reward on trajectory outcome | ✅ | ✅ (via reward_funcs) |
+| Step-level credit assignment | ✅ (OTB, StepPO-compatible) | ❌ (trajectory-level only natively) |
+| Multi-node rollout | ✅ (SGLang multi-node) | ⚠️ (experimental vLLM multi-node) |
+
+---
+
+## 5. Recommendation
+
+### 5.1 Decision Framework
+
+```
+If target model size > 70B (full-param RL)         → VeRL + Megatron-LM
+If agentic coding trajectories are core use-case    → VeRL (async tool loops)
+If commodity GPUs (≤8× A100) + any HF model        → TRL (GRPOTrainer + vLLM)
+If LoRA/QLoRA post-training is acceptable           → TRL
+If rapid prototyping / research iteration           → TRL
+If production-scale, low-latency RL pipeline        → VeRL
+If VLM post-training (small-mid scale)              → TRL (simpler)
+If VLM post-training (large scale)                  → VeRL (VeOmni)
+```
+
+### 5.2 For a "Take Any HF Model and RL Post-Train It" Framework
+
+**Primary recommendation: TRL as the default, VeRL as the scale-out path.**
+
+**Rationale:**
+
+1. **TRL covers the 80% case:** Any HF model can be loaded, any reward function can be plugged in, and the `GRPOTrainer` with co-located vLLM gives competitive throughput up to ~70B models on reasonable hardware.
+
+2. **TRL's ergonomics are essential for user adoption:** A framework goal of "any HF model" implies the interface must be familiar and accessible. TRL achieves this; VeRL does not.
+
+3. **VeRL is the right backend for scale-out:** When users graduate to full-param 70B+ runs, or when async agentic trajectories are needed, VeRL is the right sub-backend. A framework could abstract both: use TRL for the training API surface, offer VeRL as a `backend="verl"` option for production scale.
+
+4. **Algorithm-wise, GRPO + DAPO fixes is the current best practice** for agentic-coding RL. Both TRL (GRPOTrainer) and VeRL support this. Implementing DAPO's dynamic sampling filter and decoupled clip on top of TRL's GRPOTrainer is straightforward.
+
+5. **Agentic coding gap:** TRL's missing async tool-execution loop is a real gap. For a framework targeting agentic coding post-training, this should be bridged — either by adopting VeRL's AgentLoop pattern or by implementing an async wrapper over TRL's rollout phase.
+
+### 5.3 Suggested Architecture for the Framework
+
+```
+Framework Public API (HF-compatible)
+    ↓
+Trainer Abstraction Layer
+    ├── Backend: TRL GRPOTrainer (default; <70B; commodity)
+    │       ├── vLLM co-located rollout
+    │       ├── GRPO + DAPO fixes (dynamic sampling, decoupled clip)
+    │       └── Reward: RLVR (test execution) | LLM-judge | ORM
+    └── Backend: VeRL (scale-out; ≥70B; H100 clusters; agentic)
+            ├── 3D-HybridEngine + SGLang
+            ├── Async AgentLoop + SandboxFusionTool
+            └── Megatron-LM for 70B+ full-param
+
+Reward Layer (shared)
+    ├── Test-suite executor (RLVR for coding)
+    ├── Format verifier
+    ├── PRM (process reward; TRL PRMTrainer)
+    └── LLM-as-judge
+
+Algorithm Layer (shared config, maps to trainer)
+    └── GRPO / DAPO / RLOO / PPO / DPO
+```
+
+---
+
+## 6. Sources
+
+### Framework Documentation
+- VeRL GitHub: https://github.com/volcengine/verl
+- TRL GitHub: https://github.com/huggingface/trl
+- VeRL DeepWiki (architecture reference): https://deepwiki.com/search/what-is-verls-architecture-wha_d0f02939-74bd-4877-8821-2249dac5e72e
+- TRL DeepWiki (trainer reference): https://deepwiki.com/search/what-trainers-does-trl-support_cb760bf9-4c30-47cc-8f80-1b10e71a53bf
+
+### Algorithm Papers
+- **GRPO / DeepSeek-R1-Zero:** DeepSeek-AI et al. (2025). *DeepSeek-R1.* https://arxiv.org/abs/2501.12948
+- **DAPO:** Yu et al. (2025). *DAPO: Decoupled Clip and Dynamic Sampling Policy Optimization.* (ByteDance / VeRL team)
+- **Dr.GRPO:** Liu et al. (2025). *Understanding GRPO: Dr.GRPO.* Referenced in RLHF book: https://rlhfbook.com/c/06-policy-gradients
+- **REINFORCE++:** Hu (2025). *REINFORCE++: A Simple and Efficient Approach for Aligning LLMs.* Referenced in multiple 2025 papers.
+- **RLOO:** Ahmadian et al. (2024). *Back to Basics: Revisiting REINFORCE-Style Optimization for Language Models.* 
+- **GSPO:** Referenced in UC Berkeley Scalable AI lecture (Spring 2026): http://scalable-ai.eecs.berkeley.edu/assets/lecture_slides/lecture_15.pdf
+- **StepPO:** arxiv.org/html/2604.18401v1 — *StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning*
+- **ARPO:** arxiv.org/html/2507.19849v1 — *Agentic Reinforced Policy Optimization*
+
+### Benchmarks & Comparisons
+- VeRL v0.5.0 benchmark (8× H100, 1.5B model): https://rlinf.readthedocs.io/en/latest/rst_source/blog/compare_with_verl.html
+- GRPO VRAM/cost analysis on H200/B200: https://www.spheron.network/blog/grpo-fine-tuning-gpu-cloud
+- Oumi: Running GRPO in TRL and VeRL: https://oumi.ai/blog/run-grpo-training-in-oumi-using-the
+
+### Blog Posts / Surveys
+- UC Berkeley Scalable AI Lecture 15 (Spring 2026) — Algorithm comparison table: http://scalable-ai.eecs.berkeley.edu/assets/lecture_slides/lecture_15.pdf
+- "From REINFORCE to Dr. GRPO" blog (Qingfeng, 2025): https://lancelqf.github.io/note/llm_post_training
+- Sebastian Raschka — State of LLMs 2025: https://magazine.sebastianraschka.com/p/state-of-llms-2025
+- RLHF and Post-Training Book (Nathan Lambert): https://rlhfbook.com/c/06-policy-gradients
+- TRL blog — Liger GRPO (May 2025): Hugging Face blog
+- TRL blog — Co-located vLLM (Jun 2025): Hugging Face blog
+- TRL blog — VLM alignment (Aug 2025): Hugging Face blog
+- TRL blog — OpenEnv (Oct 2025): Hugging Face blog
+- TRL v1 release blog (Mar 2026): Hugging Face blog

research/05-trace-replay-distillation.md

@@ -0,0 +1,492 @@
+# Trace-Replay Distillation: Prior Art Analysis
+
+## Overview & The User's Idea
+
+**Trace-replay distillation** is a novel training paradigm where LLM application traces (interleaved reasoning steps, tool calls, observations) are replayed with multiple teacher models at each step to harvest distillation signal. The core idea:
+
+1. **Capture** a trajectory from a target LLM application (e.g., coding agent session)
+2. **Freeze** the trace at each decision point
+3. **Replay** that exact step with N different teacher models to see alternative actions
+4. **Harvest** the per-step variance as training signal: preferences, rewards, or distilled knowledge
+5. **Train** student model on this dense, step-level supervision
+
+This creates **trace-level multi-teacher distillation**—unlike traditional token-level or response-level distillation, it operates at the granularity of agentic decision-making.
+
+---
+
+## Related Work: Multi-Teacher Distillation
+
+### Classical Multi-Teacher Knowledge Distillation
+
+**Ensemble-then-Distill Approaches** (NeurIPS 2024, arXiv:2302.07215):
+- Transfer knowledge from multiple teacher LLMs to a single student
+- Key challenge: resolving knowledge conflicts between teachers
+- Methods: weighted averaging, routing, or purification of teacher rationales
+- **Gap**: Operates at **response-level**, not trace-level granularity
+
+**Knowledge Purification in Multi-Teacher KD** (ICLR 2026):
+- Introduces "Knowledge Purification" to consolidate rationales from multiple teachers
+- Five purification methods to handle conflicts and enhance efficiency
+- Router-based methods show robust generalization
+- **Gap**: No step-level replay; uses independent teacher generations
+
+**Mixture-of-Agents (MoA) Alignment** (Together.AI, ICLR 2025):
+- Distills collective intelligence from multiple LLM agents into smaller model
+- Layered architecture where agents in each layer see previous layer outputs
+- **Key insight**: LLMs generate better responses when shown other models' outputs
+- **Gap**: Operates on full responses, not replaying trajectories step-by-step
+
+---
+
+## Related Work: Trace-Level Reinforcement Learning & Distillation
+
+### Agent Distillation
+
+**Agent Distillation** (Emergent Mind, 2025):
+- Transfers multi-step agentic behaviors from powerful teachers to smaller students
+- Uses trajectory-centric training with Thought-Action-Observation format
+- Loss function: `L_AD = -E[Σ(log p_S(t_t) + log p_S(a_t))]`
+- **Gap**: Single-teacher imitation, no multi-teacher replay
+
+**SMOLAgents Distillation** (GitHub: Nardien/agent-distillation):
+- Generates trajectories from teacher agent (Qwen32B)
+- Trains student via supervised fine-tuning on actions
+- **Gap**: No multi-teacher comparison at each step
+
+### On-Policy vs Off-Policy Distillation
+
+**Key Distinction** (Aman's AI Journal):
+- **Off-Policy**: Student learns from teacher-generated trajectories (static dataset)
+- **On-Policy**: Student learns from its own rollouts, scored by teacher
+- **Multi-Teacher On-Policy**: Student rollouts scored by ensemble of teachers
+- **User's Idea**: Hybrid approach—**off-policy trace collection + on-policy multi-teacher replay**
+
+---
+
+## Related Work: Process Reward Models (PRMs)
+
+### The Step-Level Reward Paradigm
+
+**Math-Shepherd** (ACL 2024):
+- Assigns reward scores to each step of mathematical solutions
+- Automatic labeling via Monte Carlo Tree Search (MCTS)
+- **Key insight**: Step-level > outcome-level feedback for reasoning
+- **Connection**: Provides reward signal for trace-replay evaluation
+
+**OmegaPRM** (arXiv 2406.06592):
+- Divide-and-conquer MCTS algorithm for automated process supervision
+- Pinpoints first error in Chain-of-Thought via binary search
+- Collects 1.5M+ process supervision annotations
+- **Key insight**: Automated step-level error detection at scale
+- **Connection**: Could automatically label which replay steps are "good"
+
+**R-PRM: Reasoning-Driven Process Reward Modeling** (EMNLP 2025):
+- Leverages LLMs' reasoning capabilities for step evaluation
+- Three stages: cold start, self-evolution via preference optimization, inference scaling
+- **Key insight**: Direct evaluation constrains learning; reasoning about steps is better
+- **Connection**: The "judge" in multi-teacher replay should reason about step quality
+
+### Process Reward Models for Agents
+
+**AgentPRM** (arXiv 2025.02):
+- Framework for process reward models specifically for LLM agents
+- Practical directions for implementation
+- **Direct connection**: Evaluates tool-use steps, not just reasoning steps
+- **Gap**: Doesn't propose multi-teacher replay mechanism
+
+---
+
+## Related Work: Counterfactual Rollouts & Tree Search
+
+### rStar & Self-Play Reasoning
+
+**rStar: Mutual Reasoning Makes Smaller LLMs Stronger** (arXiv 2408.06195):
+- Self-play mutual generation-discrimination process
+- Uses MCTS with **human-like reasoning actions**:
+  - Propose one-step thought
+  - Complete reasoning
+  - Propose subquestions
+  - Re-answer subquestion
+  - Rephrase question
+- Two SLMs: Generator + Discriminator verify trajectories
+- **Closest precedent**: Different models take alternate steps in trajectory
+- **Key difference**: Models take **different roles**, not same role at same trace position
+
+**rStar-Math** (ICML 2025):
+- Small LLMs achieve o1-level performance via self-evolved deep thinking
+- Code-augmented CoT via extensive MCTS rollouts
+- Process Preference Model (PPM) instead of naive scoring
+- **Key insight**: High-quality trajectories from tree search enable distillation
+- **Connection**: MCTS rollouts **are** counterfactual exploration of alternative steps
+
+### Tree-of-Thoughts & MCTS
+
+**Tree-of-Thoughts** (Yao et al., 2023):
+- Multiple reasoning paths explored simultaneously
+- Deliberate decision-making via search algorithms
+- **Connection**: Provides search framework for generating replay alternatives
+
+**ReST-MCTS*** (NeurIPS 2024):
+- LLM self-training via process reward guided tree search
+- Monte Carlo rollout with self-critic mechanism
+- **Connection**: Generates diverse trajectories via search; could be extended to multi-teacher
+
+---
+
+## Related Work: Agentic Trajectory Datasets
+
+### Software Engineering Agents
+
+**SWE-Gym & OpenHands Trajectories**:
+- 67k+ agent trajectories solving GitHub issues
+- Complete execution traces: thoughts, actions, observations, tool calls
+- Generated with Qwen3-Coder-480B, Claude, GPT-4o
+- **Direct applicability**: Rich trace data for replay experiments
+- **Example**: SWE-rebench-openhands-trajectories dataset
+
+**Shepherd: Pattern-Guided Trajectory Selection** (ICLR 2026):
+- Analyzes 3,908 execution trajectories across 18 models
+- Identifies failure patterns: FA (fail to interact), OO (simultaneous actions), FT (premature completion)
+- Uses LLM-as-judge to select optimal trajectories
+- **Key insight**: Not all steps in traces are equally valuable
+- **Connection**: Suggests importance-weighting in replay
+
+### GUI & Web Agents
+
+**AgentTrek**:
+- Large-scale multimodal trajectory dataset from web tutorials
+- Guided replay demonstrations
+- **Connection**: Demonstrates feasibility of guided/counterfactual replay
+
+**r2e-gym**:
+- Procedural environments for training SWE agents
+- Collects successful trajectories via SFT
+- **Connection**: Shows trajectory collection pipelines exist
+
+---
+
+## The Closest Published Precedent
+
+### rStar: Partial Counterfactual Evaluation
+
+The **rStar** framework (arXiv 2408.06195) is the closest published work:
+
+1. **Multi-model interaction**: Two SLMs (generator + discriminator) interact over trajectories
+2. **Step-level evaluation**: Discriminator evaluates each step of generator's trajectory
+3. **MCTS exploration**: Extensive rollouts create diverse alternatives
+4. **Mutual consistency**: Agreement between models used as quality signal
+
+**Critical Differences from User's Idea**:
+
+| Aspect | rStar | User's Trace-Replay |
+|--------|-------|---------------------|
+| **Model Roles** | Fixed generator vs discriminator roles | Same role (e.g., "coding agent") |
+| **Replay Granularity** | Discriminator judges full trajectories | Re-evaluate **each step** with N models |
+| **Counterfactual** | Implicit via MCTS search | **Explicit**: Fix trace, replay step |
+| **Supervision Target** | Final trajectory selection | Per-step preference/reward data |
+| **Scale** | 2 models, self-play | N models, multi-teacher |
+
+**Verdict**: rStar demonstrates the **power of multi-model step-level evaluation**, but doesn't implement the **frozen-trace replay mechanism** at each step.
+
+---
+
+## Novelty Assessment
+
+### What IS Novel
+
+#### 1. **Trace-Freezing + Multi-Teacher Replay**
+No published work systematically:
+- Freezes a trace at step `t`
+- Replays **that exact state** with N different teachers
+- Harvests variance as per-step supervision
+
+#### 2. **Step-Level Multi-Teacher Preference Data**
+- Traditional multi-teacher: response-level preferences
+- PRMs: single-teacher step evaluation
+- **Gap**: No multi-teacher per-step comparison
+
+#### 3. **Cost-Scalable Sampling Strategies**
+The user's concern about "8000 LLM calls" suggests:
+- Value-of-information gating
+- Importance sampling for steps
+- Teacher model routing
+
+These **practical scaling mechanisms** are under-explored in literature.
+
+### What ISN'T Novel (But Under-Applied)
+
+#### 1. **Multi-Teacher Distillation**
+- Well-established concept (ICLR 2026, NeurIPS 2024)
+- Knowledge purification methods exist
+- **Gap**: Apply to **agentic traces**, not just QA
+
+#### 2. **Process Reward Models**
+- Math-Shepherd, OmegaPRM prove step-level supervision works
+- **Gap**: Multi-teacher PRM for general agentic tasks
+
+#### 3. **Counterfactual Evaluation**
+- Tree-of-Thoughts, MCTS explore alternatives
+- **Gap**: Explore alternatives at **harvested trace positions**, not just during generation
+
+### Open Territory
+
+#### 1. **Trace Replay for Tool-Use Agents**
+- SWE-Gym trajectories could be replayed
+- Tool selection (bash, edit, search) could be evaluated multi-teacher
+- **Novel**: Process-level reward for **tool-use steps**
+
+#### 2. **Reward Shaping from Multi-Teacher Variance**
+- Low variance → high teacher agreement → high confidence reward
+- High variance → explore disagreement as signal
+- **Novel**: Use variance as **reward certainty** measure
+
+#### 3. **On-Policy Trace Collection + Off-Policy Multi-Teacher Replay**
+- Student collects traces (on-policy)
+- Teachers replay steps for supervision (off-policy)
+- **Novel**: Hybrid on/off-policy RL with multi-teacher replay
+
+---
+
+## Cost & Feasibility Analysis
+
+### The Cost Problem
+
+For a **1000-step trace with 8 teachers**:
+- **Baseline**: 8000 forward passes
+- **Cost**: ~$0.008/step × 1000 × 8 = **$64 per trace**
+- **Scale**: 10k traces = **$640,000**
+
+### Practical Mitigation Strategies
+
+#### 1. **Value-of-Information Gating** (Active Selection)
+Only replay steps with **high uncertainty**:
+- Measure student model's entropy at step `t`
+- If `H(p(a_t|s_t)) > τ`, query teachers
+- Est. savings: **60-80% of steps** (based on PRM literature)
+
+#### 2. **Teacher Model Routing**
+- Route to **subset** of teachers per step
+- Learned router (RouterLLM, Chen et al. 2024)
+- Est. savings: **3-4x cost reduction**
+
+#### 3. **Step Subsampling**
+- Replay every **k-th step** (e.g., k=5)
+- Interpolate rewards for intermediate steps
+- Est. savings: **5x cost reduction**
+
+#### 4. **Model Cascade**
+- Query **weak teacher** first
+- Only query strong teacher if uncertain
+- **FrugalGPT** approach (Chen et al. 2023)
+- Est. savings: **2-3x cost reduction**
+
+### Combined Strategy Example
+
+**Tiered Replay Strategy**:
+1. Student generates trace
+2. Query **weak teacher** (e.g., 8B) at each step: $0.001/step
+3. If |reward - threshold| < ε (borderline), query **strong teacher** (e.g., 70B): $0.01/step
+4. Expected queries: 1000 weak + 200 strong = **$3/trace** (vs $64 baseline)
+
+**Feasibility**: Yes, with these strategies, **trace-replay is feasible at scale**.
+
+---
+
+## Reward Design Options
+
+Given N model predictions at step `t`, how to generate reward?
+
+### Option 1: Plurality Vote (Binary)
+```python
+reward_t = majority_vote(actions_t)  # 0 or 1
+```
+- **Pros**: Simple, interpretable
+- **Cons**: Crude, loses confidence information
+- **Best for**: High-agreement scenarios (discrete actions)
+
+### Option 2: Weighted Consensus
+```python
+reward_t = Σ w_i * score(action_i) / Σ w_i
+```
+Where `w_i` = teacher capability weight
+- **Pros**: Differentiates teacher quality
+- **Cons**: Requires teacher capability estimation
+- **Best for**: Heterogeneous teacher pool
+
+### Option 3: Preference Pairs for DPO
+```python
+# Among N actions, create (chosen, rejected) pairs
+pairs = [(best_action, worst_action), (best, second_best), ...]
+# Train via Direct Preference Optimization
+```
+- **Pros**: Leverages recent RL advances, avoids reward model training
+- **Cons**: Pair construction heuristic
+- **Best for**: When you want to **avoid explicit reward modeling**
+
+### Option 4: Variance-Weighted Reward
+```python
+mean_reward = mean(score(actions))
+variance = var(score(actions))
+reward_t = mean_reward * exp(-λ * variance)  # Lower confidence if high disagreement
+```
+- **Pros**: Quantifies uncertainty, prevents overfitting to noisy steps
+- **Cons**: Requires calibration of λ
+- **Best for**: Steps with **inherent ambiguity**
+
+### Option 5: Process Reward Model Fine-Tuning
+```python
+# Train a separate PRM on (state, action, reward) tuples from replay
+reward_t = PRM(state_t, action_t)
+```
+- **Pros**: Learns generalizable step evaluation
+- **Cons**: Requires additional model, training data
+- **Best for**: Long-term deployment with many traces
+
+### Recommendation: Hybrid Approach
+
+**For initial experiments**: **Option 3 (DPO Preference Pairs)**
+- Avoid reward model complexity
+- Leverage strong DPO baselines (Tülu 3, OpenThoughts)
+
+**For production**: **Option 5 (Train PRM)**
+- Amortizes cost across many traces
+- Enables test-time compute scaling (like rStar-Math)
+
+---
+
+## Recommendation for Framework
+
+### Proposed Architecture: **Trace-Replay with Multi-Teacher Process Supervision (TRAMPS)**
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                      Data Collection                      │
+│  ─────────────────────────────────────────────────────  │
+│  Student Model Generates Traces (SWE-Gym style)        │
+│  Store: {state_t, action_t, observation_t}_{t=1..T}    │
+└──────────────────────┬──────────────────────────────────┘
+                       │
+                       ▼
+┌─────────────────────────────────────────────────────────┐
+│                  Replay & Harvesting                     │
+│  ─────────────────────────────────────────────────────  │
+│  For each step t:                                       │
+│    ├─ Gating: Query teachers if uncertainty > τ         │
+│    ├─ Parallel: Query N teacher models                 │
+│    │    action_i ~ π_teacher_i(state_t)                │
+│    └─ Harvest:                                         │
+│         • Preferences (best vs worst)                  │
+│         • Process rewards (mean score)                 │
+│         • Variance estimates                           │
+└──────────────────────┬──────────────────────────────────┘
+                       │
+                       ▼
+┌─────────────────────────────────────────────────────────┐
+│                   Training Signal                        │
+│  ─────────────────────────────────────────────────────  │
+│  Option A: DPO on preference pairs                      │
+│  Option B: Train Process Reward Model                   │
+│  Option C: Distillation with variance weighting       │
+└──────────────────────┬──────────────────────────────────┘
+                       │
+                       ▼
+┌─────────────────────────────────────────────────────────┐
+│                  Student Fine-Tuning                     │
+│  ─────────────────────────────────────────────────────  │
+│  SFT: Mimic best teacher actions at each step         │
+│  RL: Optimize process rewards (if PRM trained)        │
+└─────────────────────────────────────────────────────────┘
+```
+
+### Key Components
+
+1. **Uncertainty-Gated Replay**
+   - Only query teachers at "interesting" steps
+   - Use student model's entropy as gating signal
+
+2. **Multi-Teacher Process Harvester**
+   - Parallel inference across N teachers
+   - Extract: preferences, rewards, variance, hidden states
+
+3. **DPO Trainer**
+   - Convert N actions into preference pairs
+   - No explicit reward model needed
+
+4. **Optional PRM Trainer**
+   - Train process reward model if compute permits
+   - Enables test-time scaling (like rStar-Math)
+
+### Baseline Implementation Path
+
+**Phase 1 (Week 1-2)**: Build on **OpenHands traces** dataset
+- Use existing SWE-Gym traces
+- Implement simple plurality vote reward
+- Validate signal quality
+
+**Phase 2 (Week 3-4)**: Add **gating** and **teacher routing**
+- Implement entropy-based step selection
+- Add learned router (small classifier)
+- Measure cost savings
+
+**Phase 3 (Week 5-6)**: **DPO integration**
+- Replace SFT with DPO on preference pairs
+- Compare vs SFT baseline
+
+**Phase 4 (Week 7-8)**: **PRM training**
+- Train small PRM on harvested data
+- Implement test-time scaling
+- Compare vs DPO
+
+---
+
+## Sources & Key Papers
+
+### Multi-Teacher Distillation
+1. **Jin et al. (2026)**. "Exploring Knowledge Purification in Multi-Teacher KD for LLMs". *ICLR 2026*. https://openreview.net/forum?id=7pvJoB4aKO
+2. **Together.AI (2024)**. "Mixture-of-Agents Alignment". *ICLR 2025 Spotlight*. https://www.together.ai/blog/moaa
+3. **Fukuda et al. (2017)**. "Multi-teacher knowledge distillation". *arXiv:2302.07215*
+
+### Agent Distillation & Trajectories
+4. **Wang et al. (2024c)**. "OpenHands: A versatile agent framework". https://github.com/All-Hands-AI/OpenHands
+5. **SWE-Gym (2024)**. "Training Software Engineering Agents and Verifiers with SWE-Gym". https://arxiv.org/abs/2412.21139
+6. **Cuadron et al. (2026)**. "Shepherd: Pattern-Guided Trajectory Selection for Coding Agents". *ICLR 2026*. https://openreview.net/forum?id=ZBOFr4ryBk
+7. **AgentTrek**. "Agent Trajectory Synthesis via Guiding Replay". https://agenttrek.github.io
+
+### Process Reward Models
+8. **Wang et al. (2024b)**. "Math-Shepherd: Verify and Reinforce LLMs Step-by-step". *ACL 2024*. https://arxiv.org/abs/2312.09152
+9. **Luo et al. (2024)**. "OmegaPRM: Automated Process Supervision". *arXiv:2406.06592*
+10. **Wang et al. (2025)**. "R-PRM: Reasoning-Driven Process Reward Modeling". *EMNLP 2025*. https://aclanthology.org/2025.emnlp-main.679.pdf
+11. **Luo et al. (2025)**. "AgentPRM: Process Reward Models for LLM Agents". *arXiv 2025.02*
+
+### Counterfactual Rollouts & Tree Search
+12. **Guan et al. (2025)**. "rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking". *ICML 2025*. https://arxiv.org/abs/2501.04519
+13. **Qi et al. (2024)**. "Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers". *arXiv:2408.06195*
+14. **Yao et al. (2023)**. "Tree of Thoughts: Deliberate Problem Solving with Large Language Models". *NeurIPS 2023*
+15. **Snell et al. (2024)**. "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters". https://arxiv.org/abs/2408.03314
+
+### Synthetic Data & Reasoning
+16. **Guha et al. (2025)**. "OpenThoughts: Data Recipes for Reasoning Models". https://huggingface.co/papers/2506.04178
+17. **Xu et al. (2024)**. "Magpie: Alignment Data Synthesis from Scratch". *ICLR 2025*. https://arxiv.org/abs/2406.08464
+18. **Lambert (2025)**. "Synthetic Data". *RLHF and Post-Training Book*. https://rlhfbook.com/c/12-synthetic-data
+
+### Multi-Agent & Distillation Theory
+19. **Aman (2024)**. "Knowledge Distillation Primer". https://aman.ai/primers/ai/knowledge-distillation
+20. **Emergent Mind (2025)**. "Agent Distillation". https://www.emergentmind.com/topics/agent-distillation
+21. **Emergent Mind (2025)**. "Process-supervised Reward Models (PRMs)". https://www.emergentmind.com/topics/process-supervised-reward-models-prms
+
+---
+
+## Summary
+
+**The user's trace-replay distillation idea is**: 
+✅ **Plausible and largely novel** at step-level granularity
+✅ **Grounded** in multi-teacher KD, PRMs, and counterfactual evaluation literature
+✅ **Feasible** with cost mitigation strategies (gating, routing, cascades)
+✅ **Actionable** via incremental framework building on existing components
+
+**Next steps**:
+1. Implement **Phase 1** on SWE-Gym traces (plurality vote reward)
+2. Compare cost vs. signal quality tradeoffs
+3. Publish as "Trace-Replay Multi-Teacher Process Supervision" 
+
+The key contribution is **operationalizing multi-teacher evaluation at the granularity of agentic decision-making**, bridging the gap between process reward models and ensemble knowledge distillation.