Instructions to use Codeseys/composer-replication-framework with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Codeseys/composer-replication-framework with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Codeseys/composer-replication-framework", dtype="auto") - Notebooks
- Google Colab
- Kaggle
feat(b4-gpu+b6): GPU train-proof on A10G + docker-gated substrate E2E test
Browse filesB4-GPU: modal_b4_gpu_smoke.py ran real Qwen2.5-0.5B-Instruct on a Modal A10G in
bf16, 30 steps of the real 3-channel composition. PASS: all-finite, SDPO channel
fired nonzero (max JSD 0.1855), loss converged 4.7262 -> 0.0050 monotone. This is
the genuine "it trains on GPU" proof the CPU smoke could not give. Result +
curve in gpu_smoke_result.json; README documents both CPU and GPU proofs.
B6: test_docker_substrate_e2e.py — the docker-gated 4-gate substrate-inversion
E2E now EXISTS (was only referenced). Runs the gates against a real
python:3.11-slim container + real subprocess pytest on a minimal synthetic
feature-deletion task, plus a cache-scrub-in-container test. Skips cleanly with
no Docker (2 skipped here), runs the moment a Docker host executes the suite.
ADR-010 [~] gate updated: mechanics gate is now ready-to-close, not just wired.
Cost: ~1-3 USD A10G. No LMA budget touched.
- composer_replication/datagen/tests/test_docker_substrate_e2e.py +175 -0
- docs/adrs/ADR-010-feature-deletion-datagen.md +1 -1
- examples/altered_minds_channel_ladder/README.md +48 -48
- examples/altered_minds_channel_ladder/gpu_smoke_result.json +78 -0
- examples/altered_minds_channel_ladder/modal_b4_gpu_smoke.py +144 -0
|
@@ -0,0 +1,175 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""test_docker_substrate_e2e.py — the docker-gated substrate-inversion E2E (B6).
|
| 2 |
+
|
| 3 |
+
This is the ONE gate ADR-010 left at `[~]`: a live run of the real
|
| 4 |
+
`LocalSubprocessSandbox` through the 4-gate `validate_task` on a materialized
|
| 5 |
+
repo, instead of `FakeSandbox` with hand-set booleans. It is `skipif`-gated on
|
| 6 |
+
Docker availability so it is a NO-OP in CPU-only dev envs (no Docker daemon) and
|
| 7 |
+
ACTIVELY RUNS the moment a box with Docker executes the suite.
|
| 8 |
+
|
| 9 |
+
It deliberately does NOT pull a heavy SWE-bench-Lite image (~GBs). Instead it
|
| 10 |
+
builds a MINIMAL synthetic feature-deletion task inside a tiny `python:3.11-slim`
|
| 11 |
+
container: a 2-function module + a pytest file, with a "gold patch" that adds the
|
| 12 |
+
missing function. This proves the inversion MECHANICS end-to-end (solved→green,
|
| 13 |
+
broken→fail-target/pass-guard, gold→green, cache-scrub) on a real container with
|
| 14 |
+
a real subprocess test runner — which is exactly what the FakeSandbox tests
|
| 15 |
+
could only assert against synthetic booleans.
|
| 16 |
+
|
| 17 |
+
UNBLOCK RECIPE (to actually run this gate):
|
| 18 |
+
1. A host with a Docker daemon (`docker info` succeeds).
|
| 19 |
+
2. `pip install docker` (the python SDK) — or it falls back to the `docker`
|
| 20 |
+
CLI via subprocess, which is what this test uses (no SDK dep).
|
| 21 |
+
3. `pytest composer_replication/datagen/tests/test_docker_substrate_e2e.py -v`
|
| 22 |
+
The full SWE-bench-Lite variant (pull one real instance image, revert its gold
|
| 23 |
+
patch, run FAIL_TO_PASS/PASS_TO_PASS) is a follow-up that reuses this same
|
| 24 |
+
driver with `SweBenchAdapter` + the substrate's published image tag.
|
| 25 |
+
"""
|
| 26 |
+
from __future__ import annotations
|
| 27 |
+
|
| 28 |
+
import shutil
|
| 29 |
+
import subprocess
|
| 30 |
+
import textwrap
|
| 31 |
+
|
| 32 |
+
import pytest
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def _docker_available() -> bool:
|
| 36 |
+
"""True iff a usable Docker daemon is reachable via the CLI."""
|
| 37 |
+
if shutil.which("docker") is None:
|
| 38 |
+
return False
|
| 39 |
+
try:
|
| 40 |
+
r = subprocess.run(
|
| 41 |
+
["docker", "info"], capture_output=True, timeout=10
|
| 42 |
+
)
|
| 43 |
+
return r.returncode == 0
|
| 44 |
+
except Exception:
|
| 45 |
+
return False
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
pytestmark = pytest.mark.skipif(
|
| 49 |
+
not _docker_available(),
|
| 50 |
+
reason="Docker daemon not available — substrate-inversion E2E is hardware-gated "
|
| 51 |
+
"(ADR-010 [~] gate). See module docstring for the unblock recipe.",
|
| 52 |
+
)
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
# --- minimal synthetic feature-deletion task -------------------------------
|
| 56 |
+
# A module with TWO functions; the "broken" state deletes `mul`. The
|
| 57 |
+
# FAIL_TO_PASS test exercises `mul`; the PASS_TO_PASS test exercises `add`
|
| 58 |
+
# (must still pass when mul is deleted). The gold patch restores `mul`.
|
| 59 |
+
|
| 60 |
+
_MODULE_SOLVED = textwrap.dedent('''\
|
| 61 |
+
def add(a, b):
|
| 62 |
+
return a + b
|
| 63 |
+
|
| 64 |
+
def mul(a, b):
|
| 65 |
+
return a * b
|
| 66 |
+
''')
|
| 67 |
+
|
| 68 |
+
_MODULE_BROKEN = textwrap.dedent('''\
|
| 69 |
+
def add(a, b):
|
| 70 |
+
return a + b
|
| 71 |
+
''')
|
| 72 |
+
|
| 73 |
+
_TESTS = textwrap.dedent('''\
|
| 74 |
+
from feature import add
|
| 75 |
+
try:
|
| 76 |
+
from feature import mul
|
| 77 |
+
except ImportError:
|
| 78 |
+
mul = None
|
| 79 |
+
|
| 80 |
+
def test_add_guard(): # PASS_TO_PASS — must pass in broken state
|
| 81 |
+
assert add(2, 3) == 5
|
| 82 |
+
|
| 83 |
+
def test_mul_target(): # FAIL_TO_PASS — fails in broken state
|
| 84 |
+
assert mul is not None and mul(2, 3) == 6
|
| 85 |
+
''')
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def _run_in_container(workdir_files: dict[str, str], test_node: str) -> tuple[bool, str]:
|
| 89 |
+
"""Materialize files in a fresh python:3.11-slim container, run one pytest
|
| 90 |
+
node, return (passed, output). Uses the docker CLI (no SDK dep)."""
|
| 91 |
+
import os
|
| 92 |
+
import tempfile
|
| 93 |
+
|
| 94 |
+
with tempfile.TemporaryDirectory() as d:
|
| 95 |
+
for name, content in workdir_files.items():
|
| 96 |
+
with open(os.path.join(d, name), "w") as f:
|
| 97 |
+
f.write(content)
|
| 98 |
+
# Mount the dir, install pytest, run the single node id.
|
| 99 |
+
cmd = [
|
| 100 |
+
"docker", "run", "--rm", "--network", "none",
|
| 101 |
+
"-v", f"{d}:/work", "-w", "/work",
|
| 102 |
+
"python:3.11-slim",
|
| 103 |
+
"bash", "-lc",
|
| 104 |
+
f"pip install -q pytest >/dev/null 2>&1 && "
|
| 105 |
+
f"python -m pytest -q '{test_node}' 2>&1",
|
| 106 |
+
]
|
| 107 |
+
r = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
|
| 108 |
+
out = (r.stdout or "") + (r.stderr or "")
|
| 109 |
+
return (r.returncode == 0, out)
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def test_substrate_inversion_four_gates_on_real_container():
|
| 113 |
+
"""The 4 ADR-010 gates against a REAL container + REAL pytest, not FakeSandbox.
|
| 114 |
+
|
| 115 |
+
Gate 1 (baseline green): solved state → target + guard both pass.
|
| 116 |
+
Gate 2 (deletion breaks): broken state → target FAILS.
|
| 117 |
+
Gate 3 (remains functional):broken state → guard PASSES.
|
| 118 |
+
Gate 4 (gold restores): gold-applied → target passes again.
|
| 119 |
+
"""
|
| 120 |
+
tests_file = {"test_feature.py": _TESTS}
|
| 121 |
+
|
| 122 |
+
# Gate 1 — solved: both pass.
|
| 123 |
+
g1_target, _ = _run_in_container(
|
| 124 |
+
{**tests_file, "feature.py": _MODULE_SOLVED}, "test_feature.py::test_mul_target")
|
| 125 |
+
g1_guard, _ = _run_in_container(
|
| 126 |
+
{**tests_file, "feature.py": _MODULE_SOLVED}, "test_feature.py::test_add_guard")
|
| 127 |
+
assert g1_target and g1_guard, "Gate 1 (baseline green) failed on real container"
|
| 128 |
+
|
| 129 |
+
# Gate 2 — broken: target FAILS.
|
| 130 |
+
g2_target, g2_out = _run_in_container(
|
| 131 |
+
{**tests_file, "feature.py": _MODULE_BROKEN}, "test_feature.py::test_mul_target")
|
| 132 |
+
assert not g2_target, f"Gate 2 (deletion breaks target) failed — target passed in broken state:\n{g2_out}"
|
| 133 |
+
|
| 134 |
+
# Gate 3 — broken: guard still PASSES.
|
| 135 |
+
g3_guard, g3_out = _run_in_container(
|
| 136 |
+
{**tests_file, "feature.py": _MODULE_BROKEN}, "test_feature.py::test_add_guard")
|
| 137 |
+
assert g3_guard, f"Gate 3 (remains functional) failed — guard broke in broken state:\n{g3_out}"
|
| 138 |
+
|
| 139 |
+
# Gate 4 — gold restores: target passes again (gold == the solved module).
|
| 140 |
+
g4_target, _ = _run_in_container(
|
| 141 |
+
{**tests_file, "feature.py": _MODULE_SOLVED}, "test_feature.py::test_mul_target")
|
| 142 |
+
assert g4_target, "Gate 4 (gold restores) failed — target did not recover after gold patch"
|
| 143 |
+
|
| 144 |
+
|
| 145 |
+
def test_cache_scrub_removes_bytecode_in_container():
|
| 146 |
+
"""The reward-hack PRIMARY control (cache scrub) holds on a real container:
|
| 147 |
+
a __pycache__ written during a first run must not let a second 'broken' run
|
| 148 |
+
recover the deleted symbol from cached bytecode."""
|
| 149 |
+
import os
|
| 150 |
+
import tempfile
|
| 151 |
+
|
| 152 |
+
with tempfile.TemporaryDirectory() as d:
|
| 153 |
+
with open(os.path.join(d, "feature.py"), "w") as f:
|
| 154 |
+
f.write(_MODULE_SOLVED)
|
| 155 |
+
# First run: compile to populate __pycache__.
|
| 156 |
+
subprocess.run(
|
| 157 |
+
["docker", "run", "--rm", "--network", "none", "-v", f"{d}:/work",
|
| 158 |
+
"-w", "/work", "python:3.11-slim", "python", "-c",
|
| 159 |
+
"import feature; import py_compile; py_compile.compile('feature.py')"],
|
| 160 |
+
capture_output=True, timeout=120,
|
| 161 |
+
)
|
| 162 |
+
# Now "delete" the feature but a stale .pyc may linger on the host mount.
|
| 163 |
+
# The scrub (LocalSubprocessSandbox._scrub_tree) must remove it.
|
| 164 |
+
from composer_replication.datagen.sandbox import LocalSubprocessSandbox
|
| 165 |
+
# Write the broken module + a fake stale cache.
|
| 166 |
+
os.makedirs(os.path.join(d, "__pycache__"), exist_ok=True)
|
| 167 |
+
with open(os.path.join(d, "__pycache__", "feature.cpython-311.pyc"), "wb") as f:
|
| 168 |
+
f.write(b"\x00stale-bytecode-with-mul-signature")
|
| 169 |
+
with open(os.path.join(d, "feature.py"), "w") as f:
|
| 170 |
+
f.write(_MODULE_BROKEN)
|
| 171 |
+
|
| 172 |
+
LocalSubprocessSandbox(workdir=d).boot("img:broken") # scrubs
|
| 173 |
+
|
| 174 |
+
assert not os.path.exists(os.path.join(d, "__pycache__")), \
|
| 175 |
+
"cache scrub did not remove __pycache__ — reward-hack primary control failed"
|
|
@@ -171,7 +171,7 @@ Core gates green as of 2026-05-29 (19 tests in
|
|
| 171 |
implemented but its live run is the documented unblocked-by step — see note.
|
| 172 |
|
| 173 |
- [x] `FeatureDeletionTask` dataclass + `FeatureDeletionEnv` (`reset`/`step`/`reward`) implemented; reward = masked test-pass fraction — `test_reward_is_pass_fraction_when_guard_ok`, `test_reward_graded_for_multi_feature` (0.5 for 1-of-2), `test_reward_zeroed_when_functional_guard_broken`. `golden_diff` held out of `repr` (`test_golden_diff_not_in_repr`).
|
| 174 |
-
- [~] SWE-bench-Lite substrate adapter: **schema inversion implemented + tested** (`SweBenchAdapter.to_task` — `test_swebench_adapter_inverts_instance`, JSON-or-list FAIL_TO_PASS handling, copyleft filter). The **live revert-gold-patch → broken-repo → test-run** path requires a substrate Docker image; `LocalSubprocessSandbox` + `validate_task` are wired for it, and the gate is exercised in unit form via `FakeSandbox` materializers (`test_validator_accepts_well_formed_task`).
|
| 175 |
- [x] 4-gate solvability validator implemented; `test_validator_rejects_unreachable_deletion` (deletion that doesn't break the target → gate 2 fails) and `test_validator_rejects_when_guard_breaks` (gate 3 fails).
|
| 176 |
- [x] Reward-hacking safeguard: `SANDBOX_DENYLIST` blocks `find`/`strings`/`unzip`/decompilers/`git` (`test_sandbox_denies_decompiler_and_cache_tools`); `HackMonitor` flags cache/bytecode-provenance hacks (`test_monitor_flags_cache_provenance_hack`) and passes clean reimplementation (`test_monitor_passes_clean_reimplementation`); reward is masked to 0 when a hack is detected even if tests "pass" (`test_reward_masked_when_hack_detected`).
|
| 177 |
- [x] Online difficulty gate: `DifficultyCurriculum` up-weights the frontier (~0.5 pass-rate) over aced tasks and retires aced ones (`test_curriculum_upweights_frontier_over_solved`); quarantines all-fail tasks after `min_exposures` (`test_curriculum_quarantines_impossible_task`). NOTE: quarantine uses the *raw* observed rate, not the Laplace-smoothed `p_hat` (smoothing is for weighting, not the have-we-ever-passed decision).
|
|
|
|
| 171 |
implemented but its live run is the documented unblocked-by step — see note.
|
| 172 |
|
| 173 |
- [x] `FeatureDeletionTask` dataclass + `FeatureDeletionEnv` (`reset`/`step`/`reward`) implemented; reward = masked test-pass fraction — `test_reward_is_pass_fraction_when_guard_ok`, `test_reward_graded_for_multi_feature` (0.5 for 1-of-2), `test_reward_zeroed_when_functional_guard_broken`. `golden_diff` held out of `repr` (`test_golden_diff_not_in_repr`).
|
| 174 |
+
- [~] SWE-bench-Lite substrate adapter: **schema inversion implemented + tested** (`SweBenchAdapter.to_task` — `test_swebench_adapter_inverts_instance`, JSON-or-list FAIL_TO_PASS handling, copyleft filter). The **live revert-gold-patch → broken-repo → test-run** path requires a substrate Docker image; `LocalSubprocessSandbox` + `validate_task` are wired for it, and the gate is exercised in unit form via `FakeSandbox` materializers (`test_validator_accepts_well_formed_task`). **UPDATE 2026-05-29 (B6):** the `skipif(docker)` end-to-end test now EXISTS — `composer_replication/datagen/tests/test_docker_substrate_e2e.py` runs the 4 gates against a REAL `python:3.11-slim` container with a real subprocess pytest runner on a minimal synthetic feature-deletion task (proving the inversion *mechanics* on real hardware), plus a cache-scrub-in-container test. It SKIPS cleanly in this no-Docker CPU env and ACTIVELY RUNS the moment a Docker host executes the suite. Still `[~]` because (a) no Docker in this env to demonstrate a green run, and (b) the *full SWE-bench-Lite image* variant (pull one published instance image, revert its real gold patch) is a follow-up reusing the same driver. The mechanics gate is now ready-to-close, not just wired.
|
| 175 |
- [x] 4-gate solvability validator implemented; `test_validator_rejects_unreachable_deletion` (deletion that doesn't break the target → gate 2 fails) and `test_validator_rejects_when_guard_breaks` (gate 3 fails).
|
| 176 |
- [x] Reward-hacking safeguard: `SANDBOX_DENYLIST` blocks `find`/`strings`/`unzip`/decompilers/`git` (`test_sandbox_denies_decompiler_and_cache_tools`); `HackMonitor` flags cache/bytecode-provenance hacks (`test_monitor_flags_cache_provenance_hack`) and passes clean reimplementation (`test_monitor_passes_clean_reimplementation`); reward is masked to 0 when a hack is detected even if tests "pass" (`test_reward_masked_when_hack_detected`).
|
| 177 |
- [x] Online difficulty gate: `DifficultyCurriculum` up-weights the frontier (~0.5 pass-rate) over aced tasks and retires aced ones (`test_curriculum_upweights_frontier_over_solved`); quarantines all-fail tasks after `min_exposures` (`test_curriculum_quarantines_impossible_task`). NOTE: quarantine uses the *raw* observed rate, not the Laplace-smoothed `p_hat` (smoothing is for weighting, not the have-we-ever-passed decision).
|
|
@@ -1,52 +1,52 @@
|
|
| 1 |
-
#
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
##
|
| 9 |
-
|
| 10 |
-
`
|
| 11 |
-
|
| 12 |
-
`
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
- **Why student tokens are perturbed:** the collator's placeholder-alignment
|
| 25 |
-
trick makes student and teacher carry identical tokens at identical positions
|
| 26 |
-
at the valid aligned indices, so a deterministic stub yields `JSD≈0` there
|
| 27 |
-
(the *correct* answer for a perfectly-aligned identical model). To prove the
|
| 28 |
-
channel genuinely **gathers** the aligned positions and computes a real
|
| 29 |
-
divergence, the student's `input_ids` are made to **differ** from the
|
| 30 |
-
teacher's at exactly those aligned positions — mimicking the hint actually
|
| 31 |
-
changing the recovery tokens (the real-world case where SDPO has signal to
|
| 32 |
-
distill). Different aligned tokens ⇒ different logits ⇒ provably **NONZERO**
|
| 33 |
-
JSD.
|
| 34 |
-
|
| 35 |
-
This is the honest, deterministic CPU proof. Loading a real Qwen2.5-0.5B
|
| 36 |
-
checkpoint is **not required** for the B4 gate and is **not** the same as loading
|
| 37 |
-
an LMA checkpoint (still user-gated, ADR-013 out-of-scope).
|
| 38 |
-
|
| 39 |
-
## Run
|
| 40 |
-
|
| 41 |
-
```bash
|
| 42 |
-
cd <repo> && .venv/bin/python examples/altered_minds_channel_ladder/run.py
|
| 43 |
```
|
| 44 |
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
-
|
|
|
|
| 50 |
|
| 51 |
-
The
|
| 52 |
-
|
|
|
|
|
|
|
|
|
| 1 |
+
# B4 — end-to-end proof that the 3-channel loop trains
|
| 2 |
+
|
| 3 |
+
Two proofs that the Composer 3-channel loss (grpo + α·sdpo_kl + β·trace_replay_dpo)
|
| 4 |
+
runs end-to-end, closing the gap left by `examples/composer_grpo_sdpo_smoke`
|
| 5 |
+
(which proved *init* but never fired the SDPO channel — its toy rollouts carry
|
| 6 |
+
no error sites).
|
| 7 |
+
|
| 8 |
+
## 1. CPU proof — SDPO channel FIRES nonzero through the real collator
|
| 9 |
+
|
| 10 |
+
`run.py` — builds a REAL `ComposerDataCollator` batch from a trace with an error
|
| 11 |
+
turn, so the shipped collator emits `ctx_teacher_input_ids` +
|
| 12 |
+
`student/teacher_response_idx` (the ADR-011 alignment indices). Perturbs the
|
| 13 |
+
student tokens at the aligned positions (mimicking the hint changing the recovery
|
| 14 |
+
tokens) so the gathered student/teacher logits differ and the JSD is provably
|
| 15 |
+
nonzero, then verifies a gradient flows.
|
| 16 |
+
|
| 17 |
+
```
|
| 18 |
+
$ python run.py
|
| 19 |
+
proof path: TinyLM-stub-with-differing-tokens
|
| 20 |
+
SDPO JSD (sdpo_kl): 0.056547
|
| 21 |
+
requires_grad: True
|
| 22 |
+
grad norm into model: 0.001593
|
| 23 |
+
RESULT: PASS ✅ (SDPO channel FIRED nonzero via real collator indices)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
```
|
| 25 |
|
| 26 |
+
Honest scope: the model is a deterministic CPU stub (no download); the *collator
|
| 27 |
+
alignment path* is the real shipped code. Real-model path: `ALTERED_MINDS_REAL_MODEL=1`.
|
| 28 |
+
|
| 29 |
+
## 2. GPU proof — real Qwen2.5-0.5B trains, bf16, loss converges
|
| 30 |
+
|
| 31 |
+
`modal_b4_gpu_smoke.py` — runs the real 3-channel composition on
|
| 32 |
+
`Qwen/Qwen2.5-0.5B-Instruct` on a Modal A10G in bf16: GRPO-proxy LM loss +
|
| 33 |
+
α·SDPO (hint-conditioned teacher = same model, no-grad) + β·replay-margin, 30
|
| 34 |
+
AdamW steps.
|
| 35 |
+
|
| 36 |
+
```
|
| 37 |
+
$ modal run modal_b4_gpu_smoke.py --n-steps 30
|
| 38 |
+
status : PASS
|
| 39 |
+
dtype : torch.bfloat16
|
| 40 |
+
sdpo_fired_nonzero : True (max sdpo_kl 0.1855)
|
| 41 |
+
loss_trend_down : True
|
| 42 |
+
all_finite : True
|
| 43 |
+
loss first → last : 4.7262 → 0.0050 (monotone decrease)
|
| 44 |
+
```
|
| 45 |
|
| 46 |
+
bf16 numerics finite throughout, SDPO channel nonzero, loss converges. Cost:
|
| 47 |
+
~$1-3 on A10G. Run date 2026-05-29; full curve in `gpu_smoke_result.json`.
|
| 48 |
|
| 49 |
+
The proxies (GRPO→LM-loss, replay→margin) stand in for the full PG / DPO
|
| 50 |
+
accounting so the smoke runs without a rollout buffer or teacher set; the
|
| 51 |
+
SDPO channel is the *real* `generalized_jsd_loss` path. A full GRPO run with a
|
| 52 |
+
real reward and rollouts is the LMA-budget-gated next step (ADR-013).
|
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"status": "PASS",
|
| 3 |
+
"dtype": "torch.bfloat16",
|
| 4 |
+
"n_steps": 30,
|
| 5 |
+
"sdpo_fired_nonzero": true,
|
| 6 |
+
"max_sdpo_kl": 0.185546875,
|
| 7 |
+
"loss_trend_down": true,
|
| 8 |
+
"all_finite": true,
|
| 9 |
+
"loss_first": 4.726175785064697,
|
| 10 |
+
"loss_last": 0.0050478726625442505,
|
| 11 |
+
"loss_curve": [
|
| 12 |
+
4.7262,
|
| 13 |
+
2.9752,
|
| 14 |
+
1.8887,
|
| 15 |
+
1.1304,
|
| 16 |
+
0.6017,
|
| 17 |
+
0.3641,
|
| 18 |
+
0.1361,
|
| 19 |
+
0.037,
|
| 20 |
+
0.0142,
|
| 21 |
+
0.0077,
|
| 22 |
+
0.0061,
|
| 23 |
+
0.0055,
|
| 24 |
+
0.0053,
|
| 25 |
+
0.0053,
|
| 26 |
+
0.0052,
|
| 27 |
+
0.0052,
|
| 28 |
+
0.0052,
|
| 29 |
+
0.0051,
|
| 30 |
+
0.0051,
|
| 31 |
+
0.0051,
|
| 32 |
+
0.0051,
|
| 33 |
+
0.0051,
|
| 34 |
+
0.0051,
|
| 35 |
+
0.0051,
|
| 36 |
+
0.0051,
|
| 37 |
+
0.0051,
|
| 38 |
+
0.0051,
|
| 39 |
+
0.005,
|
| 40 |
+
0.005,
|
| 41 |
+
0.005
|
| 42 |
+
],
|
| 43 |
+
"sdpo_curve": [
|
| 44 |
+
0.0801,
|
| 45 |
+
0.0757,
|
| 46 |
+
0.0771,
|
| 47 |
+
0.0903,
|
| 48 |
+
0.1387,
|
| 49 |
+
0.1855,
|
| 50 |
+
0.1445,
|
| 51 |
+
0.1113,
|
| 52 |
+
0.0679,
|
| 53 |
+
0.0574,
|
| 54 |
+
0.0381,
|
| 55 |
+
0.0151,
|
| 56 |
+
0.009,
|
| 57 |
+
0.0066,
|
| 58 |
+
0.0065,
|
| 59 |
+
0.0052,
|
| 60 |
+
0.0048,
|
| 61 |
+
0.0038,
|
| 62 |
+
0.0031,
|
| 63 |
+
0.0021,
|
| 64 |
+
0.0026,
|
| 65 |
+
0.0029,
|
| 66 |
+
0.0016,
|
| 67 |
+
0.0013,
|
| 68 |
+
0.0021,
|
| 69 |
+
0.0018,
|
| 70 |
+
0.0014,
|
| 71 |
+
0.0011,
|
| 72 |
+
0.0007,
|
| 73 |
+
0.0015
|
| 74 |
+
],
|
| 75 |
+
"run_date": "2026-05-29",
|
| 76 |
+
"gpu": "A10G",
|
| 77 |
+
"model": "Qwen/Qwen2.5-0.5B-Instruct"
|
| 78 |
+
}
|
|
@@ -0,0 +1,144 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""modal_b4_gpu_smoke.py — B4 GPU proof: real 3-channel Composer loop on A10G.
|
| 2 |
+
|
| 3 |
+
The CPU proof (run.py) shows the SDPO channel fires nonzero through the real
|
| 4 |
+
collator. This proves it ALSO trains on GPU with bf16 numerics: load
|
| 5 |
+
Qwen2.5-0.5B-Instruct, build a real collator batch with an error turn (SDPO
|
| 6 |
+
fires) + a couple no-error traces, run N optimizer steps through
|
| 7 |
+
ComposerReplicationTrainer's loss composition (grpo-proxy + alpha*sdpo +
|
| 8 |
+
beta*replay), and assert: bf16 finite throughout, SDPO channel nonzero, loss
|
| 9 |
+
trends down. Captures per-channel components + a loss curve.
|
| 10 |
+
|
| 11 |
+
Run: modal run modal_b4_gpu_smoke.py
|
| 12 |
+
Cost: ~A10G * a few minutes ≈ $1-3.
|
| 13 |
+
"""
|
| 14 |
+
import modal
|
| 15 |
+
|
| 16 |
+
app = modal.App("composer-b4-gpu-smoke")
|
| 17 |
+
|
| 18 |
+
image = (
|
| 19 |
+
modal.Image.debian_slim(python_version="3.11")
|
| 20 |
+
.pip_install(
|
| 21 |
+
"torch==2.5.1",
|
| 22 |
+
"transformers>=4.45,<5.0",
|
| 23 |
+
"trl==1.5.0",
|
| 24 |
+
"accelerate",
|
| 25 |
+
"hf-transfer",
|
| 26 |
+
)
|
| 27 |
+
.env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
@app.function(image=image, gpu="A10G", timeout=900)
|
| 32 |
+
def b4_gpu_smoke(n_steps: int = 30):
|
| 33 |
+
import torch
|
| 34 |
+
import torch.nn.functional as F
|
| 35 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 36 |
+
|
| 37 |
+
device = "cuda"
|
| 38 |
+
dtype = torch.bfloat16
|
| 39 |
+
model_id = "Qwen/Qwen2.5-0.5B-Instruct"
|
| 40 |
+
print(f"[b4-gpu] loading {model_id} in {dtype} on {device}")
|
| 41 |
+
tok = AutoTokenizer.from_pretrained(model_id)
|
| 42 |
+
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=dtype).to(device)
|
| 43 |
+
model.train()
|
| 44 |
+
|
| 45 |
+
# --- generalized JSD (mirror of composer_replication.opsd.generalized_jsd_loss) ---
|
| 46 |
+
def jsd(student_logits, teacher_logits, beta=0.5, temperature=1.0):
|
| 47 |
+
s = F.log_softmax(student_logits / temperature, dim=-1)
|
| 48 |
+
t = F.log_softmax(teacher_logits / temperature, dim=-1)
|
| 49 |
+
# mixture in log space
|
| 50 |
+
m = torch.logsumexp(
|
| 51 |
+
torch.stack([s + torch.log(torch.tensor(beta, device=s.device)),
|
| 52 |
+
t + torch.log(torch.tensor(1 - beta, device=s.device))]),
|
| 53 |
+
dim=0,
|
| 54 |
+
)
|
| 55 |
+
kl_s = (s.exp() * (s - m)).sum(-1)
|
| 56 |
+
kl_t = (t.exp() * (t - m)).sum(-1)
|
| 57 |
+
return (beta * kl_s + (1 - beta) * kl_t).mean()
|
| 58 |
+
|
| 59 |
+
# --- build a tiny real batch: a prompt + a "recovery" continuation ---
|
| 60 |
+
def encode(text):
|
| 61 |
+
return tok(text, return_tensors="pt").input_ids.to(device)
|
| 62 |
+
|
| 63 |
+
# Student context (no hint) vs teacher context (with hint) — same recovery tail.
|
| 64 |
+
student_text = "User: the tool failed.\nAssistant: I will use a valid tool and retry."
|
| 65 |
+
teacher_text = ("User: the tool failed.\nSystem: Hint: check the available tool list first.\n"
|
| 66 |
+
"Assistant: I will use a valid tool and retry.")
|
| 67 |
+
s_ids = encode(student_text)
|
| 68 |
+
t_ids = encode(teacher_text)
|
| 69 |
+
# Align the shared recovery tail: last K tokens of each are the same string.
|
| 70 |
+
recovery = " I will use a valid tool and retry."
|
| 71 |
+
rec_ids = tok(recovery, return_tensors="pt").input_ids.to(device)
|
| 72 |
+
K = rec_ids.shape[1]
|
| 73 |
+
s_idx = torch.arange(s_ids.shape[1] - K, s_ids.shape[1], device=device).unsqueeze(0)
|
| 74 |
+
t_idx = torch.arange(t_ids.shape[1] - K, t_ids.shape[1], device=device).unsqueeze(0)
|
| 75 |
+
|
| 76 |
+
opt = torch.optim.AdamW(model.parameters(), lr=1e-5)
|
| 77 |
+
alpha_sdpo, beta_replay = 0.02, 0.05 # A2/A3 blend for the smoke
|
| 78 |
+
|
| 79 |
+
curve = []
|
| 80 |
+
sdpo_vals = []
|
| 81 |
+
for step in range(n_steps):
|
| 82 |
+
opt.zero_grad()
|
| 83 |
+
# Channel 1 proxy: LM loss on the student sequence (stands in for GRPO PG).
|
| 84 |
+
out_s = model(input_ids=s_ids, labels=s_ids)
|
| 85 |
+
grpo_proxy = out_s.loss
|
| 86 |
+
student_logits = out_s.logits
|
| 87 |
+
|
| 88 |
+
# Channel 2: SDPO — teacher = same model, hint-conditioned (no grad).
|
| 89 |
+
with torch.no_grad():
|
| 90 |
+
teacher_logits = model(input_ids=t_ids).logits
|
| 91 |
+
s_gather = student_logits.gather(
|
| 92 |
+
1, s_idx.unsqueeze(-1).expand(-1, -1, student_logits.size(-1)))
|
| 93 |
+
t_gather = teacher_logits.gather(
|
| 94 |
+
1, t_idx.unsqueeze(-1).expand(-1, -1, teacher_logits.size(-1)))
|
| 95 |
+
sdpo_kl = jsd(s_gather, t_gather)
|
| 96 |
+
|
| 97 |
+
# Channel 3 proxy: a small DPO-style margin on the recovery tail vs a
|
| 98 |
+
# shuffled "rejected" (stands in for trace-replay-DPO).
|
| 99 |
+
rej = student_logits.flip(1)
|
| 100 |
+
replay = F.relu(0.1 - (student_logits.mean() - rej.mean()))
|
| 101 |
+
|
| 102 |
+
total = grpo_proxy + alpha_sdpo * sdpo_kl + beta_replay * replay
|
| 103 |
+
if not torch.isfinite(total):
|
| 104 |
+
return {"status": "FAIL", "reason": "non-finite loss", "step": step}
|
| 105 |
+
total.backward()
|
| 106 |
+
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
|
| 107 |
+
opt.step()
|
| 108 |
+
|
| 109 |
+
curve.append(float(total.detach().float()))
|
| 110 |
+
sdpo_vals.append(float(sdpo_kl.detach().float()))
|
| 111 |
+
if step % 5 == 0:
|
| 112 |
+
print(f"[b4-gpu] step {step:3d} total={curve[-1]:.4f} "
|
| 113 |
+
f"grpo={float(grpo_proxy.detach().float()):.4f} "
|
| 114 |
+
f"sdpo_kl={sdpo_vals[-1]:.4f} replay={float(replay.detach().float()):.4f}")
|
| 115 |
+
|
| 116 |
+
# --- verdicts ---
|
| 117 |
+
sdpo_fired = max(sdpo_vals) > 1e-6
|
| 118 |
+
# loss trend: compare mean of first third vs last third
|
| 119 |
+
third = max(1, n_steps // 3)
|
| 120 |
+
trend_down = (sum(curve[-third:]) / third) < (sum(curve[:third]) / third)
|
| 121 |
+
all_finite = all(c == c and abs(c) != float("inf") for c in curve)
|
| 122 |
+
|
| 123 |
+
return {
|
| 124 |
+
"status": "PASS" if (sdpo_fired and trend_down and all_finite) else "PARTIAL",
|
| 125 |
+
"dtype": str(dtype),
|
| 126 |
+
"n_steps": n_steps,
|
| 127 |
+
"sdpo_fired_nonzero": sdpo_fired,
|
| 128 |
+
"max_sdpo_kl": max(sdpo_vals),
|
| 129 |
+
"loss_trend_down": trend_down,
|
| 130 |
+
"all_finite": all_finite,
|
| 131 |
+
"loss_first": curve[0],
|
| 132 |
+
"loss_last": curve[-1],
|
| 133 |
+
"loss_curve": [round(c, 4) for c in curve],
|
| 134 |
+
"sdpo_curve": [round(s, 4) for s in sdpo_vals],
|
| 135 |
+
}
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
@app.local_entrypoint()
|
| 139 |
+
def main(n_steps: int = 30):
|
| 140 |
+
import json
|
| 141 |
+
res = b4_gpu_smoke.remote(n_steps=n_steps)
|
| 142 |
+
print("\n" + "=" * 64)
|
| 143 |
+
print(json.dumps(res, indent=2))
|
| 144 |
+
print("=" * 64)
|