docs/ALTERED_MINDS_TIE_IN.md

altered-minds × Composer Replication Framework

Status: Tie-in design doc. Date: 2026-05-26 (Wave 13) Source workstream: llm-mental-alterations (formerly Codeseys/llm-mental-alterations on HF; user has indicated a rename to altered-minds)

What altered-minds is studying

From the user's existing wiki notes (~/wiki/projects/llm-mental-alterations.md):

• Fine-tuning Llama-3.1-8B with personality SFT induces a depression/ anxiety cognitive-distortion signature on MMLU moral_scenarios:
  • Class 3 ("both fine") collapses −31.1pp
  • Class 0 ("both wrong") improves +4.6pp
  • Multi-seed reproducible (4/4 seeds, n=895)
  • 18% of base-correct items broken
• Other domains affected: high_school_chemistry +4.2pp, machine_learning +4.9pp (reliably improved).
• H-3 Gemma-MoE hypothesis is deferred (Hopper-only).
• Spend so far: $9.75 / $400 budget.

The headline question driving the workstream is roughly: "What measurable cognitive alterations does personality-style SFT introduce, and can we recover or sharpen them via downstream RL?"

Why this framework is the right second-stage workstream

altered-minds today is an SFT-only pipeline. A typical run:

1. Take a base model (Llama-3.1-8B).
2. Apply personality SFT.
3. Evaluate on MMLU + alteration-specific probes.
4. Document the alteration signature.

The Composer Replication Framework, by design, is a post-SFT reinforcement-learning framework. It can take any HF model — including an altered-minds-altered model — and apply:

• GRPO with verifiable rewards
• SDPO/OPSD self-distillation against the altered model's hint- conditioned forward passes
• Trace-replay DPO against N external teachers

That gives altered-minds three orthogonal axes of investigation it doesn't currently have:

Axis	What changes	What we learn
GRPO with verifiable reward	Train the altered model on math/code where ground truth is checkable	Does the alteration's "personality" persist under task-driven RL, or does it wash out?
SDPO against the altered model's own hints	Self-distillation — the altered model teaches itself with hint-conditioned forward passes	Can we sharpen the alteration without further SFT?
Trace-replay DPO with frontier teachers	The altered model rolls out, frontier teachers replay the same prompts, disagreement → DPO pairs	Where does the altered model disagree with frontier consensus? Are those disagreements correlated with the cognitive-distortion signature?

The third axis is the most interesting for altered-minds specifically. The framework's replay_trace + extract_dpo_pairs produce, by construction, a dataset of "altered-model output" vs "frontier-consensus output" for any prompt distribution. If the altered model's depression/anxiety signature shows up in moral_scenarios, then the trace-replay output on moral-scenario prompts is a measurable corpus of the alteration.

Concrete plan: altered-minds-RL spike

Phase 1 — model selection

Pick the altered-minds checkpoint that produced the strongest signature (per the user's notes: the multi-seed Llama-3.1-8B personality-SFT run where moral_scenarios class 3 collapsed −31.1pp).

Phase 2 — domain-specific replaysim

Run composer_replication.replaysim.replay_and_normalize_trace against:

• A held-out moral_scenarios test set (the alteration locus)
• A held-out high_school_chemistry test set (where altered-minds improved)
• A held-out general MMLU baseline

Teachers: framework defaults (Claude Opus 4.7, GPT-5, DeepSeek V4 Pro). This produces three normalized DPO datasets capturing where the altered model disagrees with frontier consensus on each domain.

Cost estimate: ~$0.98/trace × 100 prompts × 3 domains ≈ $300. Fits inside the user's existing $400 altered-minds budget.

Phase 3 — GRPO with the framework

Run composer_replication.recipes.trl.ComposerReplicationTrainer with:

• Channel 1 (GRPO): turned ON, reward = MMLU letter-correctness
• Channel 2 (SDPO/OPSD): turned ON at α=0.2, hint-conditioned against the altered model's own forward pass
• Channel 3 (trace-replay DPO): turned ON at β=0.4, against the Phase-2 datasets

Train for ~500 steps on a single GPU (Qwen-0.5B feasibility-test already confirmed in the framework; for Llama-8B, use Modal + the framework's ServerlessExecutor per ADR-005 — local 5090 is too small).

Phase 4 — re-evaluate

Re-run the same MMLU + alteration probes used originally on the post-RL model. Three outcomes are possible:

Outcome	Interpretation
Alteration signature persists at same magnitude	The alteration is robust to task-driven RL — useful as a lower bound on its "depth"
Alteration signature attenuates	Task-driven RL washes out personality-SFT — useful for understanding alteration brittleness
Alteration signature amplifies on channel-2-only ablation	SDPO is reinforcing the alteration; rare and significant — would be a publishable finding

Phase 5 — Decoupled DiLoCo for multi-personality experiments

Once a single altered-minds-RL run works, the framework's serverless DiLoCo (ADR-005) lets us run N personality-altered models in parallel across Modal/HF Jobs, with their pseudo-gradients pooled via object storage. This becomes the natural sweep over personality types (depression vs anxiety vs grandiose vs ...) at minimal incremental infrastructure cost.

Repo layout proposal

The Composer Replication Framework is intentionally generic. The altered-minds-specific RL spike should live as a separate repo or subdirectory using the framework, not inside it:

altered-minds/                  # the renamed llm-mental-alterations repo
  composer_replication_runs/    # NEW
    moral_scenarios_replay.py   # uses composer_replication.replaysim
    train_grpo.py               # uses composer_replication.trainer
    eval_post_rl.py             # standard altered-minds eval
  recipes/
    altered_minds.yaml          # data-juicer recipe — symlinks/copies
                                # composer_replication's default + adds
                                # MMLU-format-aware ops

The framework provides the algorithm + infrastructure. The altered-minds repo owns the experimental narrative + results.

Open questions for the user

Before we proceed to Phase 1:

1. Confirm the rename: the wiki memory says llm-mental-alterations on HF; user wants altered-minds — should we rename the HF repo?
2. Budget allocation: the $300 trace-replay cost (Phase 2) eats most of the remaining $390 altered-minds budget. Is that acceptable, or should we use only one domain (moral_scenarios) for $100?
3. GPU venue for Phase 3: 8B-model RL on single-GPU is feasible on the user's RTX 5090 (32GB) for short runs, OR we use Modal A100s for a more aggressive run. Preference?

References

• altered-minds workstream wiki: ~/wiki/projects/llm-mental-alterations.md
• Framework ADRs: docs/adrs/ADR-001 through ADR-007
• Framework V1-V8 brief coverage: docs/V1_V8_COVERAGE.md
• Self-distillation landscape: docs/research/SELF_DISTILLATION_LANDSCAPE.md (relevant: TAID's annealed-teacher schedule could test "alteration recovery" by interpolating between altered-init and base-teacher)