shutdown handoff: SHUTDOWN_HANDOFF.md

SHUTDOWN_HANDOFF.md

@@ -0,0 +1,141 @@
+# DNAThinker server3 → lab handoff (server3 shutdown 2026-05-07)
+
+server3 (the H100/H200 box that ran the multi-seed reasoning RL +
+production-route MoE eval + paper iteration) is being shut down
+permanently. **The paper is at 56 pages, all paper-grade artifacts
+mirrored to HuggingFace, all commits pushed to GitHub.** This doc
+tells the lab cluster how to keep going.
+
+---
+
+## TL;DR for the lab
+
+1. `git fetch origin && git checkout mllm-integrate-server3 && git pull` —
+   server3's branch has the paper (56 pp, multi-seed n=200/n=1000,
+   §app:rqsamples, edit-tight derivation, dnaseq autowrap macro).
+2. Optionally merge `mllm-integrate-server3` → `mllm-integrate` for
+   trunk continuity (no conflicts expected — the branches diverged
+   only in main.tex content, with server3 being the superset).
+3. **Restart the HF auto-uploader** on a long-running lab box. It is
+   the only continuously-running service we need to preserve. See
+   §"HF auto-uploader" below.
+4. There is **NO blocked or in-flight server3 work that needs lab
+   pickup**. The cycle 70zz49 T2 rebalance was rate-limited at
+   25920 rows (sufficient for §sec:rqrl); cycles 70zz44–48 finished;
+   nothing is mid-train.
+
+If you only do step 1, the paper still compiles and reproduces. The
+HF mirror at the URLs in §"Where everything is" is the source of
+truth for any artifact server3 produced.
+
+---
+
+## Where everything is (HuggingFace, all public)
+
+| Repo | What | Notes |
+|---|---|---|
+| `explcre/dnathinker_t1_v3` | T1 paper-headline trunk + fusion-SFT base | `exp_t1_fusion_sft_unified_20260428_h100/best.pt` (6.5 GB) **uploaded 2026-05-07 09:07 UTC during shutdown prep** |
+| `explcre/dnathinker_t2_dual_xa` | T2 dual-XA Variant C + 600M overfit ablation | `exp_t2_ntv3_dual_xa_FULLFT600M_natural3m_20260504_063140/best.pt` (5.4 GB) **uploaded 2026-05-07 09:08 UTC during shutdown prep** |
+| `explcre/dnathinker_t3_trunks` | T3 edit-tight 500-bp Path-A production trunk | `exp_t3_edit_tight_20260505/best.pt` + 360-bp baseline `exp_t3_ntv3_100m_rich_cond_editing_20260503_222416/best.pt` |
+| `explcre/ntv3_rich_cond_mdlm_phase4_family` | T1 trunks (v3 paper-headline + v5full15m + others) | ~7 GB |
+| `explcre/phase7_multitask` | Joint MoE LoRAs (T1+T2+T3 r=64) — **mode-collapsed**, ablation only | ~150 MB; paper §sec:joinmoe-collapse |
+| `explcre/phase8_rl` | All Phase-8 RL ckpts incl. multi-seed reasoning RL, FIXED grid, KD/BIGLR | ~5 GB; **422 files, last sync 2026-05-07 09:06 UTC** |
+| `explcre/phase8_rl/_paper_results/` | 9 paper-grade tables/CIs (FID, embed-cos, TFG, multiseed) | Reproducible from raw logs in same repo |
+| `explcre/phase5_stage_a` / `phase5_stage_b` | Phase-5 SFT bases | varies |
+| `explcre/phase6_grpo` | Phase-6 GRPO RL | varies |
+
+A combined download command for the must-haves:
+
+```bash
+huggingface-cli download explcre/dnathinker_t3_trunks --include "exp_t3_edit_tight_20260505/*" --local-dir ./_dl_t3
+huggingface-cli download explcre/dnathinker_t2_dual_xa --include "exp_t1_fusion_sft_unified_20260428_h100/*" "exp_t2_ntv3_dual_xa_FULLFT600M_natural3m_20260504_063140/*" --local-dir ./_dl_t1t2
+huggingface-cli download explcre/phase8_rl --include "_paper_results/*" "exp_phase8_reasoning_grounded_rl_*/best.pt" "exp_phase8_reasoning_grounded_rl_*/manifest.json" --local-dir ./_dl_p8
+```
+
+---
+
+## HF auto-uploader (the only running service)
+
+The auto-uploader continuously mirrors `runs/_paper_results/`,
+`runs/exp_phase8_reasoning_*`, etc. to the HF repos above. It is
+the safety net for any new artifact you produce.
+
+**Where:** `regureasoner_loop/scripts/innovations/hf_auto_uploader.py`
+**Wrapper:** `regureasoner_loop/slurm/run_hf_auto_uploader.sh`
+**Watchlist:** at the bottom of `hf_auto_uploader.py` — 60 watches
+covering every `runs/exp_*` glob the paper cares about.
+
+**To restart on a lab box** (any box with HF token + the dnathinker
+runs dir mounted):
+
+```bash
+cd $REPO_ROOT/regureasoner_loop
+nohup bash slurm/run_hf_auto_uploader.sh \
+    > /workspace/dnathinker/runs/_bench_logs/hf_auto_uploader.daemon.log 2>&1 &
+# verify:
+sleep 5 && tail /workspace/dnathinker/runs/_bench_logs/hf_auto_uploader.log
+# expected: "INFO start hf_auto_uploader · interval=30m · watches=60"
+```
+
+If the lab cluster's `runs/` path differs from
+`/workspace/dnathinker/runs/`, edit `Watch.local_dir` at the bottom
+of `hf_auto_uploader.py` to match.
+
+---
+
+## Paper compile (sanity check after pulling)
+
+```bash
+cd regureasoner_loop/paper
+rm -f main.aux main.log main.bbl main.blg main.toc main.out
+pdflatex -interaction=nonstopmode main.tex
+bibtex main
+pdflatex -interaction=nonstopmode main.tex
+pdflatex -interaction=nonstopmode main.tex
+# expected: "Output written on main.pdf (56 pages, ~5.1 MB)"
+```
+
+All required figures are tracked in `paper/figures/`. The
+`\dnaseq` macro (defined at the top of main.tex) auto-wraps long
+DNA listings at the page boundary.
+
+---
+
+## What server3 was working on, status snapshot
+
+| Cycle | Task | Status | Artifact location |
+|---|---|---|---|
+| 70zz44 | Production-route MoE end-to-end eval | DONE | `explcre/phase8_rl/_paper_results/cycle_70zz44_*` |
+| 70zz45 | Joint MoE end-task scoring | DONE | mirrored, paper §sec:joinmoe-collapse |
+| 70zz46 | T1 reasoning RL multi-seed (s=2,3) | DONE | `explcre/phase8_rl/exp_phase8_reasoning_grounded_rl_t1_*` |
+| 70zz47 | T2/T3 reasoning RL multi-seed (s=2,3) | DONE | ditto, T2 t3 dirs |
+| 70zz48 | T3 reasoning RL parallel | DONE | mirrored |
+| 70zz49 | T2 reasoning rebalance (prod_samples balanced) | RATE-LIMITED at 25920 rows | sufficient for paper, §sec:rqrl unaffected |
+
+Multi-seed reasoning RL aggregator output:
+`_paper_results/reasoning_rl_multiseed_summary.{md,json}` — T1
+0.412±0.023, T2 0.324±0.051, T3 0.225±0.026 (n=3 seeds each).
+
+---
+
+## What is NOT preserved
+
+- `/shm/dnathinker_quarantine/` (intermediate ckpt staging) was on
+  tmpfs and got cleared on remount — these were intentionally
+  non-load-bearing intermediates (best.pt's already on HF). No data
+  loss for paper purposes.
+- Local `runs/` dirs that are NOT in the HF watchlist: these are
+  training intermediates (per-step ckpts before best.pt). The
+  best.pt + `manifest.json` + `log.jsonl` for every paper-cited
+  experiment is on HF.
+- Local `.env` with OPENROUTER_API_KEY_{1..N}: backed up to
+  the user's local machine via rsync per `PROJECT_RESUME_MANIFEST.md`.
+  Only needed for re-running reasoning expansion (already complete).
+
+---
+
+## If you need to resume server3-style work elsewhere
+
+Follow `regureasoner_loop/docs/PROJECT_RESUME_MANIFEST.md` — 6-step
+playbook that pulls the repo, restores Claude memory, downloads
+critical ckpts, and verifies the paper compiles.