README_benchmark.md

# SF-Cluster benchmark reproduction (CPU-only)

Self-contained bundle that reproduces the headline **`minority_hit_rate`** table
of SF-Cluster: how often each MSA-subsetting method recovers the *minority /
rare* conformational state of a fold-switching protein, scored against locked
reference structures. No GPU required — all AlphaFold2 predictions are shipped.

## What it reproduces

Per (method arm x case) fraction of predictions that pass the §9.1 hit criterion
for the minority state. Cases: **KaiB**, **GA98**, **GB98**. Arms: four SF
methods (`mosaic_raw`, `gradient_raw`, `contrast_raw`, `region_cluster_raw`) plus
the `afcluster` (AF-Cluster / DBSCAN) baseline.

### Expected table (published targets, tolerance +/-0.02)

| arm                | KaiB   | GA98   | GB98   |
|--------------------|--------|--------|--------|
| mosaic_raw         | 0.9500 | 0.9250 | 0.1875 |
| gradient_raw       | 0.6500 | 0.5000 | 0.5000 |
| contrast_raw       | 0.6750 | 0.4500 | 0.0750 |
| region_cluster_raw | 0.5375 | 0.5000 | 0.4500 |
| afcluster          | 0.3625 | 0.4625 | 0.4375 |

KaiB/GA98/GB98 SF arms are n=80 predictions each; afcluster KaiB is n=320,
GA98/GB98 n=80.

## Hit criterion (protocol §9.1, all three must hold)

For the minority reference state:

1. C-alpha RMSD <= 3.0 A on `common_core` residues (Biopython `Superimposer`).
2. Mean pLDDT >= 70 overall.
3. Mean pLDDT >= 70 in `switch_region` (3 A displacement region).

The authoritative flag is the `<state>__hit_primary` column in `evals.tsv`.
`minority_hit_rate` = mean of that column. Minority state per case:

| Case | Minority-state column      | Note |
|------|----------------------------|------|
| KaiB | `state_A_2QKE__hit_primary`  | fold-switched 2QKE chain B |
| GA98 | `GA98_2LHC__hit_primary`     | alternate GA-fold basin |
| GB98 | `GA98_2LHC__hit_primary`     | GA98 fold is the minority conformation for GB98 |

## GA_GB CLI caveat

The scorer's `--case` accepts only `KaiB`, `GA_GB`, `Mpt53`. **Both GA98 and GB98
are scored as `--case GA_GB`**; they differ only in which reference RMSD column is
read. Consequently GB98's minority state is the `GA98_2LHC` column (the alternate
fold), not its own native `GB98_2LHD` column.

## Dependencies

Pip-only: `numpy`, `biopython`, `pyyaml`. No conda, no GPU.

```bash
pip install numpy biopython pyyaml
```

`TMalign` is **optional**: if present on `PATH` it fills the `tmalign_*` columns;
if absent those columns are `NA`. It is **not** part of the hit criterion, so the
reproduction is unaffected.

## Commands

Run from anywhere (bundle root is resolved relative to `reproduce_benchmark.py`):

```bash
# Fast path: aggregate the shipped per-prediction evals.tsv and assert targets.
python reproduce_benchmark.py --mode precomputed

# Full path: re-run the scorer over the shipped PDBs, regenerate evals.tsv,
# then aggregate and assert. Slower (parses ~1440 PDBs) but fully from scratch.
python reproduce_benchmark.py --mode score
```

Both print the case x arm table and exit non-zero if any value is outside
+/-0.02 of the published target.

Ad-hoc aggregation of an arbitrary bench root:

```bash
python eval/aggregate_hits.py --bench-root ./bench
```

## Layout

```
reproduce_benchmark.py        top-level driver (precomputed | score)
README_benchmark.md           this file
eval/
  evaluate_prediction.py      per-prediction scorer (§9.1 hit_primary)
  batch_eval.py               batch driver -> evals.tsv
  aggregate_hits.py           evals.tsv -> minority_hit_rate table
bench/
  configs/cases.yaml          case metadata (construct_start offsets)
  data/manifests/eval_regions.tsv   common_core / switch_region residue lists
  data/processed/structures/  4 reference PDBs (KaiB state_A/B, GA98, GB98)
  results/baseline_p12/<arm>/<case>/refine_per_state/   80 PDBs + evals.tsv
  results/baseline/afcluster/<case>/refine_per_state/   PDBs + evals.tsv
```

The scorer resolves its data/config root from `SF_BENCH_ROOT` (set automatically
by `reproduce_benchmark.py --mode score`), falling back to the repo layout.