README_benchmark.md

SF-Cluster benchmark reproduction (CPU-only)

Self-contained bundle that reproduces the headline minority_hit_rate table of SF-Cluster: how often each MSA-subsetting method recovers the minority / rare conformational state of a fold-switching protein, scored against locked reference structures. No GPU required — all AlphaFold2 predictions are shipped.

What it reproduces

Per (method arm x case) fraction of predictions that pass the §9.1 hit criterion for the minority state. Cases: KaiB, GA98, GB98. Arms: four SF methods (mosaic_raw, gradient_raw, contrast_raw, region_cluster_raw) plus the afcluster (AF-Cluster / DBSCAN) baseline.

Expected table (published targets, tolerance +/-0.02)

arm	KaiB	GA98	GB98
mosaic_raw	0.9500	0.9250	0.1875
gradient_raw	0.6500	0.5000	0.5000
contrast_raw	0.6750	0.4500	0.0750
region_cluster_raw	0.5375	0.5000	0.4500
afcluster	0.3625	0.4625	0.4375

KaiB/GA98/GB98 SF arms are n=80 predictions each; afcluster KaiB is n=320, GA98/GB98 n=80.

Hit criterion (protocol §9.1, all three must hold)

For the minority reference state:

1. C-alpha RMSD <= 3.0 A on common_core residues (Biopython Superimposer).
2. Mean pLDDT >= 70 overall.
3. Mean pLDDT >= 70 in switch_region (3 A displacement region).

The authoritative flag is the <state>__hit_primary column in evals.tsv. minority_hit_rate = mean of that column. Minority state per case:

Case	Minority-state column	Note
KaiB	state_A_2QKE__hit_primary	fold-switched 2QKE chain B
GA98	GA98_2LHC__hit_primary	alternate GA-fold basin
GB98	GA98_2LHC__hit_primary	GA98 fold is the minority conformation for GB98

GA_GB CLI caveat

The scorer's --case accepts only KaiB, GA_GB, Mpt53. Both GA98 and GB98 are scored as --case GA_GB; they differ only in which reference RMSD column is read. Consequently GB98's minority state is the GA98_2LHC column (the alternate fold), not its own native GB98_2LHD column.

Dependencies

Pip-only: numpy, biopython, pyyaml. No conda, no GPU.

pip install numpy biopython pyyaml

TMalign is optional: if present on PATH it fills the tmalign_* columns; if absent those columns are NA. It is not part of the hit criterion, so the reproduction is unaffected.

Commands

Run from anywhere (bundle root is resolved relative to reproduce_benchmark.py):

# Fast path: aggregate the shipped per-prediction evals.tsv and assert targets.
python reproduce_benchmark.py --mode precomputed

# Full path: re-run the scorer over the shipped PDBs, regenerate evals.tsv,
# then aggregate and assert. Slower (parses ~1440 PDBs) but fully from scratch.
python reproduce_benchmark.py --mode score

Both print the case x arm table and exit non-zero if any value is outside +/-0.02 of the published target.

Ad-hoc aggregation of an arbitrary bench root:

python eval/aggregate_hits.py --bench-root ./bench

Layout

reproduce_benchmark.py        top-level driver (precomputed | score)
README_benchmark.md           this file
eval/
  evaluate_prediction.py      per-prediction scorer (§9.1 hit_primary)
  batch_eval.py               batch driver -> evals.tsv
  aggregate_hits.py           evals.tsv -> minority_hit_rate table
bench/
  configs/cases.yaml          case metadata (construct_start offsets)
  data/manifests/eval_regions.tsv   common_core / switch_region residue lists
  data/processed/structures/  4 reference PDBs (KaiB state_A/B, GA98, GB98)
  results/baseline_p12/<arm>/<case>/refine_per_state/   80 PDBs + evals.tsv
  results/baseline/afcluster/<case>/refine_per_state/   PDBs + evals.tsv

The scorer resolves its data/config root from SF_BENCH_ROOT (set automatically by reproduce_benchmark.py --mode score), falling back to the repo layout.