File size: 4,066 Bytes
f4e8048 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 | # SF-Cluster benchmark reproduction (CPU-only)
Self-contained bundle that reproduces the headline **`minority_hit_rate`** table
of SF-Cluster: how often each MSA-subsetting method recovers the *minority /
rare* conformational state of a fold-switching protein, scored against locked
reference structures. No GPU required — all AlphaFold2 predictions are shipped.
## What it reproduces
Per (method arm x case) fraction of predictions that pass the §9.1 hit criterion
for the minority state. Cases: **KaiB**, **GA98**, **GB98**. Arms: four SF
methods (`mosaic_raw`, `gradient_raw`, `contrast_raw`, `region_cluster_raw`) plus
the `afcluster` (AF-Cluster / DBSCAN) baseline.
### Expected table (published targets, tolerance +/-0.02)
| arm | KaiB | GA98 | GB98 |
|--------------------|--------|--------|--------|
| mosaic_raw | 0.9500 | 0.9250 | 0.1875 |
| gradient_raw | 0.6500 | 0.5000 | 0.5000 |
| contrast_raw | 0.6750 | 0.4500 | 0.0750 |
| region_cluster_raw | 0.5375 | 0.5000 | 0.4500 |
| afcluster | 0.3625 | 0.4625 | 0.4375 |
KaiB/GA98/GB98 SF arms are n=80 predictions each; afcluster KaiB is n=320,
GA98/GB98 n=80.
## Hit criterion (protocol §9.1, all three must hold)
For the minority reference state:
1. C-alpha RMSD <= 3.0 A on `common_core` residues (Biopython `Superimposer`).
2. Mean pLDDT >= 70 overall.
3. Mean pLDDT >= 70 in `switch_region` (3 A displacement region).
The authoritative flag is the `<state>__hit_primary` column in `evals.tsv`.
`minority_hit_rate` = mean of that column. Minority state per case:
| Case | Minority-state column | Note |
|------|----------------------------|------|
| KaiB | `state_A_2QKE__hit_primary` | fold-switched 2QKE chain B |
| GA98 | `GA98_2LHC__hit_primary` | alternate GA-fold basin |
| GB98 | `GA98_2LHC__hit_primary` | GA98 fold is the minority conformation for GB98 |
## GA_GB CLI caveat
The scorer's `--case` accepts only `KaiB`, `GA_GB`, `Mpt53`. **Both GA98 and GB98
are scored as `--case GA_GB`**; they differ only in which reference RMSD column is
read. Consequently GB98's minority state is the `GA98_2LHC` column (the alternate
fold), not its own native `GB98_2LHD` column.
## Dependencies
Pip-only: `numpy`, `biopython`, `pyyaml`. No conda, no GPU.
```bash
pip install numpy biopython pyyaml
```
`TMalign` is **optional**: if present on `PATH` it fills the `tmalign_*` columns;
if absent those columns are `NA`. It is **not** part of the hit criterion, so the
reproduction is unaffected.
## Commands
Run from anywhere (bundle root is resolved relative to `reproduce_benchmark.py`):
```bash
# Fast path: aggregate the shipped per-prediction evals.tsv and assert targets.
python reproduce_benchmark.py --mode precomputed
# Full path: re-run the scorer over the shipped PDBs, regenerate evals.tsv,
# then aggregate and assert. Slower (parses ~1440 PDBs) but fully from scratch.
python reproduce_benchmark.py --mode score
```
Both print the case x arm table and exit non-zero if any value is outside
+/-0.02 of the published target.
Ad-hoc aggregation of an arbitrary bench root:
```bash
python eval/aggregate_hits.py --bench-root ./bench
```
## Layout
```
reproduce_benchmark.py top-level driver (precomputed | score)
README_benchmark.md this file
eval/
evaluate_prediction.py per-prediction scorer (§9.1 hit_primary)
batch_eval.py batch driver -> evals.tsv
aggregate_hits.py evals.tsv -> minority_hit_rate table
bench/
configs/cases.yaml case metadata (construct_start offsets)
data/manifests/eval_regions.tsv common_core / switch_region residue lists
data/processed/structures/ 4 reference PDBs (KaiB state_A/B, GA98, GB98)
results/baseline_p12/<arm>/<case>/refine_per_state/ 80 PDBs + evals.tsv
results/baseline/afcluster/<case>/refine_per_state/ PDBs + evals.tsv
```
The scorer resolves its data/config root from `SF_BENCH_ROOT` (set automatically
by `reproduce_benchmark.py --mode score`), falling back to the repo layout.
|