README_main_benchmark.md

Main_benchmark benchmark — scoring scripts + reference structures

Two-state / multi-conformer benchmark of 60 cases across four patterns (FS fold-switch, AL allosteric, ID disorder-to-order, OL oligomer), 6 MSA-subsampling methods each. This package ships the scoring code and the per-case reference structures + region definitions so predictions can be scored and the benchmark tables regenerated on CPU. The AlphaFold2 prediction PDBs (~16 GB) are not distributed here — bring your own, or score the reference set.

Contents

• eval/main_benchmark_evaluate.py — per-prediction scorer. For each state (a, b): Cα RMSD on the common core, optional TM-score, and hit_primary (§9.1 binding 3-condition: RMSD ≤ 3 Å on common core AND mean pLDDT ≥ 70 AND switch-region pLDDT ≥ 70). Pure CPU: numpy, biopython, pyyaml. TMalign optional (extra columns only).
• eval/aggregate_main_benchmark.py — aggregate per-(case, method) evals.tsv into main_table.csv (per case×method hit counts/rates) and pattern_summary.csv (per pattern×method means).
• bench/main_benchmark/structures/<case_id>/{state_a,state_b}.pdb — 60 cases, 120 references.
• bench/main_benchmark/annotations/<case_id>/state_region_FINAL.tsv — common-core and switch-region residue indices (evaluator input only; oracle annotation, not a feature).
• bench/main_benchmark/cases.yaml — case metadata.

Usage

pip install numpy biopython pyyaml   # TMalign optional

# Score one case×method directory of prediction PDBs:
python eval/main_benchmark_evaluate.py \
    --case SFB_FS_A1AT_1QLPA_1OPHA \
    --root /path/to/predictions/<case>/<method>/refine_per_state \
    --out evals.tsv

# Aggregate a full results tree (SF_MAIN_BENCH_RESULTS/<case>/<method>/refine_per_state/evals.tsv):
SF_MAIN_BENCH_RESULTS=/path/to/results python eval/aggregate_main_benchmark.py

SF_MAIN_BENCH_DATA defaults to the bundled bench/main_benchmark (structures + annotations + cases.yaml); override to point elsewhere. SF_MAIN_BENCH_RESULTS supplies the predictions tree.

Verification

The scorer is deterministic. Re-scoring the internal prediction set reproduces the published hit flags exactly (2594/2594 predictions across a 12-case stratified sample; main_table → pattern_summary aggregation reproduces 24/24 rows; main_table hit counts reproduce from evals.tsv 360/360). Headline: mosaic-SF beats AF-Cluster by ~5–18 pp across patterns but is statistically indistinguishable from depth-matched random subsampling.