README_main_benchmark.md

# Main_benchmark benchmark — scoring scripts + reference structures

Two-state / multi-conformer benchmark of 60 cases across four patterns
(FS fold-switch, AL allosteric, ID disorder-to-order, OL oligomer), 6 MSA-subsampling
methods each. This package ships the **scoring code** and the **per-case reference
structures + region definitions** so predictions can be scored and the benchmark
tables regenerated on CPU. The AlphaFold2 prediction PDBs (~16 GB) are **not**
distributed here — bring your own, or score the reference set.

## Contents

- `eval/main_benchmark_evaluate.py` — per-prediction scorer. For each state (a, b):
  Cα RMSD on the common core, optional TM-score, and `hit_primary` (§9.1 binding
  3-condition: RMSD ≤ 3 Å on common core AND mean pLDDT ≥ 70 AND switch-region
  pLDDT ≥ 70). Pure CPU: `numpy`, `biopython`, `pyyaml`. TMalign optional (extra
  columns only).
- `eval/aggregate_main_benchmark.py` — aggregate per-(case, method) `evals.tsv` into
  `main_table.csv` (per case×method hit counts/rates) and `pattern_summary.csv`
  (per pattern×method means).
- `bench/main_benchmark/structures/<case_id>/{state_a,state_b}.pdb` — 60 cases, 120 references.
- `bench/main_benchmark/annotations/<case_id>/state_region_FINAL.tsv` — common-core and
  switch-region residue indices (evaluator input only; oracle annotation, not a feature).
- `bench/main_benchmark/cases.yaml` — case metadata.

## Usage

```bash
pip install numpy biopython pyyaml   # TMalign optional

# Score one case×method directory of prediction PDBs:
python eval/main_benchmark_evaluate.py \
    --case SFB_FS_A1AT_1QLPA_1OPHA \
    --root /path/to/predictions/<case>/<method>/refine_per_state \
    --out evals.tsv

# Aggregate a full results tree (SF_MAIN_BENCH_RESULTS/<case>/<method>/refine_per_state/evals.tsv):
SF_MAIN_BENCH_RESULTS=/path/to/results python eval/aggregate_main_benchmark.py
```

`SF_MAIN_BENCH_DATA` defaults to the bundled `bench/main_benchmark` (structures + annotations +
cases.yaml); override to point elsewhere. `SF_MAIN_BENCH_RESULTS` supplies the predictions tree.

## Verification

The scorer is deterministic. Re-scoring the internal prediction set reproduces the
published hit flags exactly (2594/2594 predictions across a 12-case stratified sample;
`main_table` → `pattern_summary` aggregation reproduces 24/24 rows; `main_table` hit
counts reproduce from `evals.tsv` 360/360). Headline: mosaic-SF beats AF-Cluster by
~5–18 pp across patterns but is statistically indistinguishable from depth-matched
random subsampling.