Remove old README_universal_v2.md (renamed to main_benchmark)
Browse files- README_universal_v2.md +0 -50
README_universal_v2.md
DELETED
|
@@ -1,50 +0,0 @@
|
|
| 1 |
-
# Universal_v2 benchmark — scoring scripts + reference structures
|
| 2 |
-
|
| 3 |
-
Two-state / multi-conformer benchmark of 60 cases across four patterns
|
| 4 |
-
(FS fold-switch, AL allosteric, ID disorder-to-order, OL oligomer), 6 MSA-subsampling
|
| 5 |
-
methods each. This package ships the **scoring code** and the **per-case reference
|
| 6 |
-
structures + region definitions** so predictions can be scored and the benchmark
|
| 7 |
-
tables regenerated on CPU. The AlphaFold2 prediction PDBs (~16 GB) are **not**
|
| 8 |
-
distributed here — bring your own, or score the reference set.
|
| 9 |
-
|
| 10 |
-
## Contents
|
| 11 |
-
|
| 12 |
-
- `eval/universal_v2_evaluate.py` — per-prediction scorer. For each state (a, b):
|
| 13 |
-
Cα RMSD on the common core, optional TM-score, and `hit_primary` (§9.1 binding
|
| 14 |
-
3-condition: RMSD ≤ 3 Å on common core AND mean pLDDT ≥ 70 AND switch-region
|
| 15 |
-
pLDDT ≥ 70). Pure CPU: `numpy`, `biopython`, `pyyaml`. TMalign optional (extra
|
| 16 |
-
columns only).
|
| 17 |
-
- `eval/aggregate_universal_v2.py` — aggregate per-(case, method) `evals.tsv` into
|
| 18 |
-
`main_table.csv` (per case×method hit counts/rates) and `pattern_summary.csv`
|
| 19 |
-
(per pattern×method means).
|
| 20 |
-
- `bench/universal_v2/structures/<case_id>/{state_a,state_b}.pdb` — 60 cases, 120 references.
|
| 21 |
-
- `bench/universal_v2/annotations/<case_id>/state_region_FINAL.tsv` — common-core and
|
| 22 |
-
switch-region residue indices (evaluator input only; oracle annotation, not a feature).
|
| 23 |
-
- `bench/universal_v2/cases.yaml` — case metadata.
|
| 24 |
-
|
| 25 |
-
## Usage
|
| 26 |
-
|
| 27 |
-
```bash
|
| 28 |
-
pip install numpy biopython pyyaml # TMalign optional
|
| 29 |
-
|
| 30 |
-
# Score one case×method directory of prediction PDBs:
|
| 31 |
-
python eval/universal_v2_evaluate.py \
|
| 32 |
-
--case SFB_FS_A1AT_1QLPA_1OPHA \
|
| 33 |
-
--root /path/to/predictions/<case>/<method>/refine_per_state \
|
| 34 |
-
--out evals.tsv
|
| 35 |
-
|
| 36 |
-
# Aggregate a full results tree (SF_UV2_RESULTS/<case>/<method>/refine_per_state/evals.tsv):
|
| 37 |
-
SF_UV2_RESULTS=/path/to/results python eval/aggregate_universal_v2.py
|
| 38 |
-
```
|
| 39 |
-
|
| 40 |
-
`SF_UV2_DATA` defaults to the bundled `bench/universal_v2` (structures + annotations +
|
| 41 |
-
cases.yaml); override to point elsewhere. `SF_UV2_RESULTS` supplies the predictions tree.
|
| 42 |
-
|
| 43 |
-
## Verification
|
| 44 |
-
|
| 45 |
-
The scorer is deterministic. Re-scoring the internal prediction set reproduces the
|
| 46 |
-
published hit flags exactly (2594/2594 predictions across a 12-case stratified sample;
|
| 47 |
-
`main_table` → `pattern_summary` aggregation reproduces 24/24 rows; `main_table` hit
|
| 48 |
-
counts reproduce from `evals.tsv` 360/360). Headline: mosaic-SF beats AF-Cluster by
|
| 49 |
-
~5–18 pp across patterns but is statistically indistinguishable from depth-matched
|
| 50 |
-
random subsampling.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|