chq1155 commited on
Commit
6146d9b
·
verified ·
1 Parent(s): d1b1033

Remove old README_universal_v2.md (renamed to main_benchmark)

Browse files
Files changed (1) hide show
  1. README_universal_v2.md +0 -50
README_universal_v2.md DELETED
@@ -1,50 +0,0 @@
1
- # Universal_v2 benchmark — scoring scripts + reference structures
2
-
3
- Two-state / multi-conformer benchmark of 60 cases across four patterns
4
- (FS fold-switch, AL allosteric, ID disorder-to-order, OL oligomer), 6 MSA-subsampling
5
- methods each. This package ships the **scoring code** and the **per-case reference
6
- structures + region definitions** so predictions can be scored and the benchmark
7
- tables regenerated on CPU. The AlphaFold2 prediction PDBs (~16 GB) are **not**
8
- distributed here — bring your own, or score the reference set.
9
-
10
- ## Contents
11
-
12
- - `eval/universal_v2_evaluate.py` — per-prediction scorer. For each state (a, b):
13
- Cα RMSD on the common core, optional TM-score, and `hit_primary` (§9.1 binding
14
- 3-condition: RMSD ≤ 3 Å on common core AND mean pLDDT ≥ 70 AND switch-region
15
- pLDDT ≥ 70). Pure CPU: `numpy`, `biopython`, `pyyaml`. TMalign optional (extra
16
- columns only).
17
- - `eval/aggregate_universal_v2.py` — aggregate per-(case, method) `evals.tsv` into
18
- `main_table.csv` (per case×method hit counts/rates) and `pattern_summary.csv`
19
- (per pattern×method means).
20
- - `bench/universal_v2/structures/<case_id>/{state_a,state_b}.pdb` — 60 cases, 120 references.
21
- - `bench/universal_v2/annotations/<case_id>/state_region_FINAL.tsv` — common-core and
22
- switch-region residue indices (evaluator input only; oracle annotation, not a feature).
23
- - `bench/universal_v2/cases.yaml` — case metadata.
24
-
25
- ## Usage
26
-
27
- ```bash
28
- pip install numpy biopython pyyaml # TMalign optional
29
-
30
- # Score one case×method directory of prediction PDBs:
31
- python eval/universal_v2_evaluate.py \
32
- --case SFB_FS_A1AT_1QLPA_1OPHA \
33
- --root /path/to/predictions/<case>/<method>/refine_per_state \
34
- --out evals.tsv
35
-
36
- # Aggregate a full results tree (SF_UV2_RESULTS/<case>/<method>/refine_per_state/evals.tsv):
37
- SF_UV2_RESULTS=/path/to/results python eval/aggregate_universal_v2.py
38
- ```
39
-
40
- `SF_UV2_DATA` defaults to the bundled `bench/universal_v2` (structures + annotations +
41
- cases.yaml); override to point elsewhere. `SF_UV2_RESULTS` supplies the predictions tree.
42
-
43
- ## Verification
44
-
45
- The scorer is deterministic. Re-scoring the internal prediction set reproduces the
46
- published hit flags exactly (2594/2594 predictions across a 12-case stratified sample;
47
- `main_table` → `pattern_summary` aggregation reproduces 24/24 rows; `main_table` hit
48
- counts reproduce from `evals.tsv` 360/360). Headline: mosaic-SF beats AF-Cluster by
49
- ~5–18 pp across patterns but is statistically indistinguishable from depth-matched
50
- random subsampling.