sunshinepku commited on
Commit
78fe476
Β·
verified Β·
1 Parent(s): 89a4f06

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -14
README.md CHANGED
@@ -35,30 +35,30 @@ tags:
35
  ---
36
 
37
  ## πŸ“’ Latest News
38
- * πŸ›‘οΈ **[2026-07]** **Data Compliance & Robustness Upgrade!** We have rigorously refined our corpus to ensure 100% open-source compliance, resulting in a high-fidelity foundational collection of 22,413 human-written protocols[cite: 2].
39
  * ✨ **[2026-03]** **Data Split Update!** We have officially released the **Train/Test splits** for each task (PQA, ORD, ERR, GEN, REA), making it easier for the community to train and evaluate models consistently.
40
  * πŸ”₯ **[2026-03]** Our **BioProAgent** is now live on AI4S LAB! [Try it out and order wet-lab experiments here](https://yuyangsunshine.github.io/BioPro-Project/).
41
  * πŸŽ‰ **[2026-03]** Our BioProAgent has been accepted by the **ICML 2026 LLA Workshop!**
42
- * πŸ“ **[2026-01]** BioProBench paper has been updated with new experimental results on frontier models (GPT-5.4, Gemini 3.1 Pro, Claude 4.6 Opus)[cite: 2]. Read the latest on [arXiv](https://arxiv.org/pdf/2505.07889).
43
 
44
  ---
45
 
46
  ## 🌟 Introduction
47
 
48
- The realization of autonomous scientific experimentation is currently limited by LLMs' struggle to grasp the strict procedural logic and accuracy required by biological protocols[cite: 2].
49
 
50
- Enter **BioProBench**β€”the first large-scale, integrated multi-task benchmark dedicated specifically to procedural reasoning in biological contexts[cite: 2]. Moving beyond basic declarative knowledge and simple QA, BioProBench is designed to test the structured, causal, and conditional logic required for true scientific automation[cite: 2].
51
 
52
  <div align="center">
53
  <img src="https://github.com/YuyangSunshine/bioprotocolbench/blob/main/figures/overview.png?raw=true" alt="BioProBench Overview" width="1000"/>
54
  </div>
55
 
56
  ### πŸš€ Key Features
57
- * πŸ“š **Unprecedented Scale:** Grounded in **22,413** professionally authored, human-written protocols[cite: 2].
58
- * 🧩 **Rich Task Diversity:** Systematically constructed into a dataset of **523,784** structured task instances[cite: 2].
59
- * 🎯 **5 Core Procedural Tasks:** Covers Protocol Question Answering (**PQA**), Step Ordering (**ORD**), Error Correction (**ERR**), Protocol Generation (**GEN**), and Protocol Reasoning (**REA**)[cite: 2].
60
- * 🧬 **Broad Domain Coverage:** Spans **16 biological subfields** sourced from **5 authoritative repositories**[cite: 2].
61
- * πŸ”¬ **Standardized Evaluation:** Features a rigorous benchmark utilizing novel, domain-specific metrics (e.g., Step Recall, Step Precision) to accurately quantify structural fidelity[cite: 2].
62
 
63
  ---
64
 
@@ -68,7 +68,7 @@ Enter **BioProBench**β€”the first large-scale, integrated multi-task benchmark d
68
  <img src="https://github.com/YuyangSunshine/bioprotocolbench/blob/main/figures/samples.jpg?raw=true" alt="BioProBench Samples" width="1000"/>
69
  </div>
70
 
71
- We provide standardized JSON files for each task, meticulously partitioned into **Train** and **Test** splits to facilitate model fine-tuning and rigorous evaluation[cite: 2]:
72
 
73
  | Task Identifier | Core Objective | Available Files |
74
  | :--- | :--- | :--- |
@@ -85,10 +85,10 @@ We provide standardized JSON files for each task, meticulously partitioned into
85
 
86
  ## πŸ”¬ Key Findings from Frontier LLMs
87
 
88
- We evaluated 10 mainstream state-of-the-art LLMs, alongside a suite of the latest 2026 frontier models[cite: 2]. Our rigorous evaluation reveals:
89
- * **The Comprehension Plateau:** While top-tier models excel at basic declarative comprehension, performance degrades significantly on tasks demanding deep reasoning, quantitative precision, and safety awareness[cite: 2].
90
- * **The Generation Bottleneck:** Models struggle profoundly with long-form procedural generation (GEN task). Even the most advanced models omit necessary experimental steps, underscoring that open-ended procedural planning remains a formidable challenge[cite: 2].
91
- * **The Power of Grounding:** To mitigate these issues, we introduce **ProAgent**. Grounded in our BioProCorpus via a Retrieval-Augmented Generation (RAG) framework, ProAgent substantially advances the state-of-the-art, significantly improving procedural step recall and reasoning accuracy[cite: 2].
92
 
93
  ---
94
 
 
35
  ---
36
 
37
  ## πŸ“’ Latest News
38
+ * πŸ›‘οΈ **[2026-07]** **Data Compliance & Robustness Upgrade!** We have rigorously refined our corpus to ensure 100% open-source compliance, resulting in a high-fidelity foundational collection of 22,413 human-written protocols.
39
  * ✨ **[2026-03]** **Data Split Update!** We have officially released the **Train/Test splits** for each task (PQA, ORD, ERR, GEN, REA), making it easier for the community to train and evaluate models consistently.
40
  * πŸ”₯ **[2026-03]** Our **BioProAgent** is now live on AI4S LAB! [Try it out and order wet-lab experiments here](https://yuyangsunshine.github.io/BioPro-Project/).
41
  * πŸŽ‰ **[2026-03]** Our BioProAgent has been accepted by the **ICML 2026 LLA Workshop!**
42
+ * πŸ“ **[2026-01]** BioProBench paper has been updated with new experimental results on frontier models (GPT-5.4, Gemini 3.1 Pro, Claude 4.6 Opus). Read the latest on [arXiv](https://arxiv.org/pdf/2505.07889).
43
 
44
  ---
45
 
46
  ## 🌟 Introduction
47
 
48
+ The realization of autonomous scientific experimentation is currently limited by LLMs' struggle to grasp the strict procedural logic and accuracy required by biological protocols.
49
 
50
+ Enter **BioProBench**β€”the first large-scale, integrated multi-task benchmark dedicated specifically to procedural reasoning in biological contexts. Moving beyond basic declarative knowledge and simple QA, BioProBench is designed to test the structured, causal, and conditional logic required for true scientific automation.
51
 
52
  <div align="center">
53
  <img src="https://github.com/YuyangSunshine/bioprotocolbench/blob/main/figures/overview.png?raw=true" alt="BioProBench Overview" width="1000"/>
54
  </div>
55
 
56
  ### πŸš€ Key Features
57
+ * πŸ“š **Unprecedented Scale:** Grounded in **22,413** professionally authored, human-written protocols.
58
+ * 🧩 **Rich Task Diversity:** Systematically constructed into a dataset of **523,784** structured task instances.
59
+ * 🎯 **5 Core Procedural Tasks:** Covers Protocol Question Answering (**PQA**), Step Ordering (**ORD**), Error Correction (**ERR**), Protocol Generation (**GEN**), and Protocol Reasoning (**REA**).
60
+ * 🧬 **Broad Domain Coverage:** Spans **16 biological subfields** sourced from **5 authoritative repositories**.
61
+ * πŸ”¬ **Standardized Evaluation:** Features a rigorous benchmark utilizing novel, domain-specific metrics (e.g., Step Recall, Step Precision) to accurately quantify structural fidelity.
62
 
63
  ---
64
 
 
68
  <img src="https://github.com/YuyangSunshine/bioprotocolbench/blob/main/figures/samples.jpg?raw=true" alt="BioProBench Samples" width="1000"/>
69
  </div>
70
 
71
+ We provide standardized JSON files for each task, meticulously partitioned into **Train** and **Test** splits to facilitate model fine-tuning and rigorous evaluation:
72
 
73
  | Task Identifier | Core Objective | Available Files |
74
  | :--- | :--- | :--- |
 
85
 
86
  ## πŸ”¬ Key Findings from Frontier LLMs
87
 
88
+ We evaluated 10 mainstream state-of-the-art LLMs, alongside a suite of the latest 2026 frontier models. Our rigorous evaluation reveals:
89
+ * **The Comprehension Plateau:** While top-tier models excel at basic declarative comprehension, performance degrades significantly on tasks demanding deep reasoning, quantitative precision, and safety awareness.
90
+ * **The Generation Bottleneck:** Models struggle profoundly with long-form procedural generation (GEN task). Even the most advanced models omit necessary experimental steps, underscoring that open-ended procedural planning remains a formidable challenge.
91
+ * **The Power of Grounding:** To mitigate these issues, we introduce **ProAgent**. Grounded in our BioProCorpus via a Retrieval-Augmented Generation (RAG) framework, ProAgent substantially advances the state-of-the-art, significantly improving procedural step recall and reasoning accuracy.
92
 
93
  ---
94