shank commited on
Commit Β·
22cb7e7
1
Parent(s): b658e10
docs: updated the readme and added a license
Browse files
LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 shashaank
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
README.md
CHANGED
|
@@ -1,102 +1,106 @@
|
|
| 1 |
---
|
| 2 |
-
title: AgentDebugger
|
| 3 |
emoji: π
|
| 4 |
colorFrom: yellow
|
| 5 |
colorTo: green
|
| 6 |
sdk: docker
|
| 7 |
app_port: 8000
|
| 8 |
-
pinned:
|
|
|
|
| 9 |
---
|
| 10 |
|
| 11 |
# AgentDebuggerEnv π
|
| 12 |
|
| 13 |
> **Benchmarking Agentic Reasoning through the Iterative Hypothesis-Test-Fix Loop.**
|
| 14 |
|
| 15 |
-
|
| 16 |
|
| 17 |
---
|
| 18 |
|
| 19 |
-
## π
|
| 20 |
|
| 21 |
-
|
| 22 |
-
- **Red Herrings**: Following misleading error messages to the wrong function.
|
| 23 |
-
- **Stagnant Iteration**: Repeating the same failed fix attempt instead of updating their hypothesis based on new output.
|
| 24 |
-
- **Concurrency Failures**: Failing to detect or fix non-deterministic bugs (race conditions).
|
| 25 |
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
---
|
| 29 |
|
| 30 |
-
## π οΈ
|
| 31 |
|
| 32 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
|
|
|
| 38 |
|
| 39 |
---
|
| 40 |
|
| 41 |
-
## π
|
| 42 |
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
|
| 46 |
-
|
|
| 47 |
-
| **
|
| 48 |
-
| **Medium** | Medium | **Red Herring**: Interdependent functions where the error manifests far from the root cause. |
|
| 49 |
-
| **Hard** | Hard | **Race Condition**: A concurrency bug that is invisible to sequential tests. Agent must design a concurrent test to surface it. |
|
| 50 |
|
| 51 |
---
|
| 52 |
|
| 53 |
-
##
|
| 54 |
|
| 55 |
-
|
| 56 |
-
- **Observation**: Includes `buggy_code`, `test_suite`, `previous_attempts` (full history), and `current_error_output`.
|
| 57 |
-
- **Action**: Supports `submit_fix` (requires `hypothesis`), `query_context` (for deeper code analysis), and `give_up`.
|
| 58 |
-
- **Reward**: A multi-component reward including `test_progress`, `hypothesis_match`, and `efficiency_bonus`.
|
| 59 |
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
- **Baseline Script**: Includes a reference `inference.py` script that uses the OpenAI client for benchmark evaluation.
|
| 65 |
|
| 66 |
---
|
| 67 |
|
| 68 |
-
##
|
| 69 |
|
| 70 |
-
###
|
| 71 |
```bash
|
| 72 |
-
git clone https://huggingface.co/spaces/
|
| 73 |
-
cd
|
| 74 |
-
pip install -
|
| 75 |
```
|
| 76 |
|
| 77 |
-
### Running
|
| 78 |
```bash
|
| 79 |
-
# Start the
|
| 80 |
uvicorn env.server:app --host 0.0.0.0 --port 8000
|
|
|
|
| 81 |
|
| 82 |
-
#
|
| 83 |
-
|
| 84 |
-
export
|
| 85 |
-
|
| 86 |
-
python inference.py
|
| 87 |
```
|
| 88 |
|
| 89 |
---
|
| 90 |
|
| 91 |
-
##
|
|
|
|
|
|
|
| 92 |
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
| Hard | 0.18 | No |
|
| 98 |
|
| 99 |
---
|
| 100 |
|
| 101 |
-
## π License
|
| 102 |
-
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: AgentDebugger-Env π
|
| 3 |
emoji: π
|
| 4 |
colorFrom: yellow
|
| 5 |
colorTo: green
|
| 6 |
sdk: docker
|
| 7 |
app_port: 8000
|
| 8 |
+
pinned: true
|
| 9 |
+
license: mit
|
| 10 |
---
|
| 11 |
|
| 12 |
# AgentDebuggerEnv π
|
| 13 |
|
| 14 |
> **Benchmarking Agentic Reasoning through the Iterative Hypothesis-Test-Fix Loop.**
|
| 15 |
|
| 16 |
+
**AgentDebuggerEnv** is an OpenEnv-compliant benchmarking environment designed for the **Meta + PyTorch + HuggingFace OpenEnv Hackathon**. Unlike static code-repair benchmarks that only measure the final output, AgentDebuggerEnv evaluates the *cognitive trajectory* of an agent: how it forms hypotheses, interprets execution failures, and iterates toward a solution in a secure, live sandbox.
|
| 17 |
|
| 18 |
---
|
| 19 |
|
| 20 |
+
## π The Core Philosophy
|
| 21 |
|
| 22 |
+
Traditional benchmarks (like HumanEval or MBPP) are "one-shot": the model sees a prompt and writes code. Real-world engineering is **iterative**.
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
+
AgentDebuggerEnv forces agents to operate in a **live feedback loop**:
|
| 25 |
+
1. **Observe**: Analyze existing buggy code and initial test failures.
|
| 26 |
+
2. **Hypothesize**: Explicitly state a theory about the root cause (scored for accuracy).
|
| 27 |
+
3. **Act**: Submit a surgical fix or query the environment for more context.
|
| 28 |
+
4. **Verify**: Observe real-time `stdout/stderr` from a sandboxed test suite execution.
|
| 29 |
|
| 30 |
---
|
| 31 |
|
| 32 |
+
## π οΈ Technical Architecture
|
| 33 |
|
| 34 |
+
### 1. Robust Security Sandbox
|
| 35 |
+
Every submission is executed in a multi-layered isolated environment:
|
| 36 |
+
* **AST Filtering**: An Abstract Syntax Tree (AST) pass blocks dangerous imports (`os`, `sys`, `subprocess`, etc.) and builtins before execution.
|
| 37 |
+
* **Process Isolation**: Executes in a separate subprocess with hard memory (256MB) and time (10s) limits.
|
| 38 |
+
* **Thread Safety**: A specialized "Concurrency Sandbox" allows multi-threaded tests for identifying race conditions while maintaining host security.
|
| 39 |
|
| 40 |
+
### 2. High-Fidelity Feedback
|
| 41 |
+
Instead of binary `Pass/Fail` bits, the environment returns the **raw execution stream**. This allows agents to:
|
| 42 |
+
* Read stack traces.
|
| 43 |
+
* See partial progress (e.g., "6 passed, 2 failed").
|
| 44 |
+
* Detect timeouts and resource exhaustion.
|
| 45 |
|
| 46 |
---
|
| 47 |
|
| 48 |
+
## π Task Suite & Reasoning Challenges
|
| 49 |
|
| 50 |
+
| Task | Difficulty | Reasoning Challenge | Why it's hard |
|
| 51 |
+
| :--- | :--- | :--- | :--- |
|
| 52 |
+
| **Easy** | π’ Easy | **Off-by-One** | Requires basic logic verification. The error message is high-signal. |
|
| 53 |
+
| **Medium** | π‘ Medium | **Red Herring** | The symptom (MD5 hashing error) manifests far from the root cause. Agent must trace data flow backward. |
|
| 54 |
+
| **Hard** | π΄ Hard | **Race Condition** | **Invisible to sequential tests.** The agent must reason that passing tests do *not* mean the code is correct, and design a concurrent stress test. |
|
|
|
|
|
|
|
| 55 |
|
| 56 |
---
|
| 57 |
|
| 58 |
+
## π Professional Grading Methodology
|
| 59 |
|
| 60 |
+
Our graders don't just check if the code works at the end. They score the **process**:
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
+
* **Sequential Correctness (40%)**: Does the fix pass the original unit tests?
|
| 63 |
+
* **Hidden Strength (30%)**: Does the fix survive a high-concurrency (1000-thread) stress test? (Hard task only).
|
| 64 |
+
* **Hypothesis Accuracy (20%)**: Did the agent correctly identify the bug? (NLP-based keyword matching against ground truth).
|
| 65 |
+
* **Efficiency Bonus (10%)**: Did the agent solve it within 5 attempts?
|
|
|
|
| 66 |
|
| 67 |
---
|
| 68 |
|
| 69 |
+
## βοΈ Installation & Usage
|
| 70 |
|
| 71 |
+
### π¦ Local Setup
|
| 72 |
```bash
|
| 73 |
+
git clone https://huggingface.co/spaces/shashaank0707/AgentDebugger-env
|
| 74 |
+
cd AgentDebugger-env
|
| 75 |
+
pip install -e .
|
| 76 |
```
|
| 77 |
|
| 78 |
+
### π’ Running the Environment
|
| 79 |
```bash
|
| 80 |
+
# Start the FastAPI server
|
| 81 |
uvicorn env.server:app --host 0.0.0.0 --port 8000
|
| 82 |
+
```
|
| 83 |
|
| 84 |
+
### π€ Running an Agent (OpenEnv Baseline)
|
| 85 |
+
```bash
|
| 86 |
+
export OPENAI_API_KEY="your_key"
|
| 87 |
+
python inference.py --task easy --model gpt-4o
|
|
|
|
| 88 |
```
|
| 89 |
|
| 90 |
---
|
| 91 |
|
| 92 |
+
## π OpenEnv API Compliance
|
| 93 |
+
|
| 94 |
+
AgentDebuggerEnv implements the full OpenEnv specification:
|
| 95 |
|
| 96 |
+
* `POST /reset`: Initialize a task (`{"task_id": "medium"}`).
|
| 97 |
+
* `POST /step`: Submit an `Action` (supports `submit_fix`, `query_context`, `give_up`).
|
| 98 |
+
* `GET /state`: Retrieve full episode history and current environment state.
|
| 99 |
+
* `GET /health`: Standard health check for automated uptime monitoring.
|
|
|
|
| 100 |
|
| 101 |
---
|
| 102 |
|
| 103 |
+
## π Metadata & License
|
| 104 |
+
* **License**: [MIT](LICENSE)
|
| 105 |
+
* **Author**: shashaank
|
| 106 |
+
* **Hackathon**: Meta + PyTorch + HuggingFace OpenEnv 2024
|