shank commited on
Commit
22cb7e7
Β·
1 Parent(s): b658e10

docs: updated the readme and added a license

Browse files
Files changed (2) hide show
  1. LICENSE +21 -0
  2. README.md +57 -53
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 shashaank
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md CHANGED
@@ -1,102 +1,106 @@
1
  ---
2
- title: AgentDebugger Env πŸ›
3
  emoji: πŸ“ˆ
4
  colorFrom: yellow
5
  colorTo: green
6
  sdk: docker
7
  app_port: 8000
8
- pinned: false
 
9
  ---
10
 
11
  # AgentDebuggerEnv πŸ›
12
 
13
  > **Benchmarking Agentic Reasoning through the Iterative Hypothesis-Test-Fix Loop.**
14
 
15
- An OpenEnv-compliant environment designed for the **Meta + PyTorch + HuggingFace OpenEnv Hackathon**. Unlike static code-repair benchmarks, **AgentDebuggerEnv** focuses on the *trajectory* of an agent's reasoningβ€”measuring how effectively an agent forms hypotheses, observes failures, and iterates toward a solution in a live execution sandbox.
16
 
17
  ---
18
 
19
- ## πŸš€ Overview
20
 
21
- Debugging is one of the highest-leverage cognitive tasks in software engineering. Modern LLM agents often struggle with:
22
- - **Red Herrings**: Following misleading error messages to the wrong function.
23
- - **Stagnant Iteration**: Repeating the same failed fix attempt instead of updating their hypothesis based on new output.
24
- - **Concurrency Failures**: Failing to detect or fix non-deterministic bugs (race conditions).
25
 
26
- **AgentDebuggerEnv** makes these failures measurable and scorable. The environment provides a live, sandboxed feedback loop where agents submit complete code fixes and receive real-time execution results.
 
 
 
 
27
 
28
  ---
29
 
30
- ## πŸ› οΈ Core Mechanics: The Feedback Loop
31
 
32
- The environment follows the standard OpenEnv interface (`reset`, `step`, `state`) but enforces a strict **Hypothesis-Test-Fix** cycle:
 
 
 
 
33
 
34
- 1. **Hypothesis**: The agent must state its theory about the bug before every fix attempt.
35
- 2. **Execution**: The submitted code is executed in a secure sandbox with hard timeouts.
36
- 3. **Observation**: The agent receives the actual `stdout` + `stderr` from the test suite, not just a binary pass/fail.
37
- 4. **Reward**: Dense reward signal is provided at every step, scaling with test progress and hypothesis accuracy.
 
38
 
39
  ---
40
 
41
- ## πŸ“ Tasks & Difficulty
42
 
43
- The environment includes three standardized tasks designed to test different facets of agentic reasoning:
44
-
45
- | Task | Difficulty | Core Challenge |
46
- | :--- | :--- | :--- |
47
- | **Easy** | Easy | **Off-by-One**: Simple logic bug with an explicit, high-signal error message. |
48
- | **Medium** | Medium | **Red Herring**: Interdependent functions where the error manifests far from the root cause. |
49
- | **Hard** | Hard | **Race Condition**: A concurrency bug that is invisible to sequential tests. Agent must design a concurrent test to surface it. |
50
 
51
  ---
52
 
53
- ## βš™οΈ How It Works (Spec Compliance)
54
 
55
- ### Data Models
56
- - **Observation**: Includes `buggy_code`, `test_suite`, `previous_attempts` (full history), and `current_error_output`.
57
- - **Action**: Supports `submit_fix` (requires `hypothesis`), `query_context` (for deeper code analysis), and `give_up`.
58
- - **Reward**: A multi-component reward including `test_progress`, `hypothesis_match`, and `efficiency_bonus`.
59
 
60
- ### Infrastructure
61
- - **FastAPI**: Exposes standard endpoints on port 8000.
62
- - **Docker**: Fully containerized and ready for HuggingFace Spaces.
63
- - **Security**: Robust AST-based filtering to prevent malicious code escape.
64
- - **Baseline Script**: Includes a reference `inference.py` script that uses the OpenAI client for benchmark evaluation.
65
 
66
  ---
67
 
68
- ## πŸ“¦ Quick Start
69
 
70
- ### Installation
71
  ```bash
72
- git clone https://huggingface.co/spaces/shashaank/agentdebugger-env
73
- cd agentdebugger-env
74
- pip install -r requirements.txt
75
  ```
76
 
77
- ### Running Locally
78
  ```bash
79
- # Start the environment server
80
  uvicorn env.server:app --host 0.0.0.0 --port 8000
 
81
 
82
- # Run the baseline inference (requires API key)
83
- export API_BASE_URL="https://api.openai.com/v1"
84
- export MODEL_NAME="gpt-4o"
85
- export HF_TOKEN="your_key_here"
86
- python inference.py
87
  ```
88
 
89
  ---
90
 
91
- ## πŸ“Š Benchmarking Results (GPT-4o Baseline)
 
 
92
 
93
- | Task | Grader Score | Solved |
94
- | :--- | :--- | :--- |
95
- | Easy | 0.85 | Yes |
96
- | Medium | 0.50 | Mixed |
97
- | Hard | 0.18 | No |
98
 
99
  ---
100
 
101
- ## πŸ“œ License
102
- MIT License. Created by **shashaank** for the Meta / PyTorch / HuggingFace OpenEnv Hackathon.
 
 
 
1
  ---
2
+ title: AgentDebugger-Env πŸ›
3
  emoji: πŸ“ˆ
4
  colorFrom: yellow
5
  colorTo: green
6
  sdk: docker
7
  app_port: 8000
8
+ pinned: true
9
+ license: mit
10
  ---
11
 
12
  # AgentDebuggerEnv πŸ›
13
 
14
  > **Benchmarking Agentic Reasoning through the Iterative Hypothesis-Test-Fix Loop.**
15
 
16
+ **AgentDebuggerEnv** is an OpenEnv-compliant benchmarking environment designed for the **Meta + PyTorch + HuggingFace OpenEnv Hackathon**. Unlike static code-repair benchmarks that only measure the final output, AgentDebuggerEnv evaluates the *cognitive trajectory* of an agent: how it forms hypotheses, interprets execution failures, and iterates toward a solution in a secure, live sandbox.
17
 
18
  ---
19
 
20
+ ## πŸš€ The Core Philosophy
21
 
22
+ Traditional benchmarks (like HumanEval or MBPP) are "one-shot": the model sees a prompt and writes code. Real-world engineering is **iterative**.
 
 
 
23
 
24
+ AgentDebuggerEnv forces agents to operate in a **live feedback loop**:
25
+ 1. **Observe**: Analyze existing buggy code and initial test failures.
26
+ 2. **Hypothesize**: Explicitly state a theory about the root cause (scored for accuracy).
27
+ 3. **Act**: Submit a surgical fix or query the environment for more context.
28
+ 4. **Verify**: Observe real-time `stdout/stderr` from a sandboxed test suite execution.
29
 
30
  ---
31
 
32
+ ## πŸ› οΈ Technical Architecture
33
 
34
+ ### 1. Robust Security Sandbox
35
+ Every submission is executed in a multi-layered isolated environment:
36
+ * **AST Filtering**: An Abstract Syntax Tree (AST) pass blocks dangerous imports (`os`, `sys`, `subprocess`, etc.) and builtins before execution.
37
+ * **Process Isolation**: Executes in a separate subprocess with hard memory (256MB) and time (10s) limits.
38
+ * **Thread Safety**: A specialized "Concurrency Sandbox" allows multi-threaded tests for identifying race conditions while maintaining host security.
39
 
40
+ ### 2. High-Fidelity Feedback
41
+ Instead of binary `Pass/Fail` bits, the environment returns the **raw execution stream**. This allows agents to:
42
+ * Read stack traces.
43
+ * See partial progress (e.g., "6 passed, 2 failed").
44
+ * Detect timeouts and resource exhaustion.
45
 
46
  ---
47
 
48
+ ## πŸ“ Task Suite & Reasoning Challenges
49
 
50
+ | Task | Difficulty | Reasoning Challenge | Why it's hard |
51
+ | :--- | :--- | :--- | :--- |
52
+ | **Easy** | 🟒 Easy | **Off-by-One** | Requires basic logic verification. The error message is high-signal. |
53
+ | **Medium** | 🟑 Medium | **Red Herring** | The symptom (MD5 hashing error) manifests far from the root cause. Agent must trace data flow backward. |
54
+ | **Hard** | πŸ”΄ Hard | **Race Condition** | **Invisible to sequential tests.** The agent must reason that passing tests do *not* mean the code is correct, and design a concurrent stress test. |
 
 
55
 
56
  ---
57
 
58
+ ## πŸ“Š Professional Grading Methodology
59
 
60
+ Our graders don't just check if the code works at the end. They score the **process**:
 
 
 
61
 
62
+ * **Sequential Correctness (40%)**: Does the fix pass the original unit tests?
63
+ * **Hidden Strength (30%)**: Does the fix survive a high-concurrency (1000-thread) stress test? (Hard task only).
64
+ * **Hypothesis Accuracy (20%)**: Did the agent correctly identify the bug? (NLP-based keyword matching against ground truth).
65
+ * **Efficiency Bonus (10%)**: Did the agent solve it within 5 attempts?
 
66
 
67
  ---
68
 
69
+ ## βš™οΈ Installation & Usage
70
 
71
+ ### πŸ“¦ Local Setup
72
  ```bash
73
+ git clone https://huggingface.co/spaces/shashaank0707/AgentDebugger-env
74
+ cd AgentDebugger-env
75
+ pip install -e .
76
  ```
77
 
78
+ ### 🚒 Running the Environment
79
  ```bash
80
+ # Start the FastAPI server
81
  uvicorn env.server:app --host 0.0.0.0 --port 8000
82
+ ```
83
 
84
+ ### πŸ€– Running an Agent (OpenEnv Baseline)
85
+ ```bash
86
+ export OPENAI_API_KEY="your_key"
87
+ python inference.py --task easy --model gpt-4o
 
88
  ```
89
 
90
  ---
91
 
92
+ ## πŸ”— OpenEnv API Compliance
93
+
94
+ AgentDebuggerEnv implements the full OpenEnv specification:
95
 
96
+ * `POST /reset`: Initialize a task (`{"task_id": "medium"}`).
97
+ * `POST /step`: Submit an `Action` (supports `submit_fix`, `query_context`, `give_up`).
98
+ * `GET /state`: Retrieve full episode history and current environment state.
99
+ * `GET /health`: Standard health check for automated uptime monitoring.
 
100
 
101
  ---
102
 
103
+ ## πŸ“œ Metadata & License
104
+ * **License**: [MIT](LICENSE)
105
+ * **Author**: shashaank
106
+ * **Hackathon**: Meta + PyTorch + HuggingFace OpenEnv 2024