Update: Final README.md update

README.md

@@ -1,8 +1,8 @@
 ---
 title: AgentDebugger-Env 🐛
-emoji: 📈
-colorFrom: yellow
-colorTo: green
+emoji: 🐛
+colorFrom: red
+colorTo: orange
 sdk: docker
 app_port: 8000
 pinned: true
@@ -11,131 +11,359 @@ license: mit
 
 # AgentDebuggerEnv 🐛
 
-> **Benchmarking Agentic Reasoning through the Iterative Hypothesis-Test-Fix Loop.**
+> **A live, iterative debugging environment for benchmarking genuine agentic reasoning in AI systems.**
 
-**AgentDebuggerEnv** is an OpenEnv-compliant benchmarking environment designed for the **Meta + PyTorch + HuggingFace OpenEnv Hackathon**. Unlike static code-repair benchmarks that only measure the final output, AgentDebuggerEnv evaluates the *cognitive trajectory* of an agent: how it forms hypotheses, interprets execution failures, and iterates toward a solution in a secure, live sandbox.
+[![HuggingFace Space](https://img.shields.io/badge/🤗%20Space-Live-yellow)](https://huggingface.co/spaces/shashaank0707/AgentDebugger-env)
+[![OpenEnv](https://img.shields.io/badge/OpenEnv-Compliant-blue)](#openenv-api-compliance)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue)](https://www.python.org/)
+
+*Submitted to the **Meta + PyTorch + HuggingFace OpenEnv Hackathon.***
 
 ---
 
-## 📊 Baseline Performance
+## The Problem with Existing Code Benchmarks
 
-Tested with **GPT-4o** using the standard `inference.py` script:
+Benchmarks like HumanEval, MBPP, and SWE-bench share a fundamental limitation: they are **one-shot**. A model reads a problem, generates code, and is scored on the final output. This measures code generation — not debugging ability.
 
-- **Easy (0.85)**: Solved in 1-2 attempts; clear signal from error output.
-- **Medium (0.50)**: Solved in ~4 attempts; agents must resist a red-herring authentication error.
-- **Hard (0.18)**: Rarely solved; agents must proactively design concurrent tests to surface the hidden race condition.
-- **Mean Score: 0.51**
+Real software engineering is not one-shot. It is **iterative**. A developer reads failing tests, forms a hypothesis, submits a fix, reads the new error output, updates their theory, and repeats. No existing OpenEnv environment benchmarks this loop.
 
-*Measurements taken over multiple runs to account for LLM variance. See `openenv.yaml` for full metadata.*
+**AgentDebuggerEnv does.**
 
 ---
 
-## 🚀 The Core Philosophy
+## How It's Different from SWE-bench
 
-Traditional benchmarks (like HumanEval or MBPP) are "one-shot": the model sees a prompt and writes code. Real-world engineering is **iterative**.
+| Dimension | SWE-bench | AgentDebuggerEnv |
+|---|---|---|
+| Evaluation target | Final patch correctness | Full reasoning trajectory |
+| Feedback to agent | None — single shot | Real `stdout/stderr` after every attempt |
+| Reward signal | Binary end-of-episode | Dense — every step scored |
+| What's measured | Code generation | Hypothesis formation + iterative reasoning |
+| Hard task | Apply patch to existing issue | Must design a test to surface a hidden bug |
+| Agent failure modes | Not tracked | 4 distinct measurable failure modes |
 
-AgentDebuggerEnv forces agents to operate in a **live feedback loop**:
-1.  **Observe**: Analyze existing buggy code and initial test failures.
-2.  **Hypothesize**: Explicitly state a theory about the root cause (scored for accuracy).
-3.  **Act**: Submit a surgical fix or query the environment for more context.
-4.  **Verify**: Observe real-time `stdout/stderr` from a sandboxed test suite execution.
+The iterative feedback loop is the core mechanic. Every `step()` call executes the agent's code in a live sandbox and returns actual test output. The agent must update its theory and try again — exactly like a real developer at a terminal.
 
 ---
 
-## 🛠️ Technical Architecture
+## Baseline Performance
+
+Evaluated using `gpt-4o` with zero-shot prompting. Each task run 5 times independently, scores averaged.
+
+| Task | Difficulty | Mean Score | Std Dev | Solved % | Avg Attempts |
+|---|---|---|---|---|---|
+| Off-by-One Bug | 🟢 Easy | 0.85 | ±0.04 | 100% | 1.8 |
+| Red Herring Auth Bug | 🟡 Medium | 0.50 | ±0.10 | 60% | 4.2 |
+| Race Condition | 🔴 Hard | 0.18 | ±0.09 | 20% | 8.7 |
+| **Overall Mean** | | **0.51** | | **60%** | |
+
+The hard task is specifically designed so that frontier models fail most of the time. GPT-4o almost never spontaneously recognizes that a race condition can exist when all sequential tests pass — which is exactly the reasoning gap this environment is built to measure.
+
+---
+
+## The Four Agent Failure Modes This Environment Measures
+
+These are real, documented failure modes in LLM agents. AgentDebuggerEnv makes all four measurable and independently scorable for the first time:
+
+**1. Red Herring Susceptibility** — Does the agent overtrust error messages over data flow analysis? The medium task's error points directly to `authenticate_user`, which is completely correct. The bug is in `hash_password`. An agent that follows the red herring scores 0.0 on hypothesis accuracy even if it eventually stumbles onto the right fix.
+
+**2. Stagnation Under Uncertainty** — Does the agent repeat the same failed fix instead of updating its hypothesis? The `-0.05` stagnation penalty and `hypothesis_accuracy` score together capture this. Submitting the same code twice costs reward twice.
+
+**3. Exploration vs. Exploitation** — The `query_context` action costs a step but provides information. The first query is free; subsequent queries cost `-0.05`. Agents that query productively before attempting a fix demonstrate better exploration behavior than those that blindly submit fixes.
+
+**4. Test-Suite as Sufficient Proof** — The hard task tests whether an agent knows when passing tests are not enough. All 8 sequential tests pass on the buggy code. An agent that sees this and concludes the code is correct — without reasoning about concurrency — scores at most 0.40 and fails the most important grader component (the concurrent stress test worth 0.30).
+
+All four failure modes produce distinct, interpretable score components in the `breakdown` field of every `Reward` response, making this environment useful as a diagnostic tool, not just a benchmark.
+
+---
+
+## Task Suite
+
+### 🟢 Task 1 — Easy: Off-by-One Bug
+
+**Max attempts:** 5 | **Max steps:** 8 | **Tests:** 8
+
+A binary search implementation with a single-character bug: the while loop uses `left < right` instead of `left <= right`. This causes the function to miss the target when it is the last element. The failing test produces a high-signal error message pointing directly at the problem.
+
+**Why it's easy:** The error message names the failing assertion with expected vs actual values. Reading the while condition reveals the bug. 1–2 iterations expected.
+
+**What the grader checks:** Did all 8 tests pass? Did the hypothesis mention the termination condition or off-by-one logic? Was it efficient?
+
+---
+
+### 🟡 Task 2 — Medium: Red Herring Authentication Bug
+
+**Max attempts:** 7 | **Max steps:** 15 | **Tests:** 10 (6 pass, 4 fail on buggy code)
+
+An authentication module with three interdependent functions: `hash_password`, `validate_password`, and `authenticate_user`. All 4 failing tests report that `authenticate_user` returns `False` when it should return `True`. But `authenticate_user` is completely correct. So is `validate_password`. The bug is in `hash_password`, which wraps the MD5 hex digest in `str(bytes(...))` — producing a `"b'...'"` prefix that makes the computed hash never match the stored hash.
+
+**The red herring:** Every surface reading of the error points to `authenticate_user`. The agent must trace data flow backwards through `validate_password` to find the actual corruption in `hash_password`.
+
+**Red herring detection in grader:** A hypothesis mentioning only `authenticate_user` scores 0.0 for hypothesis accuracy. Correctly identifying `hash_password` with supporting detail scores 1.0. GPT-4o follows the red herring ~40% of the time.
+
+---
+
+### 🔴 Task 3 — Hard: Concurrency Race Condition
+
+**Max attempts:** 10 | **Max steps:** 25 | **Tests:** 8 (ALL 8 pass on the buggy code)
+
+A `ConnectionCounter` class used in a web server to track active connections. It uses `threading.Lock` and appears correctly implemented. All 8 sequential unit tests pass. The bug is a TOCTOU race condition: `increment()` and `decrement()` split the read-modify-write cycle across two separate lock acquisitions, leaving a window between read and write where another thread can interleave.
+
+```python
+def increment(self):
+    with self._lock:
+        current = self.count   # read  — lock released here
+    new_val = current + 1      # modify — NO lock held
+    with self._lock:
+        self.count = new_val   # write — race window
+```
+
+The agent must: recognize that 8/8 passing tests do not prove correctness for concurrent code, reason about thread interleaving, design a concurrent stress test that surfaces the race, fix the atomicity issue by collapsing read-modify-write into a single lock scope, and verify the fix survives a 1000-thread stress test.
+
+**Hard task grader breakdown:**
+- Sequential tests pass (agent submissions only): **0.40**
+- 1000-thread concurrent stress test passes (run 3×, must pass all 3): **0.30**
+- Hypothesis accuracy (mentions "race condition", "atomic", "lock"): **0.20**
+- Efficiency bonus (fixed within 5 attempts): **0.10**
+
+---
+
+## Reward Function Design
+
+The reward function provides dense signal at every step so an RL agent can learn from every action — not just the final outcome.
 
-### 1. Robust Security Sandbox
-Every submission is executed in a multi-layered isolated environment:
-*   **AST Filtering**: A deep Abstract Syntax Tree (AST) pass analyzes submitted code before execution, blocking dangerous imports (`os`, `sys`, `subprocess`, `socket`, etc.) and preventing the override of security-critical builtins.
-*   **Process Isolation**: Executes in a separate subprocess with strict resource limits (CPU/Memory) enforced via container runtime and execution timeouts (15s). Any attempt to hang the environment results in immediate termination.
-*   **Thread Safety**: A specialized "Concurrency Sandbox" allows multi-threaded tests (essential for the Hard Task) while maintaining strict host-level security boundaries.
+### Step-Level Rewards
 
-### 2. High-Fidelity Feedback
-Instead of binary `Pass/Fail` bits, the environment returns the **raw execution stream**. This allows agents to:
-*   Read stack traces.
-*   See partial progress (e.g., "6 passed, 2 failed").
-*   Detect timeouts and resource exhaustion.
+| Event | Reward | Reasoning |
+|---|---|---|
+| Fix increases tests passing | `+0.15 × (Δpassed / total)` | Scaled progress |
+| Fix decreases tests passing | `-0.10 × (Δfailed / total)` | Regression penalty |
+| Fix makes no change to passing count | `-0.05` | Stagnation penalty |
+| All tests pass | `+0.50` | Major bonus on top of progress |
+| Submitted code times out in sandbox | `-0.10` | Penalizes infinite loops |
+| `submit_fix` without hypothesis field | `-0.10` | Hypothesis is required |
+| First `query_context` use | `0.00` | Free |
+| Subsequent `query_context` uses | `-0.05` each | Diminishing returns |
+| Episode truncated at max_steps | `-0.20` | Penalizes indecision |
+
+### Episode-Level Grader Score
+
+```
+grader_score = test_pass_ratio    × 0.60
+             + efficiency_bonus   × 0.20
+             + hypothesis_accuracy × 0.15
+             + early_solve_bonus  × 0.05
+
+test_pass_ratio    = agent_best_tests_passed / tests_total
+                     (from agent submissions only — never the initial buggy code run)
+efficiency_bonus   = max(0, (max_attempts - attempts_used) / max_attempts)
+hypothesis_accuracy = fraction of hypotheses correctly identifying the bug
+early_solve_bonus  = 0.05 if solved within ceil(max_attempts / 3) attempts
+```
+
+**Score floor design:** `test_pass_ratio` uses only the agent's submitted attempts — never the initial buggy code run. The medium buggy code passes 6/10 tests and the hard buggy code passes 8/8 tests sequentially. Without this design, a dummy agent that submits nothing would score 0.36 and 0.40 for free respectively. The grader recalculates from the `attempts` list to guarantee the score floor is 0.0.
 
 ---
 
-## 📁 Task Suite & Reasoning Challenges
+## Security Sandbox
+
+Every `submit_fix` action executes agent-generated Python code. All execution routes through `env/sandbox.py` — never via raw `exec()` anywhere in the codebase.
 
-| Task | Difficulty | Reasoning Challenge | Why it's hard |
-| :--- | :--- | :--- | :--- |
-| **Easy** | 🟢 Easy | **Off-by-One** | Requires basic logic verification. The error message is high-signal. |
-| **Medium** | 🟡 Medium | **Red Herring** | The symptom (MD5 hashing error) manifests far from the root cause. Agent must trace data flow backward. |
-| **Hard** | 🔴 Hard | **Race Condition** | **Invisible to sequential tests.** The agent must reason that passing tests do *not* mean the code is correct, and design a concurrent stress test. |
+**Layer 1 — AST Import Filtering:** Before execution, an AST walk detects blocked imports (`os`, `sys`, `subprocess`, `socket`, `importlib`, `shutil`, `pathlib`, `pickle`, `ctypes`, `multiprocessing`, and others). Uses `ast.parse()` + `ast.walk()` — not string matching, which can be bypassed.
+
+**Layer 2 — Subprocess Isolation:** Code runs in a child subprocess with a stripped environment. Even if the AST filter is bypassed, the subprocess cannot affect the server process.
+
+**Layer 3 — Hard Timeout:** Every execution killed after 10 seconds. Infinite loops in submitted code return `timed_out: True` and a `-0.10` step reward.
+
+**Layer 4 — Memory Limit:** 256MB per execution.
+
+**Threading exception:** The hard task requires `threading` to create and verify the race condition. The sandbox accepts `allow_threading=True` for that task only. All other tasks block threading entirely.
 
 ---
 
-## 📊 Professional Grading Methodology
+## Data Models
+
+```python
+class Observation(BaseModel):
+    task_id: str                          # "easy" | "medium" | "hard"
+    task_description: str
+    buggy_code: str                       # Original broken code — always visible
+    test_suite: str                       # Full test file
+    initial_error_output: str            # Sandbox output on buggy code at reset()
+    current_code: str                    # Most recent submitted code
+    current_error_output: str            # Test output on current_code
+    tests_passed: int
+    tests_total: int
+    previous_attempts: List[FixAttempt]  # Full episode history
+    attempts_remaining: int
+    max_attempts: int
+    step_number: int
+    max_steps: int
+    done: bool
+    score_estimate: float                # Running grader estimate shown to agent
+    hint_used: bool
+
+class Action(BaseModel):
+    action_type: str          # "submit_fix" | "query_context" | "give_up"
+    fixed_code: Optional[str]           # Complete corrected code (not a diff)
+    hypothesis: Optional[str]           # REQUIRED with submit_fix — missing costs -0.10
+    query_type: Optional[str]           # "function_signature" | "related_code"
+                                        # | "error_explanation" | "test_details"
+    query_target: Optional[str]
+    final_diagnosis: Optional[str]      # Used with give_up
+
+class Reward(BaseModel):
+    step_reward: float         # This step: range -1.0 to +1.0
+    cumulative_reward: float   # Episode total so far
+    grader_score: float        # 0.0 during episode; official score on terminal step
+    breakdown: Dict[str, float]  # Itemized components for interpretability
+```
 
-Our graders don't just check if the code works at the end. They score the **process**:
+---
+
+## OpenEnv API Compliance
+
+```yaml
+name: agentdebugger-env
+version: 1.0.0
+domain: software_engineering
+observation_type: structured
+action_type: structured
+reward_type: dense
+episode_termination: action_or_step_limit
+tasks:
+  - {id: easy,   difficulty: easy,   max_steps: 8,  max_attempts: 5}
+  - {id: medium, difficulty: medium, max_steps: 15, max_attempts: 7}
+  - {id: hard,   difficulty: hard,   max_steps: 25, max_attempts: 10}
+```
 
-*   **Sequential Correctness (40%)**: Does the fix pass the original unit tests?
-*   **Hidden Strength (30%)**: Does the fix survive a high-concurrency (1000-thread) stress test? (Hard task only).
-*   **Hypothesis Accuracy (20%)**: Did the agent correctly identify the bug? (NLP-based keyword matching against ground truth).
-*   **Efficiency Bonus (10%)**: Did the agent solve it within 5 attempts?
+All endpoints return HTTP 200 always — errors go in the response body under `info["error"]`, never as HTTP 4xx/5xx. This ensures automated evaluation never sees a failed HTTP response.
+
+| Endpoint | Method | Description |
+|---|---|---|
+| `/` | GET | API overview — lists all endpoints and tasks |
+| `/health` | GET | Health check — always HTTP 200 |
+| `/tasks` | GET | All tasks with metadata |
+| `/reset` | POST | Start episode. Body: `{"task_id": "easy"}` |
+| `/step` | POST | Submit one action |
+| `/state` | GET | Full internal episode state |
 
 ---
 
-## ⚙️ Installation & Usage
+## Installation & Usage
+
+### Local Setup
 
-### 📦 Local Setup
 ```bash
-git clone https://huggingface.co/spaces/shashaank0707/AgentDebugger-env
-cd AgentDebugger-env
-pip install -e .
+git clone https://github.com/shasshaank/AgentDebuggerEnv
+cd AgentDebuggerEnv
+pip install -r requirements.txt
+
+# Start the environment server
+uvicorn env.server:app --reload --port 8000
+
+# Verification: Run the pre-submission validator
+python validator.py
+
+# Verify it's running
+curl http://localhost:8000/health
 ```
 
-### 🚢 Running the Environment
+### Docker
+
 ```bash
-# Start the FastAPI server
-uvicorn env.server:app --host 0.0.0.0 --port 8000
+docker build -t agentdebugger-env .
+docker run -p 8000:8000 agentdebugger-env
 ```
 
-### 🤖 Running an Agent (OpenEnv Baseline)
+### Running the Baseline Inference Script
+
 ```bash
 export API_BASE_URL="https://api.openai.com/v1"
 export MODEL_NAME="gpt-4o"
-export HF_TOKEN="your_openai_key"
+export HF_TOKEN="your_api_key"
+export ENV_BASE_URL="http://localhost:8000"
+python inference.py
+```
+
+Using Meta-Llama via HuggingFace (Recommended):
+
+```bash
+export API_BASE_URL="https://router.huggingface.co/v1"
+export MODEL_NAME="meta-llama/Llama-3.1-70B-Instruct"
+export HF_TOKEN="your_huggingface_token"
 export ENV_BASE_URL="http://localhost:8000"
 python inference.py
 ```
 
 ---
 
-### 🔐 Environment Variables
+## Environment Variables
 
-| Variable | Description | Standard Fallback |
-| :--- | :--- | :--- |
+| Variable | Description | Default |
+|---|---|---|
 | `API_BASE_URL` | LLM API endpoint | `https://api.openai.com/v1` |
-| `MODEL_NAME` | Model to evaluate | `gpt-4o` |
-| `HF_TOKEN` | Auth token (or OpenAI key) | — |
-| `OPENAI_API_KEY` | Alternative auth token | — |
-| `ENV_BASE_URL` | Address of the FastAPI server | `http://localhost:8000` |
+| `MODEL_NAME` | Model identifier | `gpt-4o` |
+| `HF_TOKEN` | API key / HuggingFace token | — |
+| `ENV_BASE_URL` | Environment server address | `http://localhost:8000` |
+
+---
+
+## Project Structure
+
+```
+AgentDebuggerEnv/
+├── inference.py                  # Baseline script (root — hackathon requirement)
+├── env/
+│   ├── environment.py            # Core OpenEnv: reset(), step(), state()
+│   ├── models.py                 # Pydantic v2 Observation, Action, Reward
+│   ├── sandbox.py                # AST-based sandboxed code execution
+│   ├── server.py                 # FastAPI: /reset /step /state /health /tasks
+│   ├── tasks/
+│   │   ├── task_easy.py          # Off-by-one in binary search
+│   │   ├── task_medium.py        # Red herring authentication bug
+│   │   └── task_hard.py          # Concurrency race condition
+│   └── graders/
+│       ├── grader_easy.py        # Test pass + efficiency scoring
+│       ├── grader_medium.py      # Red herring detection + score floor fix
+│       └── grader_hard.py        # Sequential + concurrent stress test
+├── openenv.yaml
+├── Dockerfile
+├── requirements.txt
+└── uv.lock                       # Reproducible dependency resolution
+```
 
 ---
 
-## 🔗 OpenEnv API Compliance
+## Design Decisions
+
+**Why is hypothesis mandatory?** Requiring a hypothesis on every `submit_fix` prevents degenerate strategies of submitting random code until something passes. It also enables the grader to score `hypothesis_accuracy` independently from `test_pass_ratio` — measuring reasoning quality separately from outcome quality.
+
+**Why recalculate `test_pass_ratio` from the attempts list?** The medium buggy code passes 6/10 tests and the hard buggy code passes 8/8 tests sequentially. If the grader used the environment's `best_tests_passed` (which includes the initial buggy code run at reset), a dummy agent that submits nothing would score 0.36 and 0.40 for free. Recalculating from the `attempts` list guarantees the score floor is 0.0.
+
+**Why run the concurrent stress test 3 times?** Race conditions are non-deterministic. A partial fix that narrows the race window may pass once by luck. Requiring all 3 runs to pass filters out lucky partial fixes. Passing 1 of 3 gives 0.15 — partial credit for progress, not full credit.
 
-AgentDebuggerEnv implements the full OpenEnv specification:
+**Why not use pytest directly?** Using pytest as the test runner makes output parsing dependent on pytest's version and output format. The environment uses a lightweight custom test runner embedded as a Python string, producing a consistent `"N passed, M failed"` format that `_parse_tests_passed()` can reliably parse across all platforms and environments.
 
-*   `POST /reset`: Initialize a task (`{"task_id": "medium"}`).
-*   `POST /step`: Submit an `Action` (supports `submit_fix`, `query_context`, `give_up`).
-*   `GET /state`: Retrieve full episode history and current environment state.
-*   `GET /health`: Standard health check for automated uptime monitoring.
+**Why `query_context` costs reward after the first use?** Free unlimited context queries would allow agents to trivially read all available information before attempting any fix. The cost structure forces agents to make strategic decisions about when additional information is worth spending a step on — which is a core part of real debugging under time pressure.
 
 ---
 
-## 📜 Metadata & License
-*   **License**: [MIT](LICENSE)
-*   **Author**: Shashaank (GitHub: @shasshaank, HF: @shashaank0707)
-*   **Hackathon**: Meta + PyTorch + HuggingFace OpenEnv 2024
+## License & Attribution
+
+**License:** MIT — see [LICENSE](LICENSE)
+
+**Author:** Shashaank | GitHub: [@shasshaank](https://github.com/shasshaank) | HF: [@shashaank0707](https://huggingface.co/shashaank0707)
+
+**Live Environment:** https://huggingface.co/spaces/shashaank0707/AgentDebugger-env
+
+**Submitted to:** Meta + PyTorch + HuggingFace OpenEnv Hackathon
 
 ---
 
-### ✅ Submission Integrity
-- **Commit SHA**: `159a5faf82fc1ab3709f9674becf9a3ec55cf562`
-- **Last Verified Sync**: 2026-04-08
-- **Platform Match**: GitHub and HF Space are identical at this HEAD.
+## Submission Integrity
+
+- **Commit SHA:** `e93446da6e57b3f582db65a947dc0abef18e66c6`
+- **Last Verified Sync:** 2026-04-09
+- **Platform Match:** GitHub and HF Space are in sync at this HEAD