REWARD_SYSTEM_GUIDE.md

# Reward System Implementation Guide

This document shows how the reward system is implemented in code and how to use it.

## Module Documentation

The reward system architecture is documented at the module level:

```python

import server.env

print(server.env.__doc__)

```

Output shows all 6 reward components and the final computation formula.

## Reward Constants

All reward constants are defined in `server/env.py` (lines 57-87):

```python

# Component 1: Score improvement reward

PROGRESS_SCALE = 0.25



# Component 2: Syntax/compilation fix reward

SYNTAX_FIX_BONUS = 0.35



# Component 3: Test improvement reward

TEST_PASS_REWARD_SCALE = 0.30



# Component 4: Code quality reward

QUALITY_BONUS_SCALE = 0.15



# Component 5: Stagnation penalty

STAGNATION_PENALTY = 0.10



# Component 6: Regression penalty

REGRESSION_PENALTY_SCALE = 0.20



# One-time completion bonus

COMPLETION_BONUS = 0.50



# Invalid/error penalties

INVALID_ACTION_PENALTY = 0.15

TIMEOUT_PENALTY = 0.15

```

To tune the reward system, edit these constants and re-test.

## RewardDetails Model Documentation

Located in `models.py` (lines 26-80):

```python

from models import RewardDetails

print(RewardDetails.__doc__)

```

Shows all 15 fields and their meanings:
- `value`: Final scalar reward [-1.0, +1.0]
- `progress_delta`: Score improvement component
- `syntax_reward`: Syntax fix bonus
- `test_reward`: Test improvement bonus
- `quality_bonus`: Code quality improvement
- `stagnation_penalty`: Unchanged code penalty
- `regression_penalty`: Score decline penalty
- `reason`: Human-readable explanation
- `prev_score`, `curr_score`: Score before/after
- `code_changed`: Whether code was modified

## Core Computation Method

The main reward computation is in `_compute_reward_components()` (server/env.py, lines 507-703):

```python

def _compute_reward_components(

    self,

    curr_score: float,

    prev_score: float,

    curr_grade: TaskGrade,

    code_changed: bool,

    prev_grade_score: float = 0.0,

) -> dict:

    """Compute all six reward components and return combined result."""

```

### What It Does

1. **Initializes** empty component dict
2. **Computes each component**:
   - Progress: Score improvement scaled by PROGRESS_SCALE

   - Syntax: One-time bonus if first compile

   - Test: Test pass rate improvement scaled by TEST_PASS_REWARD_SCALE
   - Quality: Code quality improvement scaled by QUALITY_BONUS_SCALE
   - Stagnation: Penalty if code unchanged
   - Regression: Penalty if score decreased
3. **Combines**: Sums positives, subtracts negatives
4. **Clamps**: Bounds result to [-1.0, +1.0]

### Key Design Decisions

- **Monotonic tracking**: Best test rate and quality in episode are tracked
- **One-time bonuses**: Syntax reward awarded once per episode
- **Scale capping**: Each component has a maximum (e.g., progress max +0.25)
- **Timeout handling**: Special penalty instead of score-based
- **Clamping**: Final reward bounded for numerical stability

## Debug Logging

When `verbose=True`, the environment prints detailed debug output via `_log_debug_step()`:

```python

env = PythonCodeReviewEnvironment(verbose=True)

obs = env.reset()

obs = env.step(action)

```

Output format:
```

Step  1 | Score: 0.698 | Delta: +0.698 | Reward: +0.4239 | Changed: False

         | Progress=+0.174 | Quality=+0.149 | Stagnation=+0.100

         | Reason: Syntax error detected: '(' was never closed

```

Shows:
- Step number
- Current score and delta from previous
- Final reward value
- Whether code changed
- Non-zero components only
- Human-readable reason

## Example: Full Episode with Rewards

```python

from server.env import PythonCodeReviewEnvironment

from models import PythonCodeReviewAction



env = PythonCodeReviewEnvironment(verbose=True)

obs = env.reset(task_id='syntax-fix-easy')



# Step 1: Analyze (no code change)

action = PythonCodeReviewAction(action_type='analyze_code')

obs = env.step(action)

print(f"Reward 1: {obs.reward_details.value:.4f}")



# Step 2: Edit with fix

code = 'x = 1; y = 2; print(x + y)'

action = PythonCodeReviewAction(action_type='edit_code', code=code)

obs = env.step(action)

print(f"Reward 2: {obs.reward_details.value:.4f}")



# Step 3: Submit

action = PythonCodeReviewAction(action_type='submit_solution')

obs = env.step(action)

print(f"Final Reward: {obs.reward_details.value:.4f}")

```

## Interpreting Rewards

### Positive Rewards (+0 to +1.0)
- **+0.5 - +1.0**: Major progress (syntax fix, many tests passing)
- **+0.2 - +0.5**: Good progress (score improvement, test gains)
- **+0.0 - +0.2**: Small progress (quality improvement, minor gains)

### Negative Rewards (−1.0 to −0)
- **−0.1 - 0**: Stagnation (analyzed without changing code)
- **−0.2 - −0.1**: Slight regression (small score drop)
- **−0.5 - −0.2**: Major regression (significant score drop)
- **−1.0 - −0.5**: Invalid action or timeout

## Tuning the Reward System

### For Faster Early Learning
↑ Increase `SYNTAX_FIX_BONUS` and `COMPLETION_BONUS`

### To Encourage Editing Over Analysis
↑ Increase `STAGNATION_PENALTY`

### To Reward Test Improvements More
↑ Increase `TEST_PASS_REWARD_SCALE`

### To Penalize Mistakes More
↑ Increase `REGRESSION_PENALTY_SCALE`

### To Balance All Components
Adjust the Scale constants (all in range 0.15-0.35 for stability)

## Accessing Documentation Programmatically

```python

from server.env import PythonCodeReviewEnvironment

from models import RewardDetails

import server.env



# Module-level architecture

print(server.env.__doc__)



# RewardDetails fields

print(RewardDetails.__doc__)



# One method

env = PythonCodeReviewEnvironment()

help(env._compute_reward_components)

```

All major functions and classes have comprehensive docstrings.