Spaces:
Sleeping
Sleeping
File size: 3,379 Bytes
d3d0e0e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | # HITL Trace Implementation Validation
## Implementation Summary
The HITL (Human-In-The-Loop) checkpoint review steps are now recorded as separate steps in `trace.json` for later review, in addition to existing event emission in `run_events.jsonl`.
## Key Changes
### File Modified
- `src/data_agent_baseline/langgraph_agent/graph.py`
### New Function Added
- `_record_hitl_step()`: Records HITL checkpoint decisions as trace steps
### Integration Points
HITL steps are recorded at 4 decision points in `_maybe_handle_planner_checkpoint()`:
1. **Approve Path**
- Decision: User approves the planner's original plan
- Step recorded with original_plan = revised_plan
2. **Cancel Path**
- Decision: User cancels the task
- Step recorded with human_instruction included
- Includes cancellation reason in message
3. **Revise Path**
- Decision: User submits steering instruction for replanning
- Replanning occurs via planner_node()
- Step recorded with both original_plan and revised_plan
- Includes delta comparison for changes
4. **Timeout Auto-Approve Path**
- Decision: Timeout elapsed, plan auto-approved
- Step recorded with timeout_seconds and default_action_on_timeout
## Trace.json Structure
Each HITL step in `trace.json["steps"]` has this structure:
```json
{
"step_index": <number>,
"action": "planner_review",
"phase": "planner_review",
"thought": "Human-in-the-loop checkpoint review (checkpoint_id=...)",
"action_input": {
"checkpoint_id": "cp_...",
"decision": "approve|cancel|revise|timeout_auto_approve"
},
"raw_response": "(human decision)",
"observation": {
"ok": true,
"tool": "planner_review",
"content": {
"checkpoint_id": "cp_...",
"decision": "...",
"message": "...",
"original_plan": { ... },
"revised_plan": { ... },
"human_instruction": "...",
"timeout_seconds": 60,
"default_action_on_timeout": "continue"
}
},
"ok": true
}
```
## Usage Example
When viewing trace.json after a guided run with HITL:
```bash
cd /data3/dataFAIR/kdd-dev/public/artifacts/runs/
jq '.steps[-1]' <run_id>/task_<id>/trace.json # View last HITL step
```
This shows:
- The checkpoint ID
- The human decision (approve, cancel, or revise)
- The original plan that was presented
- The revised plan (if applicable)
- Human steering instructions (if applicable)
- Timeout configuration
## Backward Compatibility
- No changes to existing trace structure
- HITL steps are appended to steps array
- Existing HITL event emission in run_events.jsonl unchanged
- Non-HITL runs are unaffected
## Verification
To verify implementation:
```bash
# 1. Run a guided task with HITL enabled
python main.py --mode guided --task-id task_145 ...
# 2. Check trace.json for HITL step
cat artifacts/runs/<run_id>/task_145/trace.json | jq '.steps[] | select(.action == "planner_review")'
# 3. Verify run_events.jsonl still has HITL events
cat artifacts/runs/<run_id>/run_events.jsonl | grep checkpoint_reached
```
## Benefits
- ✅ HITL decisions persist in trace.json for audit/review
- ✅ Full checkpoint lifecycle visible (creation, decision, resolution)
- ✅ Original and revised plans captured together
- ✅ Human steering instructions preserved
- ✅ Compatible with existing visualization tools
- ✅ Available for long-term historical analysis
|