docs/implementation/HITL_TRACE_VALIDATION.md

HITL Trace Implementation Validation

Implementation Summary

The HITL (Human-In-The-Loop) checkpoint review steps are now recorded as separate steps in trace.json for later review, in addition to existing event emission in run_events.jsonl.

Key Changes

File Modified

• src/data_agent_baseline/langgraph_agent/graph.py

New Function Added

• _record_hitl_step(): Records HITL checkpoint decisions as trace steps

Integration Points

HITL steps are recorded at 4 decision points in _maybe_handle_planner_checkpoint():

1. Approve Path

   • Decision: User approves the planner's original plan
   • Step recorded with original_plan = revised_plan

2. Cancel Path

   • Decision: User cancels the task
   • Step recorded with human_instruction included
   • Includes cancellation reason in message

3. Revise Path

   • Decision: User submits steering instruction for replanning
   • Replanning occurs via planner_node()
   • Step recorded with both original_plan and revised_plan
   • Includes delta comparison for changes

4. Timeout Auto-Approve Path

   • Decision: Timeout elapsed, plan auto-approved
   • Step recorded with timeout_seconds and default_action_on_timeout

Trace.json Structure

Each HITL step in trace.json["steps"] has this structure:

{
  "step_index": <number>,
  "action": "planner_review",
  "phase": "planner_review",
  "thought": "Human-in-the-loop checkpoint review (checkpoint_id=...)",
  "action_input": {
    "checkpoint_id": "cp_...",
    "decision": "approve|cancel|revise|timeout_auto_approve"
  },
  "raw_response": "(human decision)",
  "observation": {
    "ok": true,
    "tool": "planner_review",
    "content": {
      "checkpoint_id": "cp_...",
      "decision": "...",
      "message": "...",
      "original_plan": { ... },
      "revised_plan": { ... },
      "human_instruction": "...",
      "timeout_seconds": 60,
      "default_action_on_timeout": "continue"
    }
  },
  "ok": true
}

Usage Example

When viewing trace.json after a guided run with HITL:

cd /data3/dataFAIR/kdd-dev/public/artifacts/runs/
jq '.steps[-1]' <run_id>/task_<id>/trace.json  # View last HITL step

This shows:

• The checkpoint ID
• The human decision (approve, cancel, or revise)
• The original plan that was presented
• The revised plan (if applicable)
• Human steering instructions (if applicable)
• Timeout configuration

Backward Compatibility

• No changes to existing trace structure
• HITL steps are appended to steps array
• Existing HITL event emission in run_events.jsonl unchanged
• Non-HITL runs are unaffected

Verification

To verify implementation:

# 1. Run a guided task with HITL enabled
python main.py --mode guided --task-id task_145 ...

# 2. Check trace.json for HITL step
cat artifacts/runs/<run_id>/task_145/trace.json | jq '.steps[] | select(.action == "planner_review")'

# 3. Verify run_events.jsonl still has HITL events
cat artifacts/runs/<run_id>/run_events.jsonl | grep checkpoint_reached

Benefits

• ✅ HITL decisions persist in trace.json for audit/review
• ✅ Full checkpoint lifecycle visible (creation, decision, resolution)
• ✅ Original and revised plans captured together
• ✅ Human steering instructions preserved
• ✅ Compatible with existing visualization tools
• ✅ Available for long-term historical analysis