docs/overview/PHASE1_EVALUATION_GUIDE.md

Phase 1 Baseline Evaluation Harness - Usage Guide

Overview

This Phase 1 integration enables evaluation of baseline ReAct agent runs using a standardized evaluation pipeline. The system adapts baseline traces to a canonical schema and computes metrics compatible with future multi-agent systems.

Quick Start

Run Baseline Evaluation

cd /data3/dataFAIR/kdd-dev/public

# Evaluate a baseline run
dabench eval-baseline <run_id>

# Example:
dabench eval-baseline 20260613T114457Z

Output

The evaluation generates three files in <run_dir>/baseline_evaluation/:

baseline_evaluation/
├── task_results.csv          # Per-task metrics
├── summary_metrics.json      # Aggregated statistics
└── evaluation_report.md      # Scientific analysis

Command Options

dabench eval-baseline <run_id> [OPTIONS]

Arguments:
  run_id              Run ID (directory name under artifacts/runs/)

Options:
  --task-root PATH    Root directory containing task metadata
                      Default: /data3/dataFAIR/kdd-dev/public/input_full
  
  --gold-root PATH    Root directory containing gold answer files
                      Default: /data3/dataFAIR/kdd-dev/public/output
  
  --output-dir PATH   Output directory for evaluation results
                      Default: <run_dir>/baseline_evaluation

Architecture

Components

1. Canonical Evaluation Schema (src/data_agent_baseline/evaluation/__init__.py)

   • Unified trace representation for all agent types
   • Compatible with baseline and LangGraph traces
   • Extensible for future agent architectures

2. Baseline Trace Adapter (src/data_agent_baseline/evaluation/baseline_adapter.py)

   • Converts baseline traces to canonical schema
   • Enriches traces with task metadata
   • Computes derivable metrics from steps

3. Phase 1 Evaluator (src/data_agent_baseline/evaluation/phase1_evaluator.py)

   • Computes baseline-compatible metrics
   • Scores predictions against gold answers
   • Classifies failures and buckets

4. Report Generator (src/data_agent_baseline/evaluation/report_generator.py)

   • Generates CSV, JSON, and Markdown reports
   • Computes summary statistics
   • Creates scientific evaluation reports

Data Flow

Baseline Trace (trace.json)
    ↓
BaselineTraceAdapter.normalize()
    ↓
CanonicalTrace (normalized schema)
    ↓
Phase1Evaluator.evaluate_task()
    ↓
EvaluationResult (computed metrics)
    ↓
Phase1ReportGenerator.generate_all_reports()
    ↓
Output Files (CSV, JSON, MD)

Metrics

Phase 1 Metrics (Available)

✅ Accuracy Metrics:

• Overall accuracy
• Per-difficulty accuracy (Easy/Medium/Hard/Extreme)
• Answer precision, recall, F1
• Column precision, recall, F1
• Row precision, recall, F1

✅ Efficiency Metrics:

• Average steps per task
• Average tool calls per task
• Average runtime per task
• Tool diversity (unique tools / total calls)
• Tool efficiency (1 - failures / total)

✅ Reliability Metrics:

• Success rate
• Execution failure rate
• Timeout rate
• Tool error rate

✅ Failure Classification:

• Failure categories (timeout, planning, execution, wrong_answer)
• Failure stages (execution, answer_generation)
• Root causes (timeout, insufficient_steps, filter_logic, etc.)
• Recoverability assessment

✅ Bucket Distribution:

• perfect (score >= 0.999)
• wrong_row_count
• wrong_col_count
• partial_correct (score > 0.5)
• mostly_wrong (score > 0.0)
• completely_wrong
• no_gold (no gold answer available)

Phase 1 Limitations (Not Available)

❌ Token Metrics:

• Total tokens
• Prompt tokens
• Completion tokens
• Estimated cost

❌ Phase-Specific Metrics:

• Explore phase timing
• Planner phase timing
• Execute phase timing
• Critic phase timing

❌ Advanced Metrics:

• Confidence scores
• Confidence calibration
• Replan counts
• Recovery attempts
• Self-correction loops

Why? Baseline traces don't include:

• Token usage tracking
• Phase labels on steps
• Confidence scores
• Recovery/replan signals

These will be available in future phases with enhanced agents.

Output Format

task_results.csv

Per-task metrics in CSV format:

run_id,task_id,agent_type,difficulty,execution_success,execution_time,final_score,
answer_f1,column_f1,row_f1,trajectory_length,tool_calls,tool_failures,
tool_efficiency,unique_tools_used,tool_diversity,total_tokens,llm_calls,
timeout_occurred,failure_category,failure_stage,root_cause,bucket,
pred_rows,pred_cols,gold_rows,gold_cols

summary_metrics.json

Aggregated statistics:

{
  "overall": {
    "total_tasks": 10,
    "successful_tasks": 9,
    "success_rate": 0.9,
    "perfect_rate": 0.7,
    "average_score": 0.85,
    "average_trajectory_length": 8.5,
    "average_tool_calls": 6.2,
    "average_execution_time": 15.3,
    "tool_error_rate": 0.05,
    "timeout_rate": 0.0,
    "execution_failure_rate": 0.1
  },
  "by_difficulty": {
    "Easy": {...},
    "Medium": {...},
    "Hard": {...}
  },
  "failure_analysis": {
    "failure_categories": {...},
    "failure_stages": {...},
    "root_causes": {...}
  },
  "bucket_distribution": {
    "perfect": 7,
    "wrong_row_count": 2,
    "no_gold": 1
  }
}

evaluation_report.md

Scientific evaluation report with:

• Experimental setup
• Overall results table
• Performance by difficulty
• Reliability analysis
• Failure analysis (top categories and causes)
• Efficiency analysis
• Phase 1 limitations
• Recommendations for improvement

Comparing Baseline vs Multi-Agent

The canonical schema enables direct comparison:

from data_agent_baseline.evaluation.baseline_adapter import BaselineTraceAdapter
from data_agent_baseline.evaluation.phase1_evaluator import Phase1Evaluator

# Evaluate baseline run
baseline_adapter = BaselineTraceAdapter(task_root=task_root)
baseline_traces = baseline_adapter.normalize_run(baseline_run_path, run_id="baseline")

# Evaluate LangGraph run (future)
langgraph_adapter = LangGraphTraceAdapter(task_root=task_root)
langgraph_traces = langgraph_adapter.normalize_run(langgraph_run_path, run_id="langgraph")

# Compare using same evaluator
evaluator = Phase1Evaluator(gold_root=gold_root)
baseline_results = evaluator.evaluate_run(baseline_traces)
langgraph_results = evaluator.evaluate_run(langgraph_traces)

# Generate comparison report
compare_agents(baseline_results, langgraph_results)

Extending to New Agent Types

To add a new agent type:

1. Create an adapter implementing:

   class NewAgentAdapter:
       def normalize(self, trace_path: Path, run_id: str) -> CanonicalTrace:
           # Convert agent-specific trace to canonical schema
           pass
   

2. Register agent type in canonical schema:

   AgentType = Literal["baseline_react", "langgraph_agent", "new_agent"]
   

3. Evaluate using existing pipeline:

   adapter = NewAgentAdapter(task_root=task_root)
   traces = adapter.normalize_run(run_path, run_id)
   evaluator = Phase1Evaluator(gold_root=gold_root)
   results = evaluator.evaluate_run(traces)
   

Programmatic Usage

from pathlib import Path
from data_agent_baseline.evaluation.baseline_adapter import BaselineTraceAdapter
from data_agent_baseline.evaluation.phase1_evaluator import Phase1Evaluator
from data_agent_baseline.evaluation.report_generator import Phase1ReportGenerator

# Setup paths
run_path = Path("/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260613T114457Z")
task_root = Path("/data3/dataFAIR/kdd-dev/public/input_full")
gold_root = Path("/data3/dataFAIR/kdd-dev/public/output")
output_dir = run_path / "baseline_evaluation"

# Normalize traces
adapter = BaselineTraceAdapter(task_root=task_root)
canonical_traces = adapter.normalize_run(run_path, run_id="20260613T114457Z")

# Evaluate
evaluator = Phase1Evaluator(gold_root=gold_root)
results = evaluator.evaluate_run(canonical_traces)

# Generate reports
generator = Phase1ReportGenerator(output_dir=output_dir)
outputs = generator.generate_all_reports(results, run_id="20260613T114457Z")

print(f"Task results: {outputs['task_results']}")
print(f"Summary: {outputs['summary_metrics']}")
print(f"Report: {outputs['evaluation_report']}")

Troubleshooting

"Trace file not found"

Ensure the run directory contains task subdirectories with trace.json files:

<run_id>/
├── task_11/
│   └── trace.json
├── task_22/
│   └── trace.json
└── ...

"No gold answer" / bucket="no_gold"

The gold answer file is missing. Ensure <gold_root>/<task_id>/gold.csv exists.

This is not an error - tasks without gold answers are still evaluated for execution metrics.

"Error normalizing traces"

Check that trace.json is valid JSON with expected fields:

• task_id
• answer (optional)
• steps (array)
• succeeded
• e2e_elapsed_seconds

Score is 0.0 but execution succeeded

This usually means:

1. No gold answer available (bucket="no_gold"), OR
2. Prediction doesn't match gold at all (bucket="completely_wrong")

Check the bucket classification in task_results.csv.

Future Enhancements

Phase 2: Multi-Agent Support

• Add LangGraph trace adapter
• Compare baseline vs multi-agent performance
• Measure planning efficiency

Phase 3: Token Tracking

• Instrument baseline with token counting
• Compute cost metrics
• Optimize for cost/performance tradeoff

Phase 4: Confidence Calibration

• Add confidence scores to baseline
• Measure calibration accuracy
• Implement uncertainty quantification

Phase 5: Human-in-the-Loop

• Track human interventions
• Measure autonomy score
• Analyze when help is needed

References

• Baseline architecture: BASELINE_ARCHITECTURE.md
• Canonical schema: src/data_agent_baseline/evaluation/__init__.py
• Existing evaluation: src/data_agent_baseline/langgraph_agent/eval_v2.py
• Scoring function: src/data_agent_baseline/langgraph_agent/evaluator.py