stack/docs/archive/EVAL_PLAN.md

Evaluation Plan - Stack 2.9

Overview

This document outlines the comprehensive evaluation plan for Stack 2.9, detailing the methodology, hardware requirements, timeline, and result publication strategy. The evaluation will be conducted post-training to provide rigorous performance benchmarks across multiple dimensions.

Evaluation Objectives

1. Quantify Coding Ability: Measure performance on standard coding benchmarks (HumanEval, MBPP, SWE-bench)
2. Assess Tool Use Proficiency: Evaluate OpenClaw-specific tool calling accuracy and workflow completion
3. Validate Voice Integration: Test voice command processing and response generation quality
4. Benchmark Efficiency: Measure throughput, latency, and hardware requirements
5. Ensure Quality: Comprehensive testing before OpenRouter listing and public release

Hardware Requirements

Primary Evaluation Environment

• GPU: NVIDIA A100 80GB (or equivalent) with CUDA 12.x
• Count: Minimum 2 GPUs for parallel evaluation (reduces total time)
• CPU: 16+ cores (AMD EPYC / Intel Xeon)
• RAM: 128GB+ system memory
• Storage: 2TB NVMe SSD for datasets and model checkpoints
• Network: High-speed interconnect (NVLink) for multi-GPU setups

Optional/Alternative Configurations

• H100 80GB: Faster inference for time-sensitive evaluations
• A100 40GB: Sufficient for quantization tests (4-bit models)
• Multi-node cluster: For distributed evaluation across multiple machines

Software Stack

• OS: Ubuntu 22.04 LTS (or similar)
• Deep Learning Framework: PyTorch 2.1+ with CUDA support
• Inference Engine: vLLM 0.4+ for throughput benchmarking; Hugging Face Transformers for accurate sampling
• Quantization: AWQ, GPTQ, bitsandbytes for 4-bit/8-bit evaluations
• Evaluation Libraries: LangChain (for tool use), pytest (for code execution), custom scripts

Benchmark Suite

1. HumanEval (OpenAI)

• Description: 164 Python coding problems requiring function completion
• Metrics: Pass@1, Pass@10, Pass@100 (with 100+ generations for robust estimates)
• Format: Single function completion with unit test verification
• Expected Time: 2-4 hours (depending on batch size and parallelism)
• Resource Estimate: ~20GB VRAM for 32B model in FP16; ~10GB for 4-bit quantized

2. MBPP (Mostly Basic Python Programming)

• Description: 500 Python function synthesis problems from Google
• Metrics: Pass@1, execution accuracy, time to solution
• Format: Function generation with multiple test cases per problem
• Expected Time: 6-10 hours
• Resource Estimate: Similar to HumanEval

3. SWE-bench

• Description: Real-world GitHub issues requiring code modifications (full repository context)
• Metrics: Resolution rate (percentage of issues fully resolved), edit similarity, test pass rate
• Format: Multi-file problem solving with repository-level context
• Expected Time: 24-48 hours (most intensive)
• Resource Estimate: 80GB VRAM required for 128K context; may need sequence parallelism

4. Custom Tool Use Benchmark (OpenClaw)

• Description: 500 tasks covering OpenClaw-specific operations:
  • File operations (read, write, move, delete, search)
  • System commands (process management, environment queries)
  • API calls (HTTP requests, data transformation)
  • Multi-step workflows (combining multiple tools)
  • Error handling and recovery
• Metrics: Task completion rate (%), tool call accuracy (%), parameter correctness (%), workflow success (%)
• Expected Time: 4-6 hours
• Resource Estimate: Similar to HumanEval

5. Long Context Benchmark (Custom)

• Description: Synthetic and real-world tasks requiring 64K-128K token context
• Metrics: Accuracy at different context lengths (8K, 32K, 64K, 128K)
• Format: Needle-in-haystack tests, multi-document Q&A, long codebase reasoning
• Expected Time: 2-3 hours
• Resource Estimate: 80GB VRAM for full context; may need FlashAttention or similar optimizations

6. Additional Evaluations (Optional)

• GSM8K: Mathematical reasoning (1319 problems) — 2-3 hours
• MMLU: Multidisciplinary knowledge (optional) — 4-6 hours
• Voice Integration: Speech-to-text + code generation latency and accuracy (requires additional audio dataset)
• Throughput Benchmark: Tokens/second under various configurations (batch sizes, quantization)

Evaluation Process

Phase 1: Preparation (Pre-Evaluation)

1. Environment Setup

   • Provision hardware with appropriate drivers and CUDA
   • Install dependencies (PyTorch, vLLM, evaluation scripts)
   • Download model weights from Hugging Face or local storage
   • Prepare datasets (HumanEval, MBPP, SWE-bench, custom tool benchmark)

2. Validation

   • Smoke test: Generate on 5 examples from each benchmark
   • Verify evaluation scripts are functioning correctly
   • Check that output format matches expected submission format
   • Ensure results are being recorded in structured format (JSON/CSV)

Phase 2: Execution (Core Evaluation)

Schedule (Parallelized Where Possible)

Day 1:
- Morning (4h): HumanEval (batch on 2 GPUs)
- Afternoon (4h): MBPP (batch on 2 GPUs)
- Evening: Preliminary results review

Day 2:
- Morning (4h): Tool Use Benchmark (batch on 2 GPUs)
- Afternoon (4h): Long Context Benchmark (single GPU with 80GB)
- Evening: Throughput benchmarking (various configs)

Day 3:
- Full day (12h): SWE-bench (single GPU, longest-running)
- Night: GSM8K and optional evaluations (if hardware available)

Day 4:
- Morning: Final data collection
- Afternoon: Result aggregation and verification
- Evening: Generate preliminary report draft

Parallelization Strategy

• Independent benchmarks (HumanEval, MBPP, Tool Use) can run concurrently on separate GPUs
• SWE-bench requires most memory; run sequentially on dedicated GPU
• Long context tests require full 80GB; schedule during off-peak
• Throughput tests can interleave with other benchmarks (minimal impact)

Phase 3: Analysis and Reporting

1. Data Aggregation

   • Collect all JSON results into master spreadsheet
   • Compute pass@k metrics with confidence intervals
   • Cross-validate between benchmark runs (re-run if variance >2%)

2. Comparative Analysis

   • Compare against Qwen2.5-Coder-32B baseline (where publicly available)
   • Benchmark against similar models (CodeLlama-34B, StarCoder2-15B, etc.)
   • Tabulate results in standardized format

3. Report Generation

   • Create detailed markdown report with methodology
   • Generate summary tables for quick reference
   • Include error analysis and failure case examples
   • Document any issues or anomalies encountered

4. Result Verification

   • Have 2+ team members independently verify calculations
   • Re-run suspicious or outlier results
   • Ensure reproducibility claims are valid

Result Publication Strategy

1. Immediate Release (Upon Completion)

• BENCHMARKS.md: High-level summary table with scores and basic metrics
• BENCHMARKS_DETAILED.md: Full results, methodology, and sample outputs
• GitHub Release: Tag with benchmark results and evaluation scripts
• OpenRouter Dashboard Update: Push verified metrics to model listing

2. Comprehensive Report (Within 1 Week)

• PDF Report: Professional formatted document for archival
• Blog Post: Community announcement with key findings and insights
• Social Media: Twitter/LinkedIn posts highlighting achievements
• Conference Submission: Consider submitting to ML/AI conferences

3. Long-term Archiving

• Zenodo/Figshare: DOI-minted archive of datasets and results
• Papers with Code: Submission for reproducibility tracking
• Model Cards: Update Hugging Face model card with final metrics
• OpenRouter Documentation: Permanent listing of verified performance

Quality Assurance

Reproducibility

• Publish all evaluation scripts and configuration files
• Provide Docker containers or conda environments for exact replication
• Document random seeds and sampling parameters
• Include generated outputs for sampling-based benchmarks

Validation Checks

• Consistency: Same results across multiple runs (within statistical variance)
• Sanity Checks: No impossible scores (>100% pass@k), reasonable standard errors
• Baseline Comparison: Qwen2.5-Coder-32B baseline reproduced if possible
• Failure Analysis: Review failed cases for systematic issues

Transparency

• Report both median and mean scores where applicable
• Include confidence intervals and standard deviations
• Document any exclusions or filtering applied to benchmarks
• Acknowledge limitations of each benchmark

Sample Evaluation Script (Template)

#!/bin/bash
# Stack 2.9 Benchmark Evaluation Runner
# Usage: ./run_eval.sh <benchmark_name>

set -e

MODEL_PATH="Qwen/Qwen2.5-Coder-32B-Instruct"
OUTPUT_DIR="./eval_results"
BENCHMARK=$1

mkdir -p $OUTPUT_DIR

case $BENCHMARK in
  "humaneval")
    # HumanEval evaluation
    python -m evaluate.humaneval \
      --model $MODEL_PATH \
      --output $OUTPUT_DIR/humaneval.json \
      --temperature 0.2 \
      --top_p 0.95 \
      --num_samples 100
    ;;

  "mbpp")
    # MBPP evaluation
    python -m evaluate.mbpp \
      --model $MODEL_PATH \
      --output $OUTPUT_DIR/mbpp.json \
      --temperature 0.2 \
      --top_p 0.95
    ;;

  "tool_use")
    # Custom tool use benchmark
    python -m evaluate.tool_use \
      --model $MODEL_PATH \
      --dataset ./data/tool_benchmark_500.json \
      --output $OUTPUT_DIR/tool_use.json
    ;;

  "swebench")
    # SWE-bench evaluation
    python -m evaluate.swe_bench \
      --model $MODEL_PATH \
      --split test \
      --output $OUTPUT_DIR/swebench.json \
      --max_context 128000
    ;;

  *)
    echo "Unknown benchmark: $BENCHMARK"
    exit 1
    ;;
esac

echo "Evaluation complete: $BENCHMARK results saved to $OUTPUT_DIR"

Timeline Summary

Phase	Duration	Milestones
Training	2-4 weeks	Model fine-tuning complete
Prep	3-5 days	Environment setup, datasets downloaded, smoke tests
Execution	4-7 days	Run all benchmarks (parallelized)
Analysis	3-5 days	Data aggregation, verification, report writing
Publication	2-3 days	Documentation updates, GitHub release, OpenRouter listing
Total	3-5 weeks	From training completion to public results

Key Dates

• Training Completion Target: [To be determined based on training schedule]
• Start Evaluation: Day 0 (immediately after training)
• Preliminary Results: Day 7
• Final Verified Results: Day 14-21
• Public Release: Day 21-28

Risk Mitigation

Potential Issues and Mitigations

Risk	Impact	Mitigation
Hardware failure	High downtime	Use cloud GPU instances with auto-recovery; keep backups
Dataset access issues	Evaluation delay	Pre-download all datasets; mirror critical benchmarks
Model loading crashes	Evaluation blocking	Test model loading thoroughly before starting; have checkpoint recovery
Memory overflow	Benchmark crashes	Use gradient checkpointing, quantization; monitor VRAM usage
Variance in results	Reliability concerns	Run multiple seeds; average results; report confidence intervals
Time overruns	Delayed publication	Prioritize key benchmarks (HumanEval, Tool Use) if needed; run SWE-bench offline

Success Criteria

The evaluation will be considered successful if:

1. ✅ All planned benchmarks (HumanEval, MBPP, Tool Use) complete successfully
2. ✅ SWE-bench evaluation produces valid results (or documented limitations)
3. ✅ Results are reproducible (same script yields consistent scores across runs)
4. ✅ Scores are competitive with base Qwen2.5-Coder-32B model (no significant regression in coding)
5. ✅ Tool use accuracy exceeds 85% (target for fine-tuning success)
6. ✅ Full documentation published within 4 weeks post-training
7. ✅ OpenRouter listing updated with verified metrics

Contact

For questions about the evaluation plan or to request early access to results, contact:

Evaluation Lead: OpenClaw Research Team
Email: evals@openclaw.org
GitHub Issues: https://github.com/openclaw/stack-2.9/issues

---

Last Updated: 2025-04-01
Status: Draft - Awaiting training completion