u2bench-anon/u2-bench-code-review

U2-Bench Code (Anonymous Review Repo)

This repository is an anonymized code artifact accompanying the U2-Bench dataset submission to NeurIPS 2026 E&D.

What is included

• A minimal executable pipeline script (run_minimal_pipeline.sh)
• A schema validator for released JSONL files (scripts/validate_u2_schema.py)
• A minimal scoring script for prediction files (scripts/evaluate_predictions.py)
• Dependency list (requirements.txt)

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python scripts/validate_u2_schema.py --input data/u2_bench_non_stem_qa.jsonl

Evaluate predictions (optional):

python scripts/evaluate_predictions.py \
  --gold data/u2_bench_non_stem_qa.jsonl \
  --predictions data/predictions.jsonl

Or run both in one command:

./run_minimal_pipeline.sh data/u2_bench_non_stem_qa.jsonl data/predictions.jsonl

Expected dataset schema

Each line in JSONL must contain:

• qhash
• question
• gold_answer

Prediction JSONL used by evaluate_predictions.py must contain:

• qhash
• predicted

Reproducibility notes

The full training/evaluation pipeline used for construction is documented in the paper and supplement. This repository provides an executable, anonymized review artifact focused on data schema checks and release validation.

Anonymity

This repository intentionally omits author and institution identifiers for double-blind review.