| --- |
| license: mit |
| --- |
| |
| # SWE-Bench Trajectory Eval Bundle (v2) |
|
|
| Airgapped-ready bundle for the trajectory-probe downstream eval. |
|
|
| ## Changes from v1 |
|
|
| - **django dropped** (230 base_commits × 5 agents = 1139 trajectories removed). |
| django's blobs were too slow to prefetch at scale; the other 11 repos |
| give 1317 trajectories which is plenty. |
| - **Sparse-checkout blobs only**. For each (repo, base_commit), only the |
| files any trajectory's patch touches are fetched. Dramatically smaller |
| than the v1 "full-tree" approach; still enough for the graph encoder |
| to see the patched files + their containing package. |
|
|
| ## Contents (570 MB compressed) |
|
|
| - `specs.jsonl`: 1317 SWE-Bench Verified agent trajectories (down from |
| 2456). 46% resolved. 11 repos, 269 unique base_commits. |
| - `repos/`: 11 blobless clones with sparse-checkout already configured |
| per base_commit. Working trees on-disk reflect the UNION of touched |
| paths across all trajectories for each base_commit. |
| - `graphjepa/`: pipeline code (trajectory_pipeline with `extract_patch_paths` |
| and `set_sparse_paths`, trajectory_realize, trajectory_probe, checks). |
| - `harvest.log`. |
|
|
| ## Dataset |
|
|
| | Repo | Trajs | Resolved | |
| |---|---|---| |
| | sympy/sympy | 370 | 48% | |
| | sphinx-doc/sphinx | 207 | 32% | |
| | matplotlib | 167 | 41% | |
| | scikit-learn | 158 | 70% | |
| | astropy | 110 | 42% | |
| | pydata/xarray | 108 | 51% | |
| | pytest-dev/pytest | 95 | 49% | |
| | pylint-dev/pylint | 49 | 22% | |
| | psf/requests | 38 | 61% | |
| | mwaskom/seaborn | 10 | 0% | |
| | pallets/flask | 5 | 80% | |
| | **total** | **1317** | **46%** | |
|
|
| ## Downstream workflow (on airgapped big) |
|
|
| ```bash |
| huggingface-cli download IDMedicine/code-graph-trajeval-v2 traj_full_bundle.tar.gz \ |
| --local-dir outputs/traj_real |
| |
| cd /opt/train/code-transformer/code-transformer |
| tar -xzf outputs/traj_real/traj_full_bundle.tar.gz |
| rsync -a traj_full/graphjepa/ graphjepa/ |
| mkdir -p outputs/traj_real |
| cp traj_full/specs.jsonl outputs/traj_real/ |
| mv traj_full/repos outputs/traj_real/repos |
| |
| export BERT_MODEL_PATH=/opt/train/datasets/bert-base-uncased |
| GPUS=0,2,3,4,5,6,7 bash graphjepa/scripts/trajectory_full.sh |
| tail -f outputs/traj_real/logs/prebuild_shard*_gpu*.log |
| ``` |
|
|
| Because the clone already has the right blobs prefetched on laptop, |
| the airgapped big machine can checkout every base_commit without any |
| network access. |
| |
