Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,60 @@
+# External benchmark evaluation
+
+Isambard-AI is blocked by a CPU-minutes quota, so we evaluate the trained
+checkpoints on a GPU elsewhere. The models are a custom LLaMA-style Transformer
+(`src/xscript/model.py`) + SentencePiece tokenizer — pure PyTorch, using
+`F.scaled_dot_product_attention`, no flash-attn / triton / custom kernels — so
+they run on any stock GPU (or CPU, slowly). Each model is ~1B params (fits any
+16GB GPU).
+
+The benchmark harness (`src/xscript/eval/bench.py`) wraps our model into
+lm-evaluation-harness and scores Global-MMLU, Belebele, and XNLI on each run's
+training languages. It is the *same* harness we would have run on-cluster, so
+numbers are directly comparable.
+
+## 1. Export from Isambard (already done by `upload_to_hf.py`)
+
+The private HF repo mirrors the on-cluster layout:
+
+```
+src/xscript/**                       # bundled model + harness code
+tokenizers/unigram_{starved,destarved}/{sp.model,meta.json}
+runs/<name>/checkpoints/final.pt     # 15 checkpoints, fp32, ~4GB each
+models.json                          # friendly name -> tokenizer + langs + orig run
+run_benchmarks.py  requirements.txt  README.md
+```
+
+Models use friendly names `<mixture>-<starved|fair>` (e.g. `en-fair`,
+`en-ar-starved`). `models.json` maps each to its real tokenizer.
+
+## 2. Run on your GPU
+
+```bash
+# clone just the runner (or download run_benchmarks.py + requirements.txt from the repo)
+pip install torch --index-url https://download.pytorch.org/whl/cu121   # match your CUDA
+pip install -r requirements.txt
+export HF_TOKEN=hf_...        # while the repo is private
+
+# quick validation pass over all 15 runs (~200 examples/task) -- do this FIRST
+python run_benchmarks.py --repo jvonrad/xscript-eval --limit 200
+
+# full suite once the quick pass looks sane
+python run_benchmarks.py --repo jvonrad/xscript-eval
+```
+
+The runner downloads one checkpoint at a time and deletes it after eval
+(`--keep-checkpoints` to retain), so peak disk is ~5GB. Results:
+
+```
+xscript_bench/results/bench/<run>_final.json    # per-run task accuracies
+xscript_bench/results/summary.json              # everything combined
+```
+
+Send those JSONs back for analysis.
+
+## Notes
+- `--runs en-starved en-fair` limits to a subset (friendly names).
+- `--tasks xnli_en xnli_de` overrides the task list (default = the run's langs).
+- Mono runs get 3 tasks (their one language), bilingual runs get 6 (both langs).
+- Scores are ordinary accuracy (`acc,none`); raw harness output is preserved in
+  each per-run JSON for length-normalized variants.