Spaces:
Running
Running
Commit History
-
deploy: overlay server/space/{README,Dockerfile} at root Running
Zhu Jiajun (jz28583)0425205 -
arxiv-citation: ship the heterograph (citations + author/category tables)
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Without the relation tables, agents collapsed arxiv-citation into a pure tabular regression on the per-paper feature CSV — figraph and ibm-aml run the same way. That's a significant under-fit; the task is a graph task because RelBench rel-arxiv:paper-citation is a temporal heterograph. Added 5 tables under arxiv-citation/ subdir on HF: citations.csv (Paper_ID, References_Paper_ID, Submission_Date) — 1.2M rows; filtered to Submission_Date < 2023-01-01 so test-period citations (which encode the labels) do not leak. paperAuthors.csv (Paper_ID, Author_ID, Submission_Date) — 617k rows. paperCategories.csv (Paper_ID, Category_ID, Submission_Date) — 155k rows. authors.csv (Author_ID, Name, ORCID) — 144k rows. categories.csv (Category_ID, Category) — 53 rows. Manifest gets one entry per file; the auto-generated agent instruction template now lists every file declared in `files:`…0309359 -
Cross-link leaderboard <-> dataset on HF + GitHub README
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)- Landing nav adds 'Dataset ↗' link next to GitHub - Per-task panel adds a 'data ↗' pill linking to that task's HF subdir - README adds two HF badges (leaderboard space + dataset repo) at the top Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>d05b5bd -
Restore /admin/insert for maintainer leaderboard corrections
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Removed earlier for integrity, but the maintainer needs a path to fix known-bad rows whose CSVs are no longer available (e.g. the autopipe-v2 figraph 0.824 entry that should be 0.788). Same bypass-key gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>53c64b6 -
Single-repo dataset hosting on HF (GLUE-style subdirs)
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)All 4 tasks now live under one public dataset repo `lanczos/graphtestbed-data`, organized as one subdir per task: graphtestbed-data/ arxiv-citation/{train,val,test}_features.csv + sample_submission.csv figraph/... (+ edges20{14..18}.csv) ibm-aml/... ieee-fraud-detection/... Why one repo: easier to manage permissions, README, versioning. New tasks become a `git push` of a folder, not a new HF repo per dataset. Changes: - server/space/push_data.py: one-shot uploader (folder per task), writes a top-level dataset card too. - manifest.yaml: every task points at lanczos/graphtestbed-data with a new `hf_subdir: <task>` field. - graphtestbed/fetch.py: prepends hf_subdir/ when downloading. - agents/common/tasks.py: instruction text now points the agent at the HF source explicitly + reminds them not to query upstream for test labels. Test labels remain in the private companion repo (lanczos/graphtestbed-gt) and never enter the public dataset. Co-Authored-By: Claude…bd3e9ac -
Overall: only complete agents get an average; rest sink to bottom
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Previously, an agent with one task scored had its single score shown as the average — making 'autopipe-v2 → figraph 0.824' appear ranked above fully-covered agents with averages in the 0.5s. That hides the real ranking from anyone scanning the page. New rule: average is computed only when the agent has scored on every task in the manifest. Incomplete agents render '—' and sort to the bottom (still ordered among themselves by # tasks covered, then name). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>ab28b31 -
Landing table: fix per-column sort + drop tasks col + move avg to last
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Sort bug: the per-task columns (figraph, arxiv-citation, …) had data-sort attrs but the JS sortKey switch only handled rank/agent/ primary/n_submissions/first_seen, so any other key fell through to `default: return 0` and rows kept their original order despite the arrow appearing. Replaced the switch with a header-position lookup that resolves the column index from the th element itself, so any future column added to the header row sorts automatically. Also: - Agent column no longer has data-sort (per request); cursor: default. - Removed the "tasks" (n_tasks/N) column from the Overall table. - Moved the Average column to the last position so per-task scores read left-to-right before the aggregate. - Empty-row colspan adjusted from 4+n to 3+n accordingly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>bf48fd7 -
Fix kaggle status parse + add /admin/repoll for stuck rows
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Bug: kaggle CLI prints status as "SubmissionStatus.COMPLETE" (an enum repr), not "complete". The poller's `.lower() == 'complete'` check never matched, so every kaggle submission timed out at 1800s with "polled without complete" even when Kaggle had already scored it. Fix: split on the last `.` and lowercase the suffix. /admin/repoll/<run_id>: re-trigger the poll for a row whose poller bailed out, without re-submitting (and burning Kaggle quota). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>f819f00 -
Fix kaggle async: use sentinel -1.0 for pending (NOT NULL) + correct col names
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1a157a1 -
Async Kaggle scoring: submit + insert pending row + background poll
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)The HF Space reverse proxy kills any request that holds the connection open past ~5 min, so synchronous Kaggle scoring (which can take 10+ min end-to-end) was failing even when the underlying submit + scoring both succeeded — the response just never reached the client. New flow for kaggle backend: 1. Server submits to Kaggle synchronously (~30s upload). 2. Inserts a 'pending' row into submissions (NULL primary_metric). 3. Spawns a daemon thread that polls Kaggle every 15s for up to 30 min and UPDATEs the row when complete (or with secondary.error on fail). 4. Returns 200 immediately with pending=true and the run_id. 5. New GET /run/<run_id> lets clients poll for the resolved score. Leaderboard queries now filter `WHERE primary_metric IS NOT NULL` so pending rows don't pollute rankings. Client (gtb submit) recognizes pending response and prints a follow-up hint instead of trying to print scores it doesn't have yet. Removed /admin/insert — the only honest path into the leaderboard…9cb903d -
Pin kaggle==1.7.4.5 (newer versions reject KAGGLE_API_TOKEN env auth)
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>00bf799 -
Surface kaggle submit stdout + rc in error msg
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>b226034 -
Add POST /admin/insert for direct leaderboard writes
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Use case: backends like Kaggle that can't always be proxied server-side. The maintainer runs the submit + poll locally with their own creds, then POSTs the resulting score here. Same bypass-key gate as /admin/delete. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>209df55
-
Self-contain CLIProxyAPI install + ieee-fraud schema cleanup
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Cliproxyapi: - install.sh fetches binary from upstream releases (vN linux-amd64) - _vendor/LICENSE is the only committed file (binary + config.yaml + proxy.log are .gitignored — install.sh produces them) - The Python shim already had config.example.yaml at the parent level ieee-fraud-detection manifest: - files now lists train/val/test_features.csv (with identity merged in) + sample_submission, instead of raw transaction/identity tables - description spells out the temporal val split (last 20% by TransactionDT) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>26a61c7 -
Add POST /admin/delete for bypass-keyed leaderboard cleanup
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Body: {entries: [{task, agent}, ...]}; deletes all submissions matching each pair. Same bypass-key gate as /submit's dry/unlimited mode. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>18fe8a0 -
binary check: keep float dtype during validation, reject non-{0,1}
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Old check cast to int first, which silently truncated probabilities (0.0007 → 0) so prob CSVs slipped past the binary gate. Stay in float land and require exact equality with 0.0 or 1.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>769f2a4 -
Ignore .claude/ session state
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>87f5650 -
ibm-aml → binary submission for minority F1 (server: no threshold)
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Rationale: F1 with prob+0.5 threshold is unfair when positive rate is 0.19% — a well-calibrated model gets F1 ≈ 0 because almost no probs exceed 0.5. Switch to binary submission so the agent owns the threshold (typically tuned on val), which is what the IBM Multi-GNN paper reports. - manifest: pred_dtype binary, drop auc_roc secondary (it's degenerate on binary input — would just be weird) - submit.py + server/api.py: validate ints in {0,1}; score f1 from raw - agents/common/tasks.py: parameterize pred_type_desc so binary tasks tell the agent to pick its own threshold Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>b3dbec5 -
push_to_space.sh: auto-inject HF_TOKEN into push URL when set
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>640f1df -
Landing: stack panel header (description above pills); style schema pill green; drop GT loaded badge
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>cf612db -
Update Tasks table: real row counts, Kaggle backend, ibm-aml F1 metric
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>9d28e20 -
ibm-aml metric → minority F1; add per-task metric rationale
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)ibm-aml: switched primary metric from auc_pr to f1. With ~0.19% positive rate, default sklearn f1_score on binary uses pos_label=1, which IS the laundering class — exactly the minority. AUC-PR moves to secondary so existing baselines remain comparable. All 4 task descriptions now spell out the metric choice + reasoning so the auto-generated agent instruction (agents/common/tasks.py reads from manifest description) explicitly tells the model what it's optimising and why. Same text shows on the Space landing page next to each task. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>54a4248 -
Add clamp_proxy + route AI-Build-AI through it (max_tokens fix)
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Root cause: aibuildai's bundled claude-cli (2.1.44) hardcodes max_tokens=128001 in its first /v1/messages call. Anthropic's current cap for claude-sonnet-4-6 / claude-opus-4-x is 128000, so every aibuildai run on a fresh task fails with 400 invalid_request_error after ~6 seconds. The bundled SDK ignores the documented CLAUDE_CODE_MAX_OUTPUT_TOKENS env and exposes no CLI flag for this in 2.1.44. (Bug confirmed by tracing the SDK's xl(model) function — for sonnet-4 it returns upperLimit=64000, the call site appears to compute upperLimit*2+1 = 128001 somewhere.) Fix: a tiny Flask passthrough proxy at agents/cliproxyapi/clamp_proxy.py that listens on port 8318, parses POST/PUT JSON bodies, clamps any max_tokens > 128000, and forwards everything else verbatim to CLIProxyAPI (streaming preserved). The AI-Build-AI runner overrides ANTHROPIC_BASE_URL to point at the clamp port. Other agents (MLEvolve, future OpenAI/Gemini clients) bypass it entirely — they never had this bug. End-to-end veri…61a7817 -
Add Overall tab + /leaderboard JSON; stage ibm-aml + arxiv data
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Overall tab is the new first tab on the landing: per-agent average across the 4 tasks, computed only over tasks the agent has actually submitted to (so a one-task agent isn't penalised by N/A on others). Per-task columns show individual scores (or — when missing). Sorted by average desc, ties broken by task coverage. New JSON endpoint GET /leaderboard (no slug) returns the same shape as JSON for client/CI consumption: {tasks: [...], rows: [{agent, average, n_tasks, per_task: {...}}, ...]}. Manifest files: blocks for ibm-aml + arxiv-citation reduced to the four files we actually stage (train/val/test_features + sample_submission); the original blocks listed text/category aux tables we don't materialise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>5955024 -
Redesign landing as a leaderboard-first single page
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Per open-llm-leaderboard pattern: header strip + nav, per-task tabs (with submission count badges) as the first-screen content, sortable + searchable table per task, top-3 gold/silver/bronze styling, hash-routed sections (/#about /#api /#figraph etc.) so links are shareable. Quickstart / endpoints / submission-format walls collapsed into secondary About + API tabs. Server-side initial render so first paint is instant; JS only enhances (client-side sort + search + Refresh hits the existing JSON endpoint). Zero new dependencies — still plain Flask + Jinja, no Gradio. API endpoints unchanged: POST /submit, GET /leaderboard/<task>, GET /healthz all return the same JSON shapes. Bypass key + dry mode behave as before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>6741538 -
Add Kaggle scoring backend + load all 4 task GTs
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Backend dispatch: cfg.get('backend', 'gt') → existing local-GT scoring (default) cfg['backend'] == 'kaggle' → forward to Kaggle's grading API: - kaggle competitions submit -c <competition> -f <csv> -m graphtestbed-<run_id> - poll kaggle competitions submissions every 15s until status=complete - return publicScore as primary, privateScore as secondary - 600s timeout cap (Kaggle scoring is 1-5 min typically) Manifest changes: arxiv-citation: switched to RelBench rel-arxiv:paper-citation split (train < 2022-01-01, test from 2023-01-01; 193,696 rows) ibm-aml: loaded HI-Small_Trans split per IBM Multi-GNN convention (sort by Timestamp, day-by-day 60/20/20; 863,900 test rows) ieee-fraud-detection: backend=kaggle, scoring against Kaggle's full hidden test split (506,691 rows). Maintainer needs to have accepted competition rules under the configured Kaggle …17942c5 -
Trim agents/cliproxyapi surface
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Drop unused symbols + the misleading config.example.yaml so the public shim is just six things: ProxyEndpoint, anthropic_env, openai_env, openai_yaml_block, is_ready, wait_until_ready. - Remove gemini_env() — no current consumer. - Remove spawn_proxy() ctx-manager — proxy always runs out of band on this deploy (long-lived OAuth state). - Remove agents/cliproxyapi/config.example.yaml — the real working config lives in the operator's own ~/.cli-proxy-api/. - Remove empty agents/common/tasks_md/ scaffolding. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>701d9c5 -
Add Flask + Jinja2 landing page at GET /
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)Single-page UI rendered server-side: title + status pills (task count, submission count, quota, healthz/github links), three-line Quickstart, then per-task sections with description, schema/metric metadata, GT-loaded indicator, and a rank/agent/primary/subs/first-seen leaderboard table. Empty state for tasks with no entries. Inline CSS, no external assets, no JS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>464248e -
Add GT_BYPASS_KEY for unlimited submissions + dry mode
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)X-Bypass-Key header matching GT_BYPASS_KEY env (compared via hmac.compare_digest) unlocks two things: - skip the IP/day quota gate - with form param dry=1, score against GT but skip the leaderboard insert (and skip the submission archive) graphtestbed/submit.py reads GRAPHTESTBED_BYPASS_KEY env and sends X-Bypass-Key on every POST when set; --server-dry CLI flag toggles dry=1. push_to_space.sh now overlays server/space/Dockerfile at root in addition to README (HF Docker SDK looks for Dockerfile at the repo root). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>5ead61d -
Add agents/ harness integrations and HF Space scoring deployment
Zhu Jiajun (jz28583) +1: Claude Opus 4.7 (1M context)- agents/cliproxyapi: reusable shim that points any agent's SDK at one CLIProxyAPI proxy via anthropic_env / openai_env / gemini_env helpers. - agents/{ai_build_ai,mlevolve}: runners that stage GraphTestbed task data, route LLM calls through the proxy, and harvest submission CSVs. Tested end-to-end on figraph; both scored on the leaderboard (aibuildai-claude-sonnet-4-6 0.819, mlevolve-gpt-5.3-codex-spark 0.790). - agents/common: shared workspace + task-instruction + finalize helpers. - server/space/: Docker SDK Space deployment. Boot orchestrator in space_entry.py snapshot_downloads GT + leaderboard.db from the companion private dataset (lanczos/graphtestbed-gt) on startup, then runs a daemon thread that backs up sqlite + new submission CSVs every 60s via huggingface_hub.upload_file/upload_folder. - server/api.py: optional GT_ARCHIVE_DIR env writes raw submission CSVs to disk so the backup loop can ship them to the dataset repo. - graphtestbed/{submit,leaderboard}.py…d094faf
-
Initial commit: GraphTestbed v0.1.0
Kaggle-style scoring server for benchmarking ML/AI agent harnesses on heterogeneous graph datasets. Pre-launch: works end-to-end against a local server; HuggingFace dataset publication and hosted API deploy are pending. - gtb CLI: fetch / submit / leaderboard / score-local - Flask scoring API with sqlite leaderboard - Schema check, 5/day quota, score bucketing to 3dp - 4 tasks declared: ieee-fraud-detection, arxiv-citation, figraph, ibm-aml - Two-branch deploy story (main = client, server = scoring API) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>ad6901d
