Spaces:
Running
Running
Add leaderboard auto-sync implementation plan
Browse files12 TDD tasks: pure transform lib (parse/derive/validate/diff) with unit
tests, sync.mjs orchestrator (check/apply/push), page render + fallback,
SKILL.md, daily CronCreate. Push gated behind explicit --push flag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs/superpowers/plans/2026-06-02-leaderboard-sync.md
ADDED
|
@@ -0,0 +1,1123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Leaderboard Auto-Sync Implementation Plan
|
| 2 |
+
|
| 3 |
+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
| 4 |
+
|
| 5 |
+
**Goal:** Make the three landing-page leaderboards (EOG, EVA-Accuracy, EVA-Experience) render from a synced `data/leaderboard.json`, and add a Claude Code skill that daily pulls the two upstream sources, re-derives that JSON, and pushes to staging only when our displayed data changes and passes sanity checks.
|
| 6 |
+
|
| 7 |
+
**Architecture:** Pure transform functions (no I/O) parse upstream β derive our schema; they're unit-tested with small synthetic fixtures. A `sync.mjs` orchestrator does fetch/diff/git/notify around them. The page keeps its current static markup as a fallback and an inline script overlays the fetched JSON on load. Push to staging is gated behind an explicit `--push` flag so dev runs can never deploy by accident.
|
| 8 |
+
|
| 9 |
+
**Tech Stack:** Node 20 (global `fetch`, `node:test`, `node:assert`, `node:crypto`, `node:child_process`), zero npm dependencies, vanilla inline JS on the page, Tailwind-CDN markup (unchanged classes).
|
| 10 |
+
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
## File Structure
|
| 14 |
+
|
| 15 |
+
**Create:**
|
| 16 |
+
- `.claude/skills/sync-nowai-leaderboard/package.json` β marks ESM + `test` script.
|
| 17 |
+
- `.claude/skills/sync-nowai-leaderboard/lib/transform.mjs` β pure parse/derive/validate/diff functions. No I/O.
|
| 18 |
+
- `.claude/skills/sync-nowai-leaderboard/sync.mjs` β orchestrator: fetch, hash, diff, git, notify, arg parsing.
|
| 19 |
+
- `.claude/skills/sync-nowai-leaderboard/SKILL.md` β when/how to run, guardrails.
|
| 20 |
+
- `.claude/skills/sync-nowai-leaderboard/test/transform.test.mjs` β unit tests for transform.mjs.
|
| 21 |
+
- `data/leaderboard.json` β the rendered data (generated from live upstream in Task 8).
|
| 22 |
+
|
| 23 |
+
**Modify:**
|
| 24 |
+
- `index.html` β add id hooks to the three cards' dynamic regions + an inline render script with fetch + fallback.
|
| 25 |
+
|
| 26 |
+
**Responsibilities:** `transform.mjs` is the only place that knows upstream shapes and our schema (testable in isolation). `sync.mjs` owns side effects (network, fs, git). The page's inline script owns presentation. Split by responsibility so each unit fits in context.
|
| 27 |
+
|
| 28 |
+
---
|
| 29 |
+
|
| 30 |
+
### Task 1: Skill scaffolding
|
| 31 |
+
|
| 32 |
+
**Files:**
|
| 33 |
+
- Create: `.claude/skills/sync-nowai-leaderboard/package.json`
|
| 34 |
+
|
| 35 |
+
- [ ] **Step 1: Create package.json**
|
| 36 |
+
|
| 37 |
+
```json
|
| 38 |
+
{
|
| 39 |
+
"name": "sync-nowai-leaderboard",
|
| 40 |
+
"private": true,
|
| 41 |
+
"type": "module",
|
| 42 |
+
"scripts": {
|
| 43 |
+
"test": "node --test"
|
| 44 |
+
}
|
| 45 |
+
}
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
- [ ] **Step 2: Verify test runner wiring (no tests yet β exits 0 with "no test files")**
|
| 49 |
+
|
| 50 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 51 |
+
Expected: exit 0, message like `tests 0`. (If it errors on no files, that's fine β proceed; Task 2 adds the first test.)
|
| 52 |
+
|
| 53 |
+
- [ ] **Step 3: Commit**
|
| 54 |
+
|
| 55 |
+
```bash
|
| 56 |
+
git add .claude/skills/sync-nowai-leaderboard/package.json
|
| 57 |
+
git commit -m "chore: scaffold sync-nowai-leaderboard skill"
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
---
|
| 61 |
+
|
| 62 |
+
### Task 2: EOG parser (`parseEogData`)
|
| 63 |
+
|
| 64 |
+
**Files:**
|
| 65 |
+
- Create: `.claude/skills/sync-nowai-leaderboard/test/transform.test.mjs`
|
| 66 |
+
- Create: `.claude/skills/sync-nowai-leaderboard/lib/transform.mjs`
|
| 67 |
+
|
| 68 |
+
- [ ] **Step 1: Write the failing test**
|
| 69 |
+
|
| 70 |
+
Create `test/transform.test.mjs`:
|
| 71 |
+
|
| 72 |
+
```js
|
| 73 |
+
import { test } from 'node:test';
|
| 74 |
+
import assert from 'node:assert/strict';
|
| 75 |
+
import { parseEogData } from '../lib/transform.mjs';
|
| 76 |
+
|
| 77 |
+
const EOG_SAMPLE = `
|
| 78 |
+
// ββ Leaderboard Data ββ
|
| 79 |
+
const LEADERBOARD_DATA = [
|
| 80 |
+
// Closed Source
|
| 81 |
+
{ model: "Gemini 3.5 Flash (High)", type: "closed", teams: 48.0, csm: 47.9, email: 55.8, itsm: 42.5, calendar: 42.3, hr: 40.8, drive: 55.2, hybrid: 35.3, avg: 46.0 },
|
| 82 |
+
{ model: "Claude Opus 4.5", type: "closed", teams: 50.0, csm: 34.2, email: 51.9, itsm: 23.8, calendar: 43.2, hr: 32.1, drive: 49.5, hybrid: 30.7, avg: 37.0 },
|
| 83 |
+
];
|
| 84 |
+
function renderLeaderboard() {}
|
| 85 |
+
`;
|
| 86 |
+
|
| 87 |
+
test('parseEogData extracts the array of model objects', () => {
|
| 88 |
+
const arr = parseEogData(EOG_SAMPLE);
|
| 89 |
+
assert.equal(arr.length, 2);
|
| 90 |
+
assert.equal(arr[0].model, 'Gemini 3.5 Flash (High)');
|
| 91 |
+
assert.equal(arr[0].type, 'closed');
|
| 92 |
+
assert.equal(arr[0].avg, 46.0);
|
| 93 |
+
assert.equal(arr[1].model, 'Claude Opus 4.5');
|
| 94 |
+
});
|
| 95 |
+
|
| 96 |
+
test('parseEogData throws when marker missing', () => {
|
| 97 |
+
assert.throws(() => parseEogData('const X = 1;'), /LEADERBOARD_DATA not found/);
|
| 98 |
+
});
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
- [ ] **Step 2: Run test to verify it fails**
|
| 102 |
+
|
| 103 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 104 |
+
Expected: FAIL β `Cannot find module '../lib/transform.mjs'`.
|
| 105 |
+
|
| 106 |
+
- [ ] **Step 3: Write minimal implementation**
|
| 107 |
+
|
| 108 |
+
Create `lib/transform.mjs`:
|
| 109 |
+
|
| 110 |
+
```js
|
| 111 |
+
// Pure transforms: parse upstream sources and derive our leaderboard schema.
|
| 112 |
+
// No network or filesystem access lives here.
|
| 113 |
+
|
| 114 |
+
export function parseEogData(scriptText) {
|
| 115 |
+
const start = scriptText.indexOf('LEADERBOARD_DATA');
|
| 116 |
+
if (start === -1) throw new Error('LEADERBOARD_DATA not found in EOG script');
|
| 117 |
+
const open = scriptText.indexOf('[', start);
|
| 118 |
+
if (open === -1) throw new Error('EOG array open bracket not found');
|
| 119 |
+
|
| 120 |
+
// Match brackets to find the array literal end. Model names contain no [ or ].
|
| 121 |
+
let depth = 0;
|
| 122 |
+
let end = -1;
|
| 123 |
+
for (let i = open; i < scriptText.length; i++) {
|
| 124 |
+
const c = scriptText[i];
|
| 125 |
+
if (c === '[') depth++;
|
| 126 |
+
else if (c === ']') {
|
| 127 |
+
depth--;
|
| 128 |
+
if (depth === 0) { end = i; break; }
|
| 129 |
+
}
|
| 130 |
+
}
|
| 131 |
+
if (end === -1) throw new Error('EOG array close bracket not found');
|
| 132 |
+
|
| 133 |
+
let body = scriptText.slice(open, end + 1);
|
| 134 |
+
body = body.replace(/\/\/[^\n]*/g, ''); // strip line comments
|
| 135 |
+
body = body.replace(/([{,]\s*)([A-Za-z_]\w*)\s*:/g, '$1"$2":'); // quote keys
|
| 136 |
+
body = body.replace(/,(\s*[}\]])/g, '$1'); // strip trailing commas
|
| 137 |
+
|
| 138 |
+
try {
|
| 139 |
+
return JSON.parse(body);
|
| 140 |
+
} catch (e) {
|
| 141 |
+
throw new Error('EOG array parse failed: ' + e.message);
|
| 142 |
+
}
|
| 143 |
+
}
|
| 144 |
+
```
|
| 145 |
+
|
| 146 |
+
- [ ] **Step 4: Run test to verify it passes**
|
| 147 |
+
|
| 148 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 149 |
+
Expected: PASS (2/2).
|
| 150 |
+
|
| 151 |
+
- [ ] **Step 5: Commit**
|
| 152 |
+
|
| 153 |
+
```bash
|
| 154 |
+
git add .claude/skills/sync-nowai-leaderboard/lib/transform.mjs .claude/skills/sync-nowai-leaderboard/test/transform.test.mjs
|
| 155 |
+
git commit -m "feat: parse EOG LEADERBOARD_DATA array from script.js"
|
| 156 |
+
```
|
| 157 |
+
|
| 158 |
+
---
|
| 159 |
+
|
| 160 |
+
### Task 3: EOG derivation (`deriveEog`) + vendor map (`vendorFromName`)
|
| 161 |
+
|
| 162 |
+
**Files:**
|
| 163 |
+
- Modify: `.claude/skills/sync-nowai-leaderboard/test/transform.test.mjs`
|
| 164 |
+
- Modify: `.claude/skills/sync-nowai-leaderboard/lib/transform.mjs`
|
| 165 |
+
|
| 166 |
+
- [ ] **Step 1: Add failing tests**
|
| 167 |
+
|
| 168 |
+
Append to `test/transform.test.mjs`:
|
| 169 |
+
|
| 170 |
+
```js
|
| 171 |
+
import { deriveEog, vendorFromName } from '../lib/transform.mjs';
|
| 172 |
+
|
| 173 |
+
test('vendorFromName maps known model families', () => {
|
| 174 |
+
assert.equal(vendorFromName('Gemini 3.5 Flash'), 'Google');
|
| 175 |
+
assert.equal(vendorFromName('GPT-5.4'), 'OpenAI');
|
| 176 |
+
assert.equal(vendorFromName('Claude Opus 4.5'), 'Anthropic');
|
| 177 |
+
assert.equal(vendorFromName('Mystery-7B'), 'Mystery-7B'); // fallback: first token
|
| 178 |
+
});
|
| 179 |
+
|
| 180 |
+
test('deriveEog sorts by avg desc, takes top 5, computes bars', () => {
|
| 181 |
+
const arr = [
|
| 182 |
+
{ model: 'A', type: 'closed', avg: 20 },
|
| 183 |
+
{ model: 'B', type: 'open', avg: 40 },
|
| 184 |
+
{ model: 'C', type: 'closed', avg: 30 },
|
| 185 |
+
];
|
| 186 |
+
const out = deriveEog(arr);
|
| 187 |
+
assert.equal(out.metricLabel, 'Task Success Rate Β· Oracle mode');
|
| 188 |
+
assert.deepEqual(out.rows.map(r => r.model), ['B', 'C', 'A']);
|
| 189 |
+
assert.equal(out.rows[0].rank, 1);
|
| 190 |
+
assert.equal(out.rows[0].score, 40);
|
| 191 |
+
assert.equal(out.rows[0].bar, 100);
|
| 192 |
+
assert.equal(out.rows[1].bar, 75); // 30/40
|
| 193 |
+
assert.equal(out.rows[2].bar, 50); // 20/40
|
| 194 |
+
assert.equal(out.rows[0].type, 'open');
|
| 195 |
+
});
|
| 196 |
+
```
|
| 197 |
+
|
| 198 |
+
- [ ] **Step 2: Run test to verify it fails**
|
| 199 |
+
|
| 200 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 201 |
+
Expected: FAIL β `deriveEog`/`vendorFromName` not exported.
|
| 202 |
+
|
| 203 |
+
- [ ] **Step 3: Write minimal implementation**
|
| 204 |
+
|
| 205 |
+
Append to `lib/transform.mjs`:
|
| 206 |
+
|
| 207 |
+
```js
|
| 208 |
+
const round1 = (n) => Math.round(n * 10) / 10;
|
| 209 |
+
|
| 210 |
+
const VENDOR_MAP = [
|
| 211 |
+
[/gemini/i, 'Google'],
|
| 212 |
+
[/gpt|openai|\brealtime\b/i, 'OpenAI'],
|
| 213 |
+
[/whisper/i, 'OpenAI'],
|
| 214 |
+
[/claude/i, 'Anthropic'],
|
| 215 |
+
[/nova/i, 'Amazon'],
|
| 216 |
+
[/qwen/i, 'Alibaba'],
|
| 217 |
+
[/deepseek/i, 'DeepSeek'],
|
| 218 |
+
[/llama/i, 'Meta'],
|
| 219 |
+
];
|
| 220 |
+
|
| 221 |
+
export function vendorFromName(name) {
|
| 222 |
+
for (const [re, vendor] of VENDOR_MAP) if (re.test(name)) return vendor;
|
| 223 |
+
return name.split(/[\s+]/)[0];
|
| 224 |
+
}
|
| 225 |
+
|
| 226 |
+
export function deriveEog(eogArray) {
|
| 227 |
+
const sorted = [...eogArray].sort((a, b) => b.avg - a.avg).slice(0, 5);
|
| 228 |
+
const top = sorted[0]?.avg ?? 0;
|
| 229 |
+
return {
|
| 230 |
+
metricLabel: 'Task Success Rate Β· Oracle mode',
|
| 231 |
+
rows: sorted.map((m, i) => ({
|
| 232 |
+
rank: i + 1,
|
| 233 |
+
model: m.model,
|
| 234 |
+
org: vendorFromName(m.model),
|
| 235 |
+
type: m.type,
|
| 236 |
+
score: round1(m.avg),
|
| 237 |
+
bar: top ? Math.round((m.avg / top) * 100) : 0,
|
| 238 |
+
})),
|
| 239 |
+
};
|
| 240 |
+
}
|
| 241 |
+
```
|
| 242 |
+
|
| 243 |
+
- [ ] **Step 4: Run test to verify it passes**
|
| 244 |
+
|
| 245 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 246 |
+
Expected: PASS (all).
|
| 247 |
+
|
| 248 |
+
- [ ] **Step 5: Commit**
|
| 249 |
+
|
| 250 |
+
```bash
|
| 251 |
+
git add .claude/skills/sync-nowai-leaderboard/lib/transform.mjs .claude/skills/sync-nowai-leaderboard/test/transform.test.mjs
|
| 252 |
+
git commit -m "feat: derive EOG top-5 rows with vendor + relative bars"
|
| 253 |
+
```
|
| 254 |
+
|
| 255 |
+
---
|
| 256 |
+
|
| 257 |
+
### Task 4: EVA board derivation (`parseEvaData`, `deriveEvaBoard`)
|
| 258 |
+
|
| 259 |
+
**Files:**
|
| 260 |
+
- Modify: `.claude/skills/sync-nowai-leaderboard/test/transform.test.mjs`
|
| 261 |
+
- Modify: `.claude/skills/sync-nowai-leaderboard/lib/transform.mjs`
|
| 262 |
+
|
| 263 |
+
- [ ] **Step 1: Add failing tests**
|
| 264 |
+
|
| 265 |
+
Append to `test/transform.test.mjs`:
|
| 266 |
+
|
| 267 |
+
```js
|
| 268 |
+
import { parseEvaData, deriveEvaBoard } from '../lib/transform.mjs';
|
| 269 |
+
|
| 270 |
+
const EVA_SAMPLE = JSON.stringify({
|
| 271 |
+
systems: [
|
| 272 |
+
{
|
| 273 |
+
id: 'a', name: 'Gemini Live', type: 's2s', stt: '-', llm: 'gemini', tts: '-',
|
| 274 |
+
clean: {
|
| 275 |
+
'EVA-A_mean': { pooled: { point: 0.30, ci_lower: 0.28, ci_upper: 0.32 } },
|
| 276 |
+
'EVA-X_mean': { pooled: { point: 0.49, ci_lower: 0.47, ci_upper: 0.51 } },
|
| 277 |
+
},
|
| 278 |
+
},
|
| 279 |
+
{
|
| 280 |
+
id: 'b', name: 'Nova + GPT + Sonic', type: 'cascade', stt: 'Nova', llm: 'GPT', tts: 'Sonic',
|
| 281 |
+
clean: {
|
| 282 |
+
'EVA-A_mean': { pooled: { point: 0.41, ci_lower: 0.39, ci_upper: 0.43 } },
|
| 283 |
+
'EVA-X_mean': { pooled: { point: 0.20, ci_lower: 0.18, ci_upper: 0.22 } },
|
| 284 |
+
},
|
| 285 |
+
},
|
| 286 |
+
],
|
| 287 |
+
});
|
| 288 |
+
|
| 289 |
+
test('parseEvaData returns parsed object with systems array', () => {
|
| 290 |
+
const data = parseEvaData(EVA_SAMPLE);
|
| 291 |
+
assert.equal(data.systems.length, 2);
|
| 292 |
+
});
|
| 293 |
+
|
| 294 |
+
test('deriveEvaBoard ranks by accuracy point, builds subtitle + ci + bars', () => {
|
| 295 |
+
const data = parseEvaData(EVA_SAMPLE);
|
| 296 |
+
const out = deriveEvaBoard(data.systems, 'EVA-A_mean');
|
| 297 |
+
assert.equal(out.metricLabel, 'Pass@1');
|
| 298 |
+
assert.deepEqual(out.rows.map(r => r.name), ['Nova + GPT + Sonic', 'Gemini Live']);
|
| 299 |
+
assert.equal(out.rows[0].score, 0.41);
|
| 300 |
+
assert.equal(out.rows[0].subtitle, 'Mixed Models Β· Cascade');
|
| 301 |
+
assert.equal(out.rows[0].bar, 100);
|
| 302 |
+
assert.equal(out.rows[0].ciLower, 0.39);
|
| 303 |
+
assert.equal(out.rows[0].ciUpper, 0.43);
|
| 304 |
+
assert.equal(out.rows[1].subtitle, 'Google Β· Speech-to-Speech');
|
| 305 |
+
assert.equal(out.rows[1].bar, 73); // 0.30/0.41 = 0.7317 β 73
|
| 306 |
+
});
|
| 307 |
+
|
| 308 |
+
test('deriveEvaBoard ranks experience board independently', () => {
|
| 309 |
+
const data = parseEvaData(EVA_SAMPLE);
|
| 310 |
+
const out = deriveEvaBoard(data.systems, 'EVA-X_mean');
|
| 311 |
+
assert.deepEqual(out.rows.map(r => r.name), ['Gemini Live', 'Nova + GPT + Sonic']);
|
| 312 |
+
assert.equal(out.rows[0].score, 0.49);
|
| 313 |
+
});
|
| 314 |
+
```
|
| 315 |
+
|
| 316 |
+
- [ ] **Step 2: Run test to verify it fails**
|
| 317 |
+
|
| 318 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 319 |
+
Expected: FAIL β `parseEvaData`/`deriveEvaBoard` not exported.
|
| 320 |
+
|
| 321 |
+
- [ ] **Step 3: Write minimal implementation**
|
| 322 |
+
|
| 323 |
+
Append to `lib/transform.mjs`:
|
| 324 |
+
|
| 325 |
+
```js
|
| 326 |
+
const round2 = (n) => Math.round(n * 100) / 100;
|
| 327 |
+
|
| 328 |
+
export function parseEvaData(jsonText) {
|
| 329 |
+
const data = JSON.parse(jsonText);
|
| 330 |
+
if (!Array.isArray(data.systems)) throw new Error('EVA data missing systems[] array');
|
| 331 |
+
return data;
|
| 332 |
+
}
|
| 333 |
+
|
| 334 |
+
export function deriveEvaBoard(systems, metricKey) {
|
| 335 |
+
const scored = systems
|
| 336 |
+
.map((s) => ({ s, pooled: s.clean?.[metricKey]?.pooled }))
|
| 337 |
+
.filter((x) => x.pooled && typeof x.pooled.point === 'number')
|
| 338 |
+
.sort((a, b) => b.pooled.point - a.pooled.point)
|
| 339 |
+
.slice(0, 5);
|
| 340 |
+
const top = scored[0]?.pooled.point ?? 0;
|
| 341 |
+
return {
|
| 342 |
+
metricLabel: 'Pass@1',
|
| 343 |
+
rows: scored.map(({ s, pooled }, i) => ({
|
| 344 |
+
rank: i + 1,
|
| 345 |
+
name: s.name,
|
| 346 |
+
subtitle:
|
| 347 |
+
s.type === 'cascade'
|
| 348 |
+
? 'Mixed Models Β· Cascade'
|
| 349 |
+
: `${vendorFromName(s.name)} Β· Speech-to-Speech`,
|
| 350 |
+
score: round2(pooled.point),
|
| 351 |
+
ciLower: round2(pooled.ci_lower),
|
| 352 |
+
ciUpper: round2(pooled.ci_upper),
|
| 353 |
+
bar: top ? Math.round((pooled.point / top) * 100) : 0,
|
| 354 |
+
})),
|
| 355 |
+
};
|
| 356 |
+
}
|
| 357 |
+
```
|
| 358 |
+
|
| 359 |
+
- [ ] **Step 4: Run test to verify it passes**
|
| 360 |
+
|
| 361 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 362 |
+
Expected: PASS (all).
|
| 363 |
+
|
| 364 |
+
- [ ] **Step 5: Commit**
|
| 365 |
+
|
| 366 |
+
```bash
|
| 367 |
+
git add .claude/skills/sync-nowai-leaderboard/lib/transform.mjs .claude/skills/sync-nowai-leaderboard/test/transform.test.mjs
|
| 368 |
+
git commit -m "feat: derive EVA accuracy/experience boards from leaderboardStats.json"
|
| 369 |
+
```
|
| 370 |
+
|
| 371 |
+
---
|
| 372 |
+
|
| 373 |
+
### Task 5: Assemble full object (`buildLeaderboard`) + change diff (`diffBoards`, `hasChanges`)
|
| 374 |
+
|
| 375 |
+
**Files:**
|
| 376 |
+
- Modify: `.claude/skills/sync-nowai-leaderboard/test/transform.test.mjs`
|
| 377 |
+
- Modify: `.claude/skills/sync-nowai-leaderboard/lib/transform.mjs`
|
| 378 |
+
|
| 379 |
+
- [ ] **Step 1: Add failing tests**
|
| 380 |
+
|
| 381 |
+
Append to `test/transform.test.mjs`:
|
| 382 |
+
|
| 383 |
+
```js
|
| 384 |
+
import { buildLeaderboard, hasChanges, diffBoards } from '../lib/transform.mjs';
|
| 385 |
+
|
| 386 |
+
test('buildLeaderboard assembles all three boards plus meta', () => {
|
| 387 |
+
const out = buildLeaderboard({
|
| 388 |
+
eogScript: EOG_SAMPLE,
|
| 389 |
+
evaJson: EVA_SAMPLE,
|
| 390 |
+
sources: { eog: { url: 'u1', hash: 'h1' }, eva: { url: 'u2', hash: 'h2' } },
|
| 391 |
+
now: '2026-06-02T00:00:00Z',
|
| 392 |
+
});
|
| 393 |
+
assert.equal(out.updated, '2026-06-02T00:00:00Z');
|
| 394 |
+
assert.equal(out.sources.eog.hash, 'h1');
|
| 395 |
+
assert.equal(out.eog.rows[0].model, 'Gemini 3.5 Flash (High)');
|
| 396 |
+
assert.equal(out.evaAccuracy.rows[0].name, 'Nova + GPT + Sonic');
|
| 397 |
+
assert.equal(out.evaExperience.rows[0].name, 'Gemini Live');
|
| 398 |
+
});
|
| 399 |
+
|
| 400 |
+
test('hasChanges ignores meta, detects board changes', () => {
|
| 401 |
+
const base = buildLeaderboard({ eogScript: EOG_SAMPLE, evaJson: EVA_SAMPLE, sources: {}, now: 'A' });
|
| 402 |
+
const sameDataNewMeta = buildLeaderboard({ eogScript: EOG_SAMPLE, evaJson: EVA_SAMPLE, sources: { eog: { hash: 'x' } }, now: 'B' });
|
| 403 |
+
assert.equal(hasChanges(base, sameDataNewMeta), false);
|
| 404 |
+
|
| 405 |
+
const changedEog = EOG_SAMPLE.replace('avg: 46.0', 'avg: 99.0');
|
| 406 |
+
const changed = buildLeaderboard({ eogScript: changedEog, evaJson: EVA_SAMPLE, sources: {}, now: 'A' });
|
| 407 |
+
assert.equal(hasChanges(base, changed), true);
|
| 408 |
+
});
|
| 409 |
+
|
| 410 |
+
test('diffBoards produces human-readable change lines', () => {
|
| 411 |
+
const base = buildLeaderboard({ eogScript: EOG_SAMPLE, evaJson: EVA_SAMPLE, sources: {}, now: 'A' });
|
| 412 |
+
const changedEog = EOG_SAMPLE.replace('avg: 46.0', 'avg: 41.0');
|
| 413 |
+
const changed = buildLeaderboard({ eogScript: changedEog, evaJson: EVA_SAMPLE, sources: {}, now: 'A' });
|
| 414 |
+
const lines = diffBoards(base, changed);
|
| 415 |
+
assert.ok(lines.some((l) => l.startsWith('eog:')));
|
| 416 |
+
});
|
| 417 |
+
```
|
| 418 |
+
|
| 419 |
+
- [ ] **Step 2: Run test to verify it fails**
|
| 420 |
+
|
| 421 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 422 |
+
Expected: FAIL β `buildLeaderboard`/`hasChanges`/`diffBoards` not exported.
|
| 423 |
+
|
| 424 |
+
- [ ] **Step 3: Write minimal implementation**
|
| 425 |
+
|
| 426 |
+
Append to `lib/transform.mjs`:
|
| 427 |
+
|
| 428 |
+
```js
|
| 429 |
+
export function buildLeaderboard({ eogScript, evaJson, sources, now }) {
|
| 430 |
+
const eogArray = parseEogData(eogScript);
|
| 431 |
+
const evaData = parseEvaData(evaJson);
|
| 432 |
+
return {
|
| 433 |
+
updated: now,
|
| 434 |
+
sources,
|
| 435 |
+
eog: deriveEog(eogArray),
|
| 436 |
+
evaAccuracy: deriveEvaBoard(evaData.systems, 'EVA-A_mean'),
|
| 437 |
+
evaExperience: deriveEvaBoard(evaData.systems, 'EVA-X_mean'),
|
| 438 |
+
};
|
| 439 |
+
}
|
| 440 |
+
|
| 441 |
+
const BOARD_KEYS = ['eog', 'evaAccuracy', 'evaExperience'];
|
| 442 |
+
|
| 443 |
+
// Compare only the parts we render (ignore `updated`/`sources` meta).
|
| 444 |
+
export function hasChanges(oldData, newData) {
|
| 445 |
+
if (!oldData) return true;
|
| 446 |
+
const pick = (d) => JSON.stringify(BOARD_KEYS.map((k) => d[k]));
|
| 447 |
+
return pick(oldData) !== pick(newData);
|
| 448 |
+
}
|
| 449 |
+
|
| 450 |
+
export function diffBoards(oldData, newData) {
|
| 451 |
+
const label = (r) => r.model ?? r.name;
|
| 452 |
+
const lines = [];
|
| 453 |
+
for (const key of BOARD_KEYS) {
|
| 454 |
+
const oldRows = oldData?.[key]?.rows ?? [];
|
| 455 |
+
const newRows = newData[key].rows;
|
| 456 |
+
const oldStr = oldRows.map((r) => `${label(r)}=${r.score}`).join(', ');
|
| 457 |
+
const newStr = newRows.map((r) => `${label(r)}=${r.score}`).join(', ');
|
| 458 |
+
if (oldStr !== newStr) {
|
| 459 |
+
lines.push(`${key}:\n old: ${oldStr || '(none)'}\n new: ${newStr}`);
|
| 460 |
+
}
|
| 461 |
+
}
|
| 462 |
+
return lines;
|
| 463 |
+
}
|
| 464 |
+
```
|
| 465 |
+
|
| 466 |
+
- [ ] **Step 4: Run test to verify it passes**
|
| 467 |
+
|
| 468 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 469 |
+
Expected: PASS (all).
|
| 470 |
+
|
| 471 |
+
- [ ] **Step 5: Commit**
|
| 472 |
+
|
| 473 |
+
```bash
|
| 474 |
+
git add .claude/skills/sync-nowai-leaderboard/lib/transform.mjs .claude/skills/sync-nowai-leaderboard/test/transform.test.mjs
|
| 475 |
+
git commit -m "feat: assemble full leaderboard object + change detection/diff"
|
| 476 |
+
```
|
| 477 |
+
|
| 478 |
+
---
|
| 479 |
+
|
| 480 |
+
### Task 6: Sanity gates (`validate`)
|
| 481 |
+
|
| 482 |
+
**Files:**
|
| 483 |
+
- Modify: `.claude/skills/sync-nowai-leaderboard/test/transform.test.mjs`
|
| 484 |
+
- Modify: `.claude/skills/sync-nowai-leaderboard/lib/transform.mjs`
|
| 485 |
+
|
| 486 |
+
- [ ] **Step 1: Add failing tests**
|
| 487 |
+
|
| 488 |
+
Append to `test/transform.test.mjs`:
|
| 489 |
+
|
| 490 |
+
```js
|
| 491 |
+
import { validate } from '../lib/transform.mjs';
|
| 492 |
+
|
| 493 |
+
function boardWith(rows) { return { metricLabel: 'x', rows }; }
|
| 494 |
+
function dataWith(eogRows, accRows, expRows) {
|
| 495 |
+
return { eog: boardWith(eogRows), evaAccuracy: boardWith(accRows), evaExperience: boardWith(expRows) };
|
| 496 |
+
}
|
| 497 |
+
|
| 498 |
+
test('validate passes clean data', () => {
|
| 499 |
+
const good = dataWith(
|
| 500 |
+
[{ rank: 1, model: 'A', score: 46 }, { rank: 2, model: 'B', score: 40 }, { rank: 3, model: 'C', score: 30 }],
|
| 501 |
+
[{ rank: 1, name: 'X', score: 0.4 }, { rank: 2, name: 'Y', score: 0.3 }, { rank: 3, name: 'Z', score: 0.2 }],
|
| 502 |
+
[{ rank: 1, name: 'X', score: 0.5 }, { rank: 2, name: 'Y', score: 0.4 }, { rank: 3, name: 'Z', score: 0.3 }],
|
| 503 |
+
);
|
| 504 |
+
assert.deepEqual(validate(good, null), []);
|
| 505 |
+
});
|
| 506 |
+
|
| 507 |
+
test('validate flags too-few rows', () => {
|
| 508 |
+
const bad = dataWith([{ rank: 1, model: 'A', score: 46 }], [], []);
|
| 509 |
+
const issues = validate(bad, null);
|
| 510 |
+
assert.ok(issues.some((i) => i.includes('eog') && i.includes('<3')));
|
| 511 |
+
});
|
| 512 |
+
|
| 513 |
+
test('validate flags out-of-range scores', () => {
|
| 514 |
+
const bad = dataWith(
|
| 515 |
+
[{ rank: 1, model: 'A', score: 150 }, { rank: 2, model: 'B', score: 40 }, { rank: 3, model: 'C', score: 30 }],
|
| 516 |
+
[{ rank: 1, name: 'X', score: 0.4 }, { rank: 2, name: 'Y', score: 0.3 }, { rank: 3, name: 'Z', score: 0.2 }],
|
| 517 |
+
[{ rank: 1, name: 'X', score: 0.5 }, { rank: 2, name: 'Y', score: 0.4 }, { rank: 3, name: 'Z', score: 0.3 }],
|
| 518 |
+
);
|
| 519 |
+
assert.ok(validate(bad, null).some((i) => i.includes('out of')));
|
| 520 |
+
});
|
| 521 |
+
|
| 522 |
+
test('validate flags suspicious top-score swing vs previous', () => {
|
| 523 |
+
const prev = dataWith(
|
| 524 |
+
[{ rank: 1, model: 'A', score: 46 }, { rank: 2, model: 'B', score: 40 }, { rank: 3, model: 'C', score: 30 }],
|
| 525 |
+
[{ rank: 1, name: 'X', score: 0.4 }, { rank: 2, name: 'Y', score: 0.3 }, { rank: 3, name: 'Z', score: 0.2 }],
|
| 526 |
+
[{ rank: 1, name: 'X', score: 0.5 }, { rank: 2, name: 'Y', score: 0.4 }, { rank: 3, name: 'Z', score: 0.3 }],
|
| 527 |
+
);
|
| 528 |
+
const now = dataWith(
|
| 529 |
+
[{ rank: 1, model: 'A', score: 10 }, { rank: 2, model: 'B', score: 8 }, { rank: 3, model: 'C', score: 6 }],
|
| 530 |
+
[{ rank: 1, name: 'X', score: 0.4 }, { rank: 2, name: 'Y', score: 0.3 }, { rank: 3, name: 'Z', score: 0.2 }],
|
| 531 |
+
[{ rank: 1, name: 'X', score: 0.5 }, { rank: 2, name: 'Y', score: 0.4 }, { rank: 3, name: 'Z', score: 0.3 }],
|
| 532 |
+
);
|
| 533 |
+
assert.ok(validate(now, prev).some((i) => i.includes('swing')));
|
| 534 |
+
});
|
| 535 |
+
```
|
| 536 |
+
|
| 537 |
+
- [ ] **Step 2: Run test to verify it fails**
|
| 538 |
+
|
| 539 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 540 |
+
Expected: FAIL β `validate` not exported.
|
| 541 |
+
|
| 542 |
+
- [ ] **Step 3: Write minimal implementation**
|
| 543 |
+
|
| 544 |
+
Append to `lib/transform.mjs`:
|
| 545 |
+
|
| 546 |
+
```js
|
| 547 |
+
// Returns an array of human-readable issue strings; empty array = clean.
|
| 548 |
+
export function validate(newData, oldData) {
|
| 549 |
+
const issues = [];
|
| 550 |
+
const specs = [
|
| 551 |
+
['eog', newData.eog, 0, 100, 25, oldData?.eog],
|
| 552 |
+
['evaAccuracy', newData.evaAccuracy, 0, 1, 0.25, oldData?.evaAccuracy],
|
| 553 |
+
['evaExperience', newData.evaExperience, 0, 1, 0.25, oldData?.evaExperience],
|
| 554 |
+
];
|
| 555 |
+
for (const [name, board, min, max, guard, oldBoard] of specs) {
|
| 556 |
+
if (!board || !Array.isArray(board.rows)) {
|
| 557 |
+
issues.push(`${name}: missing rows`);
|
| 558 |
+
continue;
|
| 559 |
+
}
|
| 560 |
+
if (board.rows.length < 3) issues.push(`${name}: only ${board.rows.length} rows (<3)`);
|
| 561 |
+
for (const r of board.rows) {
|
| 562 |
+
if (r.score < min || r.score > max) {
|
| 563 |
+
issues.push(`${name}: score ${r.score} out of [${min},${max}]`);
|
| 564 |
+
}
|
| 565 |
+
}
|
| 566 |
+
const oldTop = oldBoard?.rows?.[0]?.score;
|
| 567 |
+
const newTop = board.rows[0]?.score;
|
| 568 |
+
if (typeof oldTop === 'number' && typeof newTop === 'number') {
|
| 569 |
+
const delta = Math.abs(newTop - oldTop);
|
| 570 |
+
if (delta > guard) issues.push(`${name}: top score swing ${delta.toFixed(2)} > ${guard}`);
|
| 571 |
+
}
|
| 572 |
+
}
|
| 573 |
+
return issues;
|
| 574 |
+
}
|
| 575 |
+
```
|
| 576 |
+
|
| 577 |
+
- [ ] **Step 4: Run test to verify it passes**
|
| 578 |
+
|
| 579 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 580 |
+
Expected: PASS (all).
|
| 581 |
+
|
| 582 |
+
- [ ] **Step 5: Commit**
|
| 583 |
+
|
| 584 |
+
```bash
|
| 585 |
+
git add .claude/skills/sync-nowai-leaderboard/lib/transform.mjs .claude/skills/sync-nowai-leaderboard/test/transform.test.mjs
|
| 586 |
+
git commit -m "feat: add sanity-gate validation for derived leaderboard"
|
| 587 |
+
```
|
| 588 |
+
|
| 589 |
+
---
|
| 590 |
+
|
| 591 |
+
### Task 7: Orchestrator `sync.mjs` (fetch, hash, git, notify, arg parsing)
|
| 592 |
+
|
| 593 |
+
**Files:**
|
| 594 |
+
- Create: `.claude/skills/sync-nowai-leaderboard/sync.mjs`
|
| 595 |
+
|
| 596 |
+
Modes: `--check` (diff only, no writes), `--apply` (write `data/leaderboard.json` + git commit, NO push), `--push` (implies apply + push to `origin`/staging). Default is `--check`. Push is opt-in so dev runs never deploy.
|
| 597 |
+
|
| 598 |
+
- [ ] **Step 1: Create sync.mjs**
|
| 599 |
+
|
| 600 |
+
```js
|
| 601 |
+
#!/usr/bin/env node
|
| 602 |
+
// Orchestrator for the leaderboard sync. Side effects only; pure logic in lib/transform.mjs.
|
| 603 |
+
import { readFile, writeFile } from 'node:fs/promises';
|
| 604 |
+
import { existsSync } from 'node:fs';
|
| 605 |
+
import { createHash } from 'node:crypto';
|
| 606 |
+
import { execFileSync } from 'node:child_process';
|
| 607 |
+
import { join } from 'node:path';
|
| 608 |
+
import { buildLeaderboard, hasChanges, diffBoards, validate } from './lib/transform.mjs';
|
| 609 |
+
|
| 610 |
+
const EOG_URL = 'https://raw.githubusercontent.com/EnterpriseOps-Gym/EnterpriseOps-Gym.github.io/main/script.js';
|
| 611 |
+
const EVA_URL = 'https://raw.githubusercontent.com/ServiceNow/eva/main/website/src/data/leaderboardStats.json';
|
| 612 |
+
|
| 613 |
+
function repoRoot() {
|
| 614 |
+
return execFileSync('git', ['rev-parse', '--show-toplevel'], { encoding: 'utf8' }).trim();
|
| 615 |
+
}
|
| 616 |
+
|
| 617 |
+
function sha256(text) {
|
| 618 |
+
return 'sha256:' + createHash('sha256').update(text).digest('hex');
|
| 619 |
+
}
|
| 620 |
+
|
| 621 |
+
async function fetchText(url) {
|
| 622 |
+
const res = await fetch(url);
|
| 623 |
+
if (!res.ok) throw new Error(`fetch ${url} β HTTP ${res.status}`);
|
| 624 |
+
return res.text();
|
| 625 |
+
}
|
| 626 |
+
|
| 627 |
+
async function notify(message) {
|
| 628 |
+
const url = process.env.TEAMS_WEBHOOK_URL;
|
| 629 |
+
if (!url) {
|
| 630 |
+
console.log('[notify] (no TEAMS_WEBHOOK_URL set) ' + message);
|
| 631 |
+
return;
|
| 632 |
+
}
|
| 633 |
+
try {
|
| 634 |
+
const res = await fetch(url, {
|
| 635 |
+
method: 'POST',
|
| 636 |
+
headers: { 'Content-Type': 'application/json' },
|
| 637 |
+
body: JSON.stringify({ text: message }),
|
| 638 |
+
});
|
| 639 |
+
if (!res.ok) console.error(`[notify] webhook HTTP ${res.status}`);
|
| 640 |
+
} catch (e) {
|
| 641 |
+
console.error('[notify] webhook failed: ' + e.message);
|
| 642 |
+
}
|
| 643 |
+
}
|
| 644 |
+
|
| 645 |
+
function git(root, args) {
|
| 646 |
+
return execFileSync('git', args, { cwd: root, encoding: 'utf8' });
|
| 647 |
+
}
|
| 648 |
+
|
| 649 |
+
async function main() {
|
| 650 |
+
const args = new Set(process.argv.slice(2));
|
| 651 |
+
const mode = args.has('--push') ? 'push' : args.has('--apply') ? 'apply' : 'check';
|
| 652 |
+
|
| 653 |
+
const root = repoRoot();
|
| 654 |
+
const dataPath = join(root, 'data', 'leaderboard.json');
|
| 655 |
+
|
| 656 |
+
// 1. Fetch upstream.
|
| 657 |
+
let eogScript, evaJson;
|
| 658 |
+
try {
|
| 659 |
+
[eogScript, evaJson] = await Promise.all([fetchText(EOG_URL), fetchText(EVA_URL)]);
|
| 660 |
+
} catch (e) {
|
| 661 |
+
console.error('[sync] fetch failed: ' + e.message);
|
| 662 |
+
await notify('Leaderboard sync: upstream fetch failed β ' + e.message);
|
| 663 |
+
process.exit(1);
|
| 664 |
+
}
|
| 665 |
+
|
| 666 |
+
// 2. Build new data.
|
| 667 |
+
let newData;
|
| 668 |
+
try {
|
| 669 |
+
newData = buildLeaderboard({
|
| 670 |
+
eogScript,
|
| 671 |
+
evaJson,
|
| 672 |
+
sources: {
|
| 673 |
+
eog: { url: EOG_URL, hash: sha256(eogScript), fetchedAt: new Date().toISOString() },
|
| 674 |
+
eva: { url: EVA_URL, hash: sha256(evaJson), fetchedAt: new Date().toISOString() },
|
| 675 |
+
},
|
| 676 |
+
now: new Date().toISOString(),
|
| 677 |
+
});
|
| 678 |
+
} catch (e) {
|
| 679 |
+
console.error('[sync] parse/derive failed: ' + e.message);
|
| 680 |
+
await notify('Leaderboard sync: parse failed (schema drift?) β ' + e.message);
|
| 681 |
+
process.exit(1);
|
| 682 |
+
}
|
| 683 |
+
|
| 684 |
+
// 3. Load existing.
|
| 685 |
+
const oldData = existsSync(dataPath)
|
| 686 |
+
? JSON.parse(await readFile(dataPath, 'utf8'))
|
| 687 |
+
: null;
|
| 688 |
+
|
| 689 |
+
// 4. Change detection.
|
| 690 |
+
if (!hasChanges(oldData, newData)) {
|
| 691 |
+
console.log('[sync] no display changes. Done.');
|
| 692 |
+
return;
|
| 693 |
+
}
|
| 694 |
+
const changeLines = diffBoards(oldData, newData);
|
| 695 |
+
console.log('[sync] changes detected:\n' + changeLines.join('\n'));
|
| 696 |
+
|
| 697 |
+
// 5. Sanity gates.
|
| 698 |
+
const issues = validate(newData, oldData);
|
| 699 |
+
if (issues.length) {
|
| 700 |
+
console.error('[sync] sanity gates failed:\n - ' + issues.join('\n - '));
|
| 701 |
+
await notify('Leaderboard sync ABORTED β sanity gates failed:\n' + issues.join('\n'));
|
| 702 |
+
process.exit(2);
|
| 703 |
+
}
|
| 704 |
+
|
| 705 |
+
if (mode === 'check') {
|
| 706 |
+
console.log('[sync] --check: no writes. Re-run with --apply (commit) or --push (commit + staging).');
|
| 707 |
+
return;
|
| 708 |
+
}
|
| 709 |
+
|
| 710 |
+
// 6. Write + commit.
|
| 711 |
+
await writeFile(dataPath, JSON.stringify(newData, null, 2) + '\n');
|
| 712 |
+
git(root, ['add', dataPath]);
|
| 713 |
+
const today = new Date().toISOString().slice(0, 10);
|
| 714 |
+
const msg = `Sync leaderboard: EOG + EVA ${today}\n\n${changeLines.join('\n')}`;
|
| 715 |
+
git(root, ['commit', '-m', msg]);
|
| 716 |
+
console.log('[sync] committed.');
|
| 717 |
+
|
| 718 |
+
// 7. Push (staging only) if requested.
|
| 719 |
+
if (mode === 'push') {
|
| 720 |
+
try {
|
| 721 |
+
git(root, ['push', 'origin', 'HEAD:main']);
|
| 722 |
+
console.log('[sync] pushed to origin (staging).');
|
| 723 |
+
await notify(`Leaderboard synced to staging (${changeLines.length} board change(s)). Review + promote to prod.`);
|
| 724 |
+
} catch (e) {
|
| 725 |
+
console.error('[sync] push failed: ' + e.message);
|
| 726 |
+
await notify('Leaderboard sync: commit OK but push failed β ' + e.message);
|
| 727 |
+
process.exit(3);
|
| 728 |
+
}
|
| 729 |
+
} else {
|
| 730 |
+
console.log('[sync] --apply: committed locally, not pushed. Use --push to deploy to staging.');
|
| 731 |
+
}
|
| 732 |
+
}
|
| 733 |
+
|
| 734 |
+
main().catch((e) => {
|
| 735 |
+
console.error('[sync] unexpected error: ' + e.message);
|
| 736 |
+
process.exit(1);
|
| 737 |
+
});
|
| 738 |
+
```
|
| 739 |
+
|
| 740 |
+
- [ ] **Step 2: Run check mode against live upstream (smoke test, no writes)**
|
| 741 |
+
|
| 742 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node sync.mjs --check`
|
| 743 |
+
Expected: prints `[sync] changes detected:` with EOG/EVA lines (since no `data/leaderboard.json` exists yet), then `--check: no writes`. Exit 0. No files changed (`git status` clean except untracked plan/docs).
|
| 744 |
+
|
| 745 |
+
- [ ] **Step 3: Commit**
|
| 746 |
+
|
| 747 |
+
```bash
|
| 748 |
+
git add .claude/skills/sync-nowai-leaderboard/sync.mjs
|
| 749 |
+
git commit -m "feat: sync orchestrator with check/apply/push modes + notify"
|
| 750 |
+
```
|
| 751 |
+
|
| 752 |
+
---
|
| 753 |
+
|
| 754 |
+
### Task 8: Generate initial `data/leaderboard.json` from live upstream
|
| 755 |
+
|
| 756 |
+
**Files:**
|
| 757 |
+
- Create: `data/leaderboard.json` (generated)
|
| 758 |
+
|
| 759 |
+
- [ ] **Step 1: Generate + commit locally (NO push)**
|
| 760 |
+
|
| 761 |
+
Run: `cd .claude/skills/sync-nowai-leaderboard && node sync.mjs --apply`
|
| 762 |
+
Expected: `changes detected` β sanity gates pass β `committed.` β `--apply: committed locally, not pushed.` Exit 0.
|
| 763 |
+
|
| 764 |
+
- [ ] **Step 2: Inspect the generated file**
|
| 765 |
+
|
| 766 |
+
Run: `cat data/leaderboard.json` (from repo root)
|
| 767 |
+
Expected: valid JSON with `eog.rows` (5 entries, top = Gemini 3.5 Flash ~46), `evaAccuracy.rows`, `evaExperience.rows` (5 each), `sources` with sha256 hashes, `updated` timestamp.
|
| 768 |
+
|
| 769 |
+
Verify ranges: EOG scores 0β100, EVA scores 0β1. If anything looks malformed, STOP β the parser needs fixing before proceeding (revisit Task 2/4).
|
| 770 |
+
|
| 771 |
+
- [ ] **Step 3: (commit already made by --apply in Step 1 β verify)**
|
| 772 |
+
|
| 773 |
+
Run: `git log --oneline -1`
|
| 774 |
+
Expected: top commit is the `Sync leaderboard:` commit created by the script.
|
| 775 |
+
|
| 776 |
+
---
|
| 777 |
+
|
| 778 |
+
### Task 9: Page render β id hooks + inline fetch/fallback script
|
| 779 |
+
|
| 780 |
+
**Files:**
|
| 781 |
+
- Modify: `index.html`
|
| 782 |
+
|
| 783 |
+
The three cards keep their current static rows as the never-empty fallback. We add `id` hooks to the dynamic regions and an inline script that overlays the fetched JSON on load.
|
| 784 |
+
|
| 785 |
+
- [ ] **Step 1: Add id hooks to the EOG card**
|
| 786 |
+
|
| 787 |
+
In `index.html`, the EOG top-model block currently reads:
|
| 788 |
+
|
| 789 |
+
```html
|
| 790 |
+
<div class="text-infinite-blue font-bold text-xl leading-tight">Claude Opus 4.5</div>
|
| 791 |
+
<div class="text-on-surface-variant text-sm font-medium">Anthropic</div>
|
| 792 |
+
```
|
| 793 |
+
|
| 794 |
+
Replace with:
|
| 795 |
+
|
| 796 |
+
```html
|
| 797 |
+
<div id="eog-top-name" class="text-infinite-blue font-bold text-xl leading-tight">Claude Opus 4.5</div>
|
| 798 |
+
<div id="eog-top-org" class="text-on-surface-variant text-sm font-medium">Anthropic</div>
|
| 799 |
+
```
|
| 800 |
+
|
| 801 |
+
The EOG score span currently reads:
|
| 802 |
+
|
| 803 |
+
```html
|
| 804 |
+
<span class="text-infinite-blue font-bold text-3xl tabular-nums leading-none">37.4<span class="text-on-surface-variant text-lg font-medium ml-0.5">%</span></span>
|
| 805 |
+
```
|
| 806 |
+
|
| 807 |
+
Replace with (add id on the outer span):
|
| 808 |
+
|
| 809 |
+
```html
|
| 810 |
+
<span id="eog-top-score" class="text-infinite-blue font-bold text-3xl tabular-nums leading-none">37.4<span class="text-on-surface-variant text-lg font-medium ml-0.5">%</span></span>
|
| 811 |
+
```
|
| 812 |
+
|
| 813 |
+
The EOG runners wrapper currently reads `<div class="space-y-4">` (the one immediately containing `<!-- Runner 2 -->`). Replace its opening tag with:
|
| 814 |
+
|
| 815 |
+
```html
|
| 816 |
+
<div id="eog-runners" class="space-y-4">
|
| 817 |
+
```
|
| 818 |
+
|
| 819 |
+
- [ ] **Step 2: Add id hooks to the EVA-Accuracy split**
|
| 820 |
+
|
| 821 |
+
The accuracy top block currently reads:
|
| 822 |
+
|
| 823 |
+
```html
|
| 824 |
+
<div class="text-infinite-blue font-bold text-base leading-tight">Nova + GPT-5.4 + Sonic</div>
|
| 825 |
+
<div class="text-on-surface-variant text-[11px] font-medium">Mixed Models Β· Cascade</div>
|
| 826 |
+
```
|
| 827 |
+
|
| 828 |
+
Replace with:
|
| 829 |
+
|
| 830 |
+
```html
|
| 831 |
+
<div id="eva-acc-name" class="text-infinite-blue font-bold text-base leading-tight">Nova + GPT-5.4 + Sonic</div>
|
| 832 |
+
<div id="eva-acc-sub" class="text-on-surface-variant text-[11px] font-medium">Mixed Models Β· Cascade</div>
|
| 833 |
+
```
|
| 834 |
+
|
| 835 |
+
The accuracy top score currently reads:
|
| 836 |
+
|
| 837 |
+
```html
|
| 838 |
+
<span class="text-infinite-blue font-bold text-2xl tabular-nums">0.41</span>
|
| 839 |
+
```
|
| 840 |
+
|
| 841 |
+
Replace with:
|
| 842 |
+
|
| 843 |
+
```html
|
| 844 |
+
<span id="eva-acc-score" class="text-infinite-blue font-bold text-2xl tabular-nums">0.41</span>
|
| 845 |
+
```
|
| 846 |
+
|
| 847 |
+
The accuracy runners wrapper currently reads `<div class="space-y-2.5">` (the one whose first child row contains `Claude Opus 4.5` at `0.32`). Replace its opening tag with:
|
| 848 |
+
|
| 849 |
+
```html
|
| 850 |
+
<div id="eva-acc-runners" class="space-y-2.5">
|
| 851 |
+
```
|
| 852 |
+
|
| 853 |
+
- [ ] **Step 3: Add id hooks to the EVA-Experience split**
|
| 854 |
+
|
| 855 |
+
The experience top block currently reads:
|
| 856 |
+
|
| 857 |
+
```html
|
| 858 |
+
<div class="text-infinite-blue font-bold text-base leading-tight">Gemini Live</div>
|
| 859 |
+
<div class="text-on-surface-variant text-[11px] font-medium">Google Β· Speech-to-Speech</div>
|
| 860 |
+
```
|
| 861 |
+
|
| 862 |
+
Replace with:
|
| 863 |
+
|
| 864 |
+
```html
|
| 865 |
+
<div id="eva-exp-name" class="text-infinite-blue font-bold text-base leading-tight">Gemini Live</div>
|
| 866 |
+
<div id="eva-exp-sub" class="text-on-surface-variant text-[11px] font-medium">Google Β· Speech-to-Speech</div>
|
| 867 |
+
```
|
| 868 |
+
|
| 869 |
+
The experience top score currently reads:
|
| 870 |
+
|
| 871 |
+
```html
|
| 872 |
+
<span class="text-infinite-blue font-bold text-2xl tabular-nums">0.49</span>
|
| 873 |
+
```
|
| 874 |
+
|
| 875 |
+
Replace with:
|
| 876 |
+
|
| 877 |
+
```html
|
| 878 |
+
<span id="eva-exp-score" class="text-infinite-blue font-bold text-2xl tabular-nums">0.49</span>
|
| 879 |
+
```
|
| 880 |
+
|
| 881 |
+
The experience runners wrapper currently reads `<div class="space-y-2.5">` (the one whose first child row contains `GPT-Realtime` at `0.47`). Replace its opening tag with:
|
| 882 |
+
|
| 883 |
+
```html
|
| 884 |
+
<div id="eva-exp-runners" class="space-y-2.5">
|
| 885 |
+
```
|
| 886 |
+
|
| 887 |
+
- [ ] **Step 4: Add the inline render script before `</body>`**
|
| 888 |
+
|
| 889 |
+
Find the closing `</body>` tag in `index.html` and insert this block immediately before it:
|
| 890 |
+
|
| 891 |
+
```html
|
| 892 |
+
<script>
|
| 893 |
+
// Overlay synced leaderboard data on the static fallback markup.
|
| 894 |
+
// On any failure, the static markup already in the DOM remains.
|
| 895 |
+
(function () {
|
| 896 |
+
function esc(s) {
|
| 897 |
+
return String(s).replace(/[&<>"']/g, (c) => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c]));
|
| 898 |
+
}
|
| 899 |
+
|
| 900 |
+
function eogRunnerRow(r) {
|
| 901 |
+
return `<div class="grid grid-cols-[20px_1fr_100px_45px] items-center gap-3">
|
| 902 |
+
<span class="text-on-surface-variant text-xs tabular-nums font-medium">${r.rank}</span>
|
| 903 |
+
<span class="text-on-surface text-sm font-medium">${esc(r.model)}</span>
|
| 904 |
+
<div class="h-1.5 w-full bg-infinite-soft rounded-full overflow-hidden">
|
| 905 |
+
<div class="h-full fill-infinite rounded-full" style="width: ${r.bar}%;"></div>
|
| 906 |
+
</div>
|
| 907 |
+
<span class="text-infinite-blue text-sm font-bold text-right tabular-nums">${r.score.toFixed(1)}</span>
|
| 908 |
+
</div>`;
|
| 909 |
+
}
|
| 910 |
+
|
| 911 |
+
function evaRunnerRow(r, fill) {
|
| 912 |
+
return `<div class="grid grid-cols-[18px_1fr_80px_35px] items-center gap-2 text-xs">
|
| 913 |
+
<span class="text-on-surface-variant tabular-nums font-medium">${r.rank}</span>
|
| 914 |
+
<span class="text-on-surface font-medium truncate">${esc(r.name)}</span>
|
| 915 |
+
<div class="h-1 w-full bg-${fill}-soft rounded-full overflow-hidden">
|
| 916 |
+
<div class="h-full fill-${fill} rounded-full" style="width: ${r.bar}%;"></div>
|
| 917 |
+
</div>
|
| 918 |
+
<span class="text-infinite-blue font-bold text-right tabular-nums">${r.score.toFixed(2)}</span>
|
| 919 |
+
</div>`;
|
| 920 |
+
}
|
| 921 |
+
|
| 922 |
+
function setText(id, value) {
|
| 923 |
+
const el = document.getElementById(id);
|
| 924 |
+
if (el) el.textContent = value;
|
| 925 |
+
}
|
| 926 |
+
|
| 927 |
+
function renderEog(board) {
|
| 928 |
+
const top = board.rows[0];
|
| 929 |
+
if (!top) return;
|
| 930 |
+
setText('eog-top-name', top.model);
|
| 931 |
+
setText('eog-top-org', top.org);
|
| 932 |
+
const score = document.getElementById('eog-top-score');
|
| 933 |
+
if (score) score.innerHTML = `${top.score.toFixed(1)}<span class="text-on-surface-variant text-lg font-medium ml-0.5">%</span>`;
|
| 934 |
+
const runners = document.getElementById('eog-runners');
|
| 935 |
+
if (runners) runners.innerHTML = board.rows.slice(1).map(eogRunnerRow).join('');
|
| 936 |
+
}
|
| 937 |
+
|
| 938 |
+
function renderEvaSplit(board, prefix, fill) {
|
| 939 |
+
const top = board.rows[0];
|
| 940 |
+
if (!top) return;
|
| 941 |
+
setText(`${prefix}-name`, top.name);
|
| 942 |
+
setText(`${prefix}-sub`, top.subtitle);
|
| 943 |
+
setText(`${prefix}-score`, top.score.toFixed(2));
|
| 944 |
+
const runners = document.getElementById(`${prefix}-runners`);
|
| 945 |
+
if (runners) runners.innerHTML = board.rows.slice(1).map((r) => evaRunnerRow(r, fill)).join('');
|
| 946 |
+
}
|
| 947 |
+
|
| 948 |
+
fetch('./data/leaderboard.json', { cache: 'no-cache' })
|
| 949 |
+
.then((res) => { if (!res.ok) throw new Error('HTTP ' + res.status); return res.json(); })
|
| 950 |
+
.then((data) => {
|
| 951 |
+
if (data.eog) renderEog(data.eog);
|
| 952 |
+
if (data.evaAccuracy) renderEvaSplit(data.evaAccuracy, 'eva-acc', 'indigo');
|
| 953 |
+
if (data.evaExperience) renderEvaSplit(data.evaExperience, 'eva-exp', 'blue');
|
| 954 |
+
})
|
| 955 |
+
.catch((err) => {
|
| 956 |
+
console.warn('Leaderboard JSON load failed; showing static fallback.', err);
|
| 957 |
+
});
|
| 958 |
+
})();
|
| 959 |
+
</script>
|
| 960 |
+
```
|
| 961 |
+
|
| 962 |
+
- [ ] **Step 5: Commit**
|
| 963 |
+
|
| 964 |
+
```bash
|
| 965 |
+
git add index.html
|
| 966 |
+
git commit -m "feat: render leaderboards from data/leaderboard.json with static fallback"
|
| 967 |
+
```
|
| 968 |
+
|
| 969 |
+
---
|
| 970 |
+
|
| 971 |
+
### Task 10: Verify render in browser
|
| 972 |
+
|
| 973 |
+
**Files:** none (verification only)
|
| 974 |
+
|
| 975 |
+
- [ ] **Step 1: Serve the site locally**
|
| 976 |
+
|
| 977 |
+
The user runs a local web server (IntelliJ) at the project root. Confirm the page is reachable (e.g. `http://localhost:63342/NowAI-Bench/index.html` or the user's configured URL). If not running, start one: `cd <repo root> && python3 -m http.server 8080` and use `http://localhost:8080/`.
|
| 978 |
+
|
| 979 |
+
- [ ] **Step 2: Load the page and assert via Chrome DevTools MCP**
|
| 980 |
+
|
| 981 |
+
Navigate to the page. Using `mcp__plugin_chrome-devtools-mcp_chrome-devtools__evaluate_script`, run:
|
| 982 |
+
|
| 983 |
+
```js
|
| 984 |
+
() => ({
|
| 985 |
+
eogRunners: document.querySelectorAll('#eog-runners > div').length,
|
| 986 |
+
accRunners: document.querySelectorAll('#eva-acc-runners > div').length,
|
| 987 |
+
expRunners: document.querySelectorAll('#eva-exp-runners > div').length,
|
| 988 |
+
eogTop: document.getElementById('eog-top-name').textContent,
|
| 989 |
+
accTop: document.getElementById('eva-acc-name').textContent,
|
| 990 |
+
expTop: document.getElementById('eva-exp-name').textContent,
|
| 991 |
+
})
|
| 992 |
+
```
|
| 993 |
+
|
| 994 |
+
Expected: each `*Runners` count is `4` (ranks 2β5; top is rendered separately), totaling 5 per board. `eogTop` reflects the synced #1 (e.g. "Gemini 3.5 Flash (High)"), not the old static "Claude Opus 4.5".
|
| 995 |
+
|
| 996 |
+
- [ ] **Step 3: Check console for errors**
|
| 997 |
+
|
| 998 |
+
Use `mcp__plugin_chrome-devtools-mcp_chrome-devtools__list_console_messages`.
|
| 999 |
+
Expected: no errors. (A "JSON load failed" warning means the server isn't serving `data/leaderboard.json` at the expected relative path β fix the path/serving before continuing.)
|
| 1000 |
+
|
| 1001 |
+
- [ ] **Step 4: Screenshot for visual confirmation**
|
| 1002 |
+
|
| 1003 |
+
Use `mcp__plugin_chrome-devtools-mcp_chrome-devtools__take_screenshot` and confirm the three cards show 5 rows each with bars, matching the prior styling.
|
| 1004 |
+
|
| 1005 |
+
---
|
| 1006 |
+
|
| 1007 |
+
### Task 11: Write `SKILL.md`
|
| 1008 |
+
|
| 1009 |
+
**Files:**
|
| 1010 |
+
- Create: `.claude/skills/sync-nowai-leaderboard/SKILL.md`
|
| 1011 |
+
|
| 1012 |
+
- [ ] **Step 1: Create SKILL.md**
|
| 1013 |
+
|
| 1014 |
+
```markdown
|
| 1015 |
+
---
|
| 1016 |
+
name: sync-nowai-leaderboard
|
| 1017 |
+
description: Use when syncing the NowAI-Bench landing-page leaderboards from upstream EOG + EVA sources, or when a scheduled run fires to check for new results. Pulls both sources, re-derives data/leaderboard.json, and pushes to staging only on sane changes.
|
| 1018 |
+
---
|
| 1019 |
+
|
| 1020 |
+
# Sync NowAI-Bench Leaderboard
|
| 1021 |
+
|
| 1022 |
+
Pulls the two upstream leaderboard sources, re-derives `data/leaderboard.json`,
|
| 1023 |
+
and (when run with `--push`) deploys the change to the **staging** HF Space only.
|
| 1024 |
+
|
| 1025 |
+
## Sources
|
| 1026 |
+
- EOG: `https://raw.githubusercontent.com/EnterpriseOps-Gym/EnterpriseOps-Gym.github.io/main/script.js` (`LEADERBOARD_DATA` array)
|
| 1027 |
+
- EVA: `https://raw.githubusercontent.com/ServiceNow/eva/main/website/src/data/leaderboardStats.json` (`systems[]`)
|
| 1028 |
+
|
| 1029 |
+
## How to run
|
| 1030 |
+
|
| 1031 |
+
From the repo root:
|
| 1032 |
+
|
| 1033 |
+
1. **Preview (always do this first):**
|
| 1034 |
+
`node .claude/skills/sync-nowai-leaderboard/sync.mjs --check`
|
| 1035 |
+
Prints the diff vs the committed `data/leaderboard.json`. No writes.
|
| 1036 |
+
|
| 1037 |
+
2. **Inspect the printed diff.** If it looks sane (no garbled names, scores in
|
| 1038 |
+
range, no wild swings), proceed. If sanity gates already flagged issues, the
|
| 1039 |
+
script aborts β investigate upstream before forcing anything.
|
| 1040 |
+
|
| 1041 |
+
3. **Apply + deploy to staging:**
|
| 1042 |
+
`node .claude/skills/sync-nowai-leaderboard/sync.mjs --push`
|
| 1043 |
+
Writes the file, commits, and pushes to `origin` (staging). Notifies via
|
| 1044 |
+
Teams webhook if `TEAMS_WEBHOOK_URL` is set.
|
| 1045 |
+
|
| 1046 |
+
To commit locally without deploying, use `--apply` instead of `--push`.
|
| 1047 |
+
|
| 1048 |
+
## Guardrails (NON-NEGOTIABLE)
|
| 1049 |
+
- **Push to `origin` (staging) ONLY. NEVER push to `prod`.** Promotion to prod is
|
| 1050 |
+
a human decision made after reviewing staging.
|
| 1051 |
+
- The script exits non-zero and notifies (does not commit) if: a fetch fails, a
|
| 1052 |
+
source fails to parse, any board has <3 rows, any score is out of range
|
| 1053 |
+
(EOG 0β100, EVA 0β1), or the top score swings >25 (EOG) / >0.25 (EVA) vs the
|
| 1054 |
+
committed data. Do not override these by editing the JSON by hand to bypass.
|
| 1055 |
+
|
| 1056 |
+
## Tests
|
| 1057 |
+
`cd .claude/skills/sync-nowai-leaderboard && node --test`
|
| 1058 |
+
|
| 1059 |
+
## Open TBD
|
| 1060 |
+
- Teams webhook URL not yet provisioned. Until then, notifications print to the
|
| 1061 |
+
run log. Set `TEAMS_WEBHOOK_URL` (as a secret/env var, never committed) to enable.
|
| 1062 |
+
```
|
| 1063 |
+
|
| 1064 |
+
- [ ] **Step 2: Verify the skill is discoverable**
|
| 1065 |
+
|
| 1066 |
+
Run: `git check-ignore .claude/skills/sync-nowai-leaderboard/SKILL.md`
|
| 1067 |
+
Expected: **no output** (file is NOT ignored β the `!.claude/skills/` rule keeps it tracked). If it prints the path, the `.gitignore` negation is wrong β fix before committing.
|
| 1068 |
+
|
| 1069 |
+
- [ ] **Step 3: Commit**
|
| 1070 |
+
|
| 1071 |
+
```bash
|
| 1072 |
+
git add .claude/skills/sync-nowai-leaderboard/SKILL.md
|
| 1073 |
+
git commit -m "docs: add SKILL.md for sync-nowai-leaderboard"
|
| 1074 |
+
```
|
| 1075 |
+
|
| 1076 |
+
---
|
| 1077 |
+
|
| 1078 |
+
### Task 12: Schedule the daily run
|
| 1079 |
+
|
| 1080 |
+
**Files:** none (creates a user-level Claude Code cron, not a repo file)
|
| 1081 |
+
|
| 1082 |
+
- [ ] **Step 1: Create the daily schedule**
|
| 1083 |
+
|
| 1084 |
+
Use the `CronCreate` tool (load its schema via ToolSearch: `select:CronCreate`). Create a daily job whose prompt instructs Claude to run the skill end-to-end:
|
| 1085 |
+
|
| 1086 |
+
- Schedule: daily (e.g. `0 14 * * *` β 14:00 UTC β 7am PT; adjust to preference).
|
| 1087 |
+
- Prompt (verbatim intent): "Invoke the sync-nowai-leaderboard skill in the NowAI-Bench repo. Run `node .claude/skills/sync-nowai-leaderboard/sync.mjs --check` first; if it reports sane changes and no gate failures, run it again with `--push` to deploy to staging. Report the diff summary. Never touch prod."
|
| 1088 |
+
|
| 1089 |
+
- [ ] **Step 2: Confirm it registered**
|
| 1090 |
+
|
| 1091 |
+
Use the `CronList` tool (`select:CronList`). Expected: the new daily job appears.
|
| 1092 |
+
|
| 1093 |
+
Note: the schedule lives in user-level Claude Code config, not the repo. It must be re-created on any machine that should run the sync.
|
| 1094 |
+
|
| 1095 |
+
---
|
| 1096 |
+
|
| 1097 |
+
## Self-Review
|
| 1098 |
+
|
| 1099 |
+
**Spec coverage:**
|
| 1100 |
+
- Page refactor to `data.json` + inline render + fallback β Tasks 8, 9, 10. β
|
| 1101 |
+
- Sync skill in `.claude/skills/` β Tasks 1, 7, 11. β
|
| 1102 |
+
- Transform/mapping rules (EOG avg sort, EVA A/X boards, subtitles, vendor map, bars, CI) β Tasks 2β5. β
|
| 1103 |
+
- Compare-derived-output change detection β Task 5 (`hasChanges`). β
|
| 1104 |
+
- Sanity gates (HTTP, β₯3 rows, range, Ξ guard) β Task 6 + wired in Task 7. β
|
| 1105 |
+
- Staging-only push, never prod β Task 7 (`--push` β `origin` only), Task 11 guardrails. β
|
| 1106 |
+
- Provenance hashes in `sources` β Task 7 (`sha256`). β
|
| 1107 |
+
- Teams webhook notify (TBD) + fallback β Task 7 `notify()`, Task 11 Open TBD. β
|
| 1108 |
+
- Daily cadence via CronCreate β Task 12. β
|
| 1109 |
+
- `.claude/skills/` tracked, rest ignored β verified in Task 11 Step 2. β
|
| 1110 |
+
- Tests (transform unit + render check) β Tasks 2β6 (unit), Task 10 (render). β
|
| 1111 |
+
|
| 1112 |
+
**Refinement noted vs spec:** spec described `--apply` as write+commit+push. Plan splits this: `--apply` = write+commit (no push), `--push` = write+commit+push to staging. Push becomes explicit opt-in so dev/init runs (Task 8) cannot accidentally deploy. Safer; same end behavior for the cron (`--push`).
|
| 1113 |
+
|
| 1114 |
+
**Placeholder scan:** no TBD/TODO in code steps; the only "TBD" is the Teams webhook, which is a real external dependency documented as such. β
|
| 1115 |
+
|
| 1116 |
+
**Type consistency:** `parseEogData`/`deriveEog`/`parseEvaData`/`deriveEvaBoard`/`buildLeaderboard`/`hasChanges`/`diffBoards`/`validate`/`vendorFromName` names match across lib, tests, and `sync.mjs` imports. EOG rows expose `{rank,model,org,type,score,bar}`; EVA rows expose `{rank,name,subtitle,score,ciLower,ciUpper,bar}` β consistent between derive functions, the render script, and `diffBoards` (`label = r.model ?? r.name`). β
|
| 1117 |
+
|
| 1118 |
+
---
|
| 1119 |
+
|
| 1120 |
+
## Out of Scope (per spec)
|
| 1121 |
+
- Auto-promotion to prod.
|
| 1122 |
+
- Per-domain / radar view (v1.1).
|
| 1123 |
+
- Historical trend tracking.
|