Add agents file and skills

.agents/skills/sync-nowai-leaderboard/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: sync-nowai-leaderboard
+description: Use when syncing the NowAI-Bench landing-page leaderboards from upstream EOG + EVA sources, or when a scheduled run fires to check for new results. Pulls both sources, re-derives data/leaderboard.json, and pushes to staging only on sane changes.
+---
+
+# Sync NowAI-Bench Leaderboard
+
+Pulls the two upstream leaderboard sources, re-derives `data/leaderboard.json`,
+and (when run with `--push`) deploys the change to the **staging** HF Space only.
+
+## Sources
+- EOG: `https://raw.githubusercontent.com/EnterpriseOps-Gym/EnterpriseOps-Gym.github.io/main/script.js` (`LEADERBOARD_DATA` array)
+- EVA: `https://raw.githubusercontent.com/ServiceNow/eva/main/website/src/data/leaderboardStats.json` (`systems[]`)
+
+## How to run
+
+From the repo root:
+
+1. **Preview (always do this first):**
+   `node .Codex/skills/sync-nowai-leaderboard/sync.mjs --check`
+   Prints the diff vs the committed `data/leaderboard.json`. No writes.
+
+2. **Inspect the printed diff.** If it looks sane (no garbled names, scores in
+   range, no wild swings), proceed. If sanity gates already flagged issues, the
+   script aborts — investigate upstream before forcing anything.
+
+3. **Apply + deploy to staging:**
+   `node .Codex/skills/sync-nowai-leaderboard/sync.mjs --push`
+   Writes the file, commits, and pushes to `origin` (staging). Posts an Adaptive
+   Card to Teams (success / failure / abort) if `TEAMS_WEBHOOK_URL` is set.
+
+   To commit locally without deploying, use `--apply` instead of `--push`.
+
+## Guardrails (NON-NEGOTIABLE)
+- **Push to `origin` (staging) ONLY. NEVER push to `prod`.** Promotion to prod is
+  a human decision made after reviewing staging. The script refuses to push
+  unless `origin` resolves to the staging Space (`NowAI-Bench-Staging`).
+- The script exits non-zero and notifies (does not commit) if: a fetch fails, a
+  source fails to parse, any board has <3 rows, any score is out of range
+  (EOG 0–100, EVA 0–1), or the top score swings >25 (EOG) / >0.25 (EVA) vs the
+  committed data. Do not override these by editing the JSON by hand to bypass.
+
+## Tests
+`cd .Codex/skills/sync-nowai-leaderboard && node --test`
+
+## Teams notifications
+- Set `TEAMS_WEBHOOK_URL` (secret/env var, never committed — keep it in the
+  gitignored `.env`) to enable. The URL is a Power Automate "Workflows" webhook
+  (`*.powerplatform.com`), so `lib/teams.mjs` posts an **Adaptive Card** in the
+  `message/attachments` envelope that template expects — not a raw MessageCard.
+- Cards: green ✅ success (with "Open staging" button), red ❌ failure, amber ⚠️
+  sanity-gate abort, blue 📋 Friday heartbeat (no changes detected — shows last
+  date `data/leaderboard.json` actually changed). No card on "no changes" Mon–Thu.
+- The daily 14:00 launchd job (`~/.nowai-sync/`) loads `TEAMS_WEBHOOK_URL` from
+  `.env` and also fires a local macOS desktop notification for every outcome.
+
+## Logs
+
+Every launchd run appends to `~/Library/Logs/nowai-sync.log`. Each entry includes
+timestamp, `sync exit=<code>`, full sync output, and a `RESULT:` summary line.
+Tail it to confirm the job is running during quiet weeks with no upstream changes:
+
+```
+tail -50 ~/Library/Logs/nowai-sync.log
+```

.agents/skills/sync-nowai-leaderboard/lib/teams.mjs

@@ -0,0 +1,107 @@
+// Posts an Adaptive Card to a Microsoft Teams channel via a Power Automate
+// "Workflows" webhook (the *.powerplatform.com / *.logic.azure.com replacement
+// for the retired webhook.office.com Office 365 connector).
+//
+// The default "Post to a channel when a webhook request is received" workflow
+// template expects an Adaptive Card wrapped in a message/attachments envelope —
+// NOT a raw MessageCard. The card below mirrors the look of Plato chat's Teams
+// MessageCard (bold title, colored accent, subtitle, fact table, optional link).
+//
+// Secret comes from TEAMS_WEBHOOK_URL in the environment (never committed).
+
+// Adaptive Card TextBlock colors map to outcome semantics.
+const PALETTE = {
+  success: 'Good',     // green
+  info: 'Accent',      // blue
+  warning: 'Warning',  // amber
+  error: 'Attention',  // red
+};
+
+export function buildCard({ kind = 'info', title, subtitle, facts = [], openUrl, openLabel = 'Open staging' }) {
+  // Header band: shaded `emphasis` container that bleeds to the card edges, so the
+  // colored title + subtitle read as a distinct zone above the fact table (the
+  // visual hierarchy of a Teams connector card / Plato chat alert).
+  const header = {
+    type: 'Container',
+    style: 'emphasis',
+    bleed: true,
+    items: [
+      {
+        type: 'TextBlock',
+        text: title,
+        weight: 'Bolder',
+        size: 'Large',
+        color: PALETTE[kind] ?? 'Default',
+        wrap: true,
+        spacing: 'None',
+      },
+    ],
+  };
+  if (subtitle) {
+    header.items.push({ type: 'TextBlock', text: subtitle, isSubtle: true, spacing: 'Small', wrap: true });
+  }
+
+  const body = [header];
+  if (facts.length) {
+    body.push({
+      type: 'FactSet',
+      spacing: 'Medium',
+      separator: true,
+      facts: facts.map(([title, value]) => ({ title, value: String(value) })),
+    });
+  }
+
+  const card = {
+    $schema: 'http://adaptivecards.io/schemas/adaptive-card.json',
+    type: 'AdaptiveCard',
+    version: '1.4',
+    body,
+    msteams: { width: 'Full' },
+  };
+  if (openUrl) {
+    card.actions = [{ type: 'Action.OpenUrl', title: openLabel, url: openUrl }];
+  }
+  return {
+    type: 'message',
+    attachments: [{ contentType: 'application/vnd.microsoft.card.adaptive', content: card }],
+  };
+}
+
+// Returns true on a 2xx post, false otherwise (never throws — notification
+// failure must not mask the underlying sync result).
+export async function postTeams(opts) {
+  const url = process.env.TEAMS_WEBHOOK_URL;
+  if (!url) {
+    console.log('[teams] no TEAMS_WEBHOOK_URL set; skipping card.');
+    return false;
+  }
+  try {
+    const res = await fetch(url, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify(buildCard(opts)),
+    });
+    if (!res.ok) {
+      console.error(`[teams] HTTP ${res.status}: ${await res.text().catch(() => '')}`);
+      return false;
+    }
+    return true;
+  } catch (e) {
+    console.error('[teams] post failed: ' + e.message);
+    return false;
+  }
+}
+
+// CLI (manual testing only): node teams.mjs <kind> <title> <subtitle> [name=value ...] [--open URL]
+const isMain = import.meta.url === `file://${process.argv[1]}`;
+if (isMain) {
+  const argv = process.argv.slice(2);
+  let openUrl;
+  const openIdx = argv.indexOf('--open');
+  if (openIdx !== -1) { openUrl = argv[openIdx + 1]; argv.splice(openIdx, 2); }
+  const [kind, title, subtitle, ...rest] = argv;
+  const facts = rest.map((s) => { const i = s.indexOf('='); return [s.slice(0, i), s.slice(i + 1)]; });
+  const ok = await postTeams({ kind, title, subtitle, facts, openUrl });
+  console.log(ok ? '[teams] posted.' : '[teams] not posted.');
+  process.exit(ok ? 0 : 1);
+}

.agents/skills/sync-nowai-leaderboard/lib/transform.mjs

@@ -0,0 +1,164 @@
+// Pure transforms: parse upstream sources and derive our leaderboard schema.
+// No network or filesystem access lives here.
+
+export function parseEogData(scriptText) {
+  const start = scriptText.indexOf('LEADERBOARD_DATA');
+  if (start === -1) throw new Error('LEADERBOARD_DATA not found in EOG script');
+  const open = scriptText.indexOf('[', start);
+  if (open === -1) throw new Error('EOG array open bracket not found');
+
+  // Match brackets to find the array literal end. Model names contain no [ or ].
+  let depth = 0;
+  let end = -1;
+  for (let i = open; i < scriptText.length; i++) {
+    const c = scriptText[i];
+    if (c === '[') depth++;
+    else if (c === ']') {
+      depth--;
+      if (depth === 0) { end = i; break; }
+    }
+  }
+  if (end === -1) throw new Error('EOG array close bracket not found');
+
+  let body = scriptText.slice(open, end + 1);
+  body = body.replace(/\/\/[^\n]*/g, '');              // strip line comments
+  body = body.replace(/([{,]\s*)([A-Za-z_]\w*)\s*:/g, '$1"$2":'); // quote keys
+  body = body.replace(/,(\s*[}\]])/g, '$1');           // strip trailing commas
+
+  try {
+    return JSON.parse(body);
+  } catch (e) {
+    throw new Error('EOG array parse failed: ' + e.message);
+  }
+}
+
+const round1 = (n) => Math.round(n * 10) / 10;
+
+const VENDOR_MAP = [
+  [/gemini/i, 'Google'],
+  [/gpt|openai|\brealtime\b/i, 'OpenAI'],
+  [/whisper/i, 'OpenAI'],
+  [/claude/i, 'Anthropic'],
+  [/nova/i, 'Amazon'],
+  [/qwen/i, 'Alibaba'],
+  [/deepseek/i, 'DeepSeek'],
+  [/llama/i, 'Meta'],
+];
+
+export function vendorFromName(name) {
+  for (const [re, vendor] of VENDOR_MAP) if (re.test(name)) return vendor;
+  return name.split(/[\s+]/)[0];
+}
+
+export function deriveEog(eogArray) {
+  const sorted = [...eogArray].sort((a, b) => b.avg - a.avg).slice(0, 5);
+  return {
+    metricLabel: 'Task Success Rate · Oracle mode',
+    rows: sorted.map((m, i) => ({
+      rank: i + 1,
+      model: m.model,
+      org: vendorFromName(m.model),
+      type: m.type,
+      score: round1(m.avg),
+      // bar = absolute % of max. EOG score is already a 0–100 success rate.
+      bar: Math.round(m.avg),
+    })),
+  };
+}
+
+const round2 = (n) => Math.round(n * 100) / 100;
+
+export function parseEvaData(jsonText) {
+  const data = JSON.parse(jsonText);
+  if (!Array.isArray(data.systems)) throw new Error('EVA data missing systems[] array');
+  return data;
+}
+
+export function deriveEvaBoard(systems, metricKey) {
+  const scored = systems
+    .map((s) => ({ s, pooled: s.clean?.[metricKey]?.pooled }))
+    .filter((x) => x.pooled && typeof x.pooled.point === 'number')
+    .sort((a, b) => b.pooled.point - a.pooled.point)
+    .slice(0, 3);
+  return {
+    metricLabel: 'Pass@1',
+    rows: scored.map(({ s, pooled }, i) => ({
+      rank: i + 1,
+      name: s.name,
+      subtitle:
+        s.type === 'cascade'
+          ? 'Mixed Models · Cascade'
+          : `${vendorFromName(s.name)} · Speech-to-Speech`,
+      score: round2(pooled.point),
+      ciLower: round2(pooled.ci_lower),
+      ciUpper: round2(pooled.ci_upper),
+      // bar = absolute % of max. EVA Pass@1 is a 0–1 proportion.
+      bar: Math.round(pooled.point * 100),
+    })),
+  };
+}
+
+export function buildLeaderboard({ eogScript, evaJson, sources, now }) {
+  const eogArray = parseEogData(eogScript);
+  const evaData = parseEvaData(evaJson);
+  return {
+    updated: now,
+    sources,
+    eog: deriveEog(eogArray),
+    evaAccuracy: deriveEvaBoard(evaData.systems, 'EVA-A_pass'),
+    evaExperience: deriveEvaBoard(evaData.systems, 'EVA-X_pass'),
+  };
+}
+
+const BOARD_KEYS = ['eog', 'evaAccuracy', 'evaExperience'];
+
+// Compare only the parts we render (ignore `updated`/`sources` meta).
+export function hasChanges(oldData, newData) {
+  if (!oldData) return true;
+  const pick = (d) => JSON.stringify(BOARD_KEYS.map((k) => d[k]));
+  return pick(oldData) !== pick(newData);
+}
+
+export function diffBoards(oldData, newData) {
+  const label = (r) => r.model ?? r.name;
+  const lines = [];
+  for (const key of BOARD_KEYS) {
+    const oldRows = oldData?.[key]?.rows ?? [];
+    const newRows = newData[key].rows;
+    const oldStr = oldRows.map((r) => `${label(r)}=${r.score}`).join(', ');
+    const newStr = newRows.map((r) => `${label(r)}=${r.score}`).join(', ');
+    if (oldStr !== newStr) {
+      lines.push(`${key}:\n  old: ${oldStr || '(none)'}\n  new: ${newStr}`);
+    }
+  }
+  return lines;
+}
+
+// Returns an array of human-readable issue strings; empty array = clean.
+export function validate(newData, oldData) {
+  const issues = [];
+  const specs = [
+    ['eog', newData.eog, 0, 100, 25, oldData?.eog],
+    ['evaAccuracy', newData.evaAccuracy, 0, 1, 0.25, oldData?.evaAccuracy],
+    ['evaExperience', newData.evaExperience, 0, 1, 0.25, oldData?.evaExperience],
+  ];
+  for (const [name, board, min, max, guard, oldBoard] of specs) {
+    if (!board || !Array.isArray(board.rows)) {
+      issues.push(`${name}: missing rows`);
+      continue;
+    }
+    if (board.rows.length < 3) issues.push(`${name}: only ${board.rows.length} rows (<3)`);
+    for (const r of board.rows) {
+      if (r.score < min || r.score > max) {
+        issues.push(`${name}: score ${r.score} out of [${min},${max}]`);
+      }
+    }
+    const oldTop = oldBoard?.rows?.[0]?.score;
+    const newTop = board.rows[0]?.score;
+    if (typeof oldTop === 'number' && typeof newTop === 'number') {
+      const delta = Math.abs(newTop - oldTop);
+      if (delta > guard) issues.push(`${name}: top score swing ${delta.toFixed(2)} > ${guard}`);
+    }
+  }
+  return issues;
+}

.agents/skills/sync-nowai-leaderboard/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "sync-nowai-leaderboard",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "test": "node --test"
+  }
+}

.agents/skills/sync-nowai-leaderboard/sync.mjs

@@ -0,0 +1,266 @@
+#!/usr/bin/env node
+// Orchestrator for the leaderboard sync. Side effects only; pure logic in lib/transform.mjs.
+import { readFile, writeFile, mkdir } from 'node:fs/promises';
+import { existsSync, readFileSync, writeFileSync } from 'node:fs';
+import { createHash } from 'node:crypto';
+import { execFileSync } from 'node:child_process';
+import { join, dirname } from 'node:path';
+import { buildLeaderboard, hasChanges, diffBoards, validate } from './lib/transform.mjs';
+import { postTeams } from './lib/teams.mjs';
+
+const STAGING_URL = 'https://huggingface.co/spaces/ServiceNow-AI/NowAI-Bench-Staging';
+
+const EOG_URL = 'https://raw.githubusercontent.com/EnterpriseOps-Gym/EnterpriseOps-Gym.github.io/main/script.js';
+const EVA_URL = 'https://raw.githubusercontent.com/ServiceNow/eva/main/website/src/data/leaderboardStats.json';
+
+// Guard: `origin` must point at the staging Space. Pushing only ever targets a
+// remote whose URL contains this slug, so a misconfigured `origin` can never
+// deploy to the prod Space (ServiceNow-AI/NowAI-Bench).
+const STAGING_SLUG = 'NowAI-Bench-Staging';
+const FETCH_TIMEOUT_MS = 30_000;
+
+function repoRoot() {
+  return execFileSync('git', ['rev-parse', '--show-toplevel'], { encoding: 'utf8' }).trim();
+}
+
+function sha256(text) {
+  return 'sha256:' + createHash('sha256').update(text).digest('hex');
+}
+
+async function fetchText(url, timeoutMs = FETCH_TIMEOUT_MS) {
+  const controller = new AbortController();
+  const timer = setTimeout(() => controller.abort(), timeoutMs);
+  try {
+    const res = await fetch(url, { signal: controller.signal });
+    if (!res.ok) throw new Error(`fetch ${url} → HTTP ${res.status}`);
+    return await res.text();
+  } finally {
+    clearTimeout(timer);
+  }
+}
+
+// Shared subtitle line for every Teams card: "NowAI-Bench Sync · 2026-06-12 14:00 UTC".
+function subtitle() {
+  return 'NowAI-Bench Sync · ' + new Date().toISOString().replace('T', ' ').slice(0, 16) + ' UTC';
+}
+
+// Convenience: an error/abort card with a consistent title.
+function notifyFailure(facts, kind = 'error') {
+  return postTeams({
+    kind,
+    title: kind === 'warning' ? '⚠️ Leaderboard sync aborted' : '❌ Leaderboard sync failed',
+    subtitle: subtitle(),
+    facts,
+  });
+}
+
+function git(root, args) {
+  return execFileSync('git', args, { cwd: root, encoding: 'utf8' });
+}
+
+// Quarter helpers ─────────────────────────────────────────────────────────────
+
+function quarterForDate(d) {
+  return `${d.getUTCFullYear()}Q${Math.floor(d.getUTCMonth() / 3) + 1}`;
+}
+
+// Read index.html, extract the year+quarter label, compare to today's quarter.
+// Returns { needsUpdate, current, expected, html, htmlPath }.
+function detectQuarterLabel(root) {
+  const htmlPath = join(root, 'index.html');
+  const html = readFileSync(htmlPath, 'utf8');
+  const RE = /(\d{4})<span class="text-on-surface-variant text-base font-medium ml-1">(Q[1-4])<\/span>/;
+  const m = html.match(RE);
+  if (!m) return { needsUpdate: false };
+  const current = `${m[1]}${m[2]}`;
+  const expected = quarterForDate(new Date());
+  return { needsUpdate: current !== expected, current, expected, html, htmlPath };
+}
+
+function applyQuarterLabel({ html, htmlPath, expected }) {
+  const year = expected.slice(0, 4);
+  const q = expected.slice(4);
+  const updated = html.replace(
+    /(\d{4})<span class="text-on-surface-variant text-base font-medium ml-1">(Q[1-4])<\/span>/,
+    `${year}<span class="text-on-surface-variant text-base font-medium ml-1">${q}</span>`,
+  );
+  writeFileSync(htmlPath, updated);
+}
+
+async function main() {
+  const args = new Set(process.argv.slice(2));
+  const mode = args.has('--push') ? 'push' : args.has('--apply') ? 'apply' : 'check';
+
+  const root = repoRoot();
+  const dataPath = join(root, 'data', 'leaderboard.json');
+
+  // 1. Fetch upstream.
+  let eogScript, evaJson;
+  try {
+    [eogScript, evaJson] = await Promise.all([fetchText(EOG_URL), fetchText(EVA_URL)]);
+  } catch (e) {
+    console.error('[sync] fetch failed: ' + e.message);
+    await notifyFailure([['Stage', 'upstream fetch'], ['Error', e.message]]);
+    process.exit(1);
+  }
+
+  // 2. Build new data.
+  let newData;
+  try {
+    newData = buildLeaderboard({
+      eogScript,
+      evaJson,
+      sources: {
+        eog: { url: EOG_URL, hash: sha256(eogScript), fetchedAt: new Date().toISOString() },
+        eva: { url: EVA_URL, hash: sha256(evaJson), fetchedAt: new Date().toISOString() },
+      },
+      now: new Date().toISOString(),
+    });
+  } catch (e) {
+    console.error('[sync] parse/derive failed: ' + e.message);
+    await notifyFailure([['Stage', 'parse / derive (schema drift?)'], ['Error', e.message]]);
+    process.exit(1);
+  }
+
+  // 3. Load existing.
+  let oldData = null;
+  if (existsSync(dataPath)) {
+    try {
+      oldData = JSON.parse(await readFile(dataPath, 'utf8'));
+    } catch (e) {
+      console.error('[sync] existing leaderboard.json is unreadable: ' + e.message);
+      await notifyFailure([['Stage', 'read existing leaderboard.json'], ['Error', e.message]]);
+      process.exit(1);
+    }
+  }
+
+  // 3.5. Detect quarter label mismatch in index.html.
+  const quarterInfo = detectQuarterLabel(root);
+  if (quarterInfo.needsUpdate) {
+    console.log(`[sync] quarter label: ${quarterInfo.current} → ${quarterInfo.expected}`);
+  }
+
+  // 4. Change detection.
+  const leaderboardChanged = hasChanges(oldData, newData);
+
+  if (!leaderboardChanged && !quarterInfo.needsUpdate) {
+    console.log('[sync] no display changes. Done.');
+    // Friday heartbeat: post a card so the Teams channel confirms the job is alive
+    // even during quiet weeks with no ranking changes.
+    if (new Date().getDay() === 5) {
+      let lastChanged = 'unknown';
+      try {
+        lastChanged = git(root, ['log', '-1', '--format=%as', '--', 'data/leaderboard.json']).trim() || 'unknown';
+      } catch (_) {}
+      await postTeams({
+        kind: 'info',
+        title: '📋 Leaderboard sync — no changes this week',
+        subtitle: subtitle(),
+        facts: [
+          ['Status', 'Up to date — no ranking changes detected'],
+          ['Last changed', lastChanged],
+        ],
+        openUrl: STAGING_URL,
+        openLabel: 'Open staging',
+      });
+    }
+    return;
+  }
+
+  let changeLines = [];
+  if (leaderboardChanged) {
+    changeLines = diffBoards(oldData, newData);
+    console.log('[sync] changes detected:\n' + changeLines.join('\n'));
+
+    // 5. Sanity gates (only when leaderboard data changes).
+    const issues = validate(newData, oldData);
+    if (issues.length) {
+      console.error('[sync] sanity gates failed:\n - ' + issues.join('\n - '));
+      await notifyFailure(
+        [['Reason', 'sanity gates failed'], ...issues.map((s, i) => [`Issue ${i + 1}`, s])],
+        'warning',
+      );
+      process.exit(2);
+    }
+  }
+
+  if (mode === 'check') {
+    if (quarterInfo.needsUpdate) {
+      console.log(`[sync] header quarter: ${quarterInfo.current} → ${quarterInfo.expected}`);
+    }
+    console.log('[sync] --check: no writes. Re-run with --apply (commit) or --push (commit + staging).');
+    return;
+  }
+
+  // 6. Write + commit (apply and push only; never in check mode).
+  if (mode !== 'apply' && mode !== 'push') return;
+
+  // For push: verify `origin` is staging BEFORE writing/committing, so a push
+  // from a wrong checkout can't leave a dangling local commit to roll back.
+  if (mode === 'push') {
+    const originUrl = git(root, ['remote', 'get-url', 'origin']).trim();
+    if (!originUrl.includes(STAGING_SLUG)) {
+      console.error(`[sync] refusing to push: origin is not the staging remote (${originUrl}). Nothing written.`);
+      await notifyFailure([['Reason', 'push refused — origin is not staging'], ['origin', originUrl]]);
+      process.exit(4);
+    }
+  }
+
+  if (leaderboardChanged) {
+    await mkdir(dirname(dataPath), { recursive: true });
+    await writeFile(dataPath, JSON.stringify(newData, null, 2) + '\n');
+    git(root, ['add', dataPath]);
+  }
+
+  if (quarterInfo.needsUpdate) {
+    applyQuarterLabel(quarterInfo);
+    git(root, ['add', join(root, 'index.html')]);
+  }
+
+  const today = new Date().toISOString().slice(0, 10);
+  const parts = [];
+  if (leaderboardChanged) parts.push('EOG + EVA');
+  if (quarterInfo.needsUpdate) parts.push(`header → ${quarterInfo.expected}`);
+  const commitDetail = leaderboardChanged ? '\n\n' + changeLines.join('\n') : '';
+  const msg = `Sync leaderboard: ${parts.join(', ')} ${today}${commitDetail}`;
+  git(root, ['commit', '-m', msg]);
+  console.log('[sync] committed.');
+
+  // 7. Push (staging only) if requested. Origin already verified as staging above.
+  if (mode === 'push') {
+    try {
+      git(root, ['push', 'origin', 'HEAD:main']);
+      console.log('[sync] pushed to origin (staging).');
+      const noticeFacts = [];
+      if (leaderboardChanged) {
+        noticeFacts.push(['Boards changed', String(changeLines.length)]);
+        noticeFacts.push(['Detail', changeLines.join(' · ')]);
+      }
+      if (quarterInfo.needsUpdate) {
+        noticeFacts.push(['Header updated', `${quarterInfo.current} → ${quarterInfo.expected}`]);
+      }
+      if (!leaderboardChanged) {
+        noticeFacts.push(['Note', 'Quarter label only — no ranking changes']);
+      }
+      noticeFacts.push(['Next', 'Review staging, then promote to prod']);
+      await postTeams({
+        kind: 'success',
+        title: leaderboardChanged ? '✅ Staging ready for review' : '📅 Header updated — staging ready',
+        subtitle: subtitle(),
+        facts: noticeFacts,
+        openUrl: STAGING_URL,
+      });
+    } catch (e) {
+      console.error('[sync] push failed: ' + e.message);
+      await notifyFailure([['Stage', 'git push to staging'], ['Note', 'committed locally OK'], ['Error', e.message]]);
+      process.exit(3);
+    }
+  } else {
+    console.log('[sync] --apply: committed locally, not pushed. Use --push to deploy to staging.');
+  }
+}
+
+main().catch((e) => {
+  console.error('[sync] unexpected error: ' + e.message);
+  process.exit(1);
+});

.agents/skills/sync-nowai-leaderboard/test/transform.test.mjs

@@ -0,0 +1,178 @@
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { parseEogData } from '../lib/transform.mjs';
+
+const EOG_SAMPLE = `
+// ── Leaderboard Data ──
+const LEADERBOARD_DATA = [
+  // Closed Source
+  { model: "Gemini 3.5 Flash (High)", type: "closed", teams: 48.0, csm: 47.9, email: 55.8, itsm: 42.5, calendar: 42.3, hr: 40.8, drive: 55.2, hybrid: 35.3, avg: 46.0 },
+  { model: "Claude Opus 4.5", type: "closed", teams: 50.0, csm: 34.2, email: 51.9, itsm: 23.8, calendar: 43.2, hr: 32.1, drive: 49.5, hybrid: 30.7, avg: 37.0 },
+];
+function renderLeaderboard() {}
+`;
+
+test('parseEogData extracts the array of model objects', () => {
+  const arr = parseEogData(EOG_SAMPLE);
+  assert.equal(arr.length, 2);
+  assert.equal(arr[0].model, 'Gemini 3.5 Flash (High)');
+  assert.equal(arr[0].type, 'closed');
+  assert.equal(arr[0].avg, 46.0);
+  assert.equal(arr[1].model, 'Claude Opus 4.5');
+});
+
+test('parseEogData throws when marker missing', () => {
+  assert.throws(() => parseEogData('const X = 1;'), /LEADERBOARD_DATA not found/);
+});
+
+import { deriveEog, vendorFromName } from '../lib/transform.mjs';
+
+test('vendorFromName maps known model families', () => {
+  assert.equal(vendorFromName('Gemini 3.5 Flash'), 'Google');
+  assert.equal(vendorFromName('GPT-5.4'), 'OpenAI');
+  assert.equal(vendorFromName('Claude Opus 4.5'), 'Anthropic');
+  assert.equal(vendorFromName('Mystery-7B'), 'Mystery-7B'); // fallback: first token
+});
+
+test('deriveEog sorts by avg desc, takes top 5, computes bars', () => {
+  const arr = [
+    { model: 'A', type: 'closed', avg: 20 },
+    { model: 'B', type: 'open', avg: 40 },
+    { model: 'C', type: 'closed', avg: 30 },
+  ];
+  const out = deriveEog(arr);
+  assert.equal(out.metricLabel, 'Task Success Rate · Oracle mode');
+  assert.deepEqual(out.rows.map(r => r.model), ['B', 'C', 'A']);
+  assert.equal(out.rows[0].rank, 1);
+  assert.equal(out.rows[0].score, 40);
+  assert.equal(out.rows[0].bar, 40);   // absolute: avg 40 of max 100
+  assert.equal(out.rows[1].bar, 30);   // avg 30
+  assert.equal(out.rows[2].bar, 20);   // avg 20
+  assert.equal(out.rows[0].type, 'open');
+});
+
+import { parseEvaData, deriveEvaBoard } from '../lib/transform.mjs';
+
+const EVA_SAMPLE = JSON.stringify({
+  systems: [
+    {
+      id: 'a', name: 'Gemini Live', type: 's2s', stt: '-', llm: 'gemini', tts: '-',
+      clean: {
+        'EVA-A_pass': { pooled: { point: 0.30, ci_lower: 0.28, ci_upper: 0.32 } },
+        'EVA-X_pass': { pooled: { point: 0.49, ci_lower: 0.47, ci_upper: 0.51 } },
+      },
+    },
+    {
+      id: 'b', name: 'Nova + GPT + Sonic', type: 'cascade', stt: 'Nova', llm: 'GPT', tts: 'Sonic',
+      clean: {
+        'EVA-A_pass': { pooled: { point: 0.41, ci_lower: 0.39, ci_upper: 0.43 } },
+        'EVA-X_pass': { pooled: { point: 0.20, ci_lower: 0.18, ci_upper: 0.22 } },
+      },
+    },
+  ],
+});
+
+test('parseEvaData returns parsed object with systems array', () => {
+  const data = parseEvaData(EVA_SAMPLE);
+  assert.equal(data.systems.length, 2);
+});
+
+test('deriveEvaBoard ranks by accuracy point, builds subtitle + ci + bars', () => {
+  const data = parseEvaData(EVA_SAMPLE);
+  const out = deriveEvaBoard(data.systems, 'EVA-A_pass');
+  assert.equal(out.metricLabel, 'Pass@1');
+  assert.deepEqual(out.rows.map(r => r.name), ['Nova + GPT + Sonic', 'Gemini Live']);
+  assert.equal(out.rows[0].score, 0.41);
+  assert.equal(out.rows[0].subtitle, 'Mixed Models · Cascade');
+  assert.equal(out.rows[0].bar, 41);   // absolute: 0.41 × 100
+  assert.equal(out.rows[0].ciLower, 0.39);
+  assert.equal(out.rows[0].ciUpper, 0.43);
+  assert.equal(out.rows[1].subtitle, 'Google · Speech-to-Speech');
+  assert.equal(out.rows[1].bar, 30); // 0.30 × 100
+});
+
+test('deriveEvaBoard ranks experience board independently', () => {
+  const data = parseEvaData(EVA_SAMPLE);
+  const out = deriveEvaBoard(data.systems, 'EVA-X_pass');
+  assert.deepEqual(out.rows.map(r => r.name), ['Gemini Live', 'Nova + GPT + Sonic']);
+  assert.equal(out.rows[0].score, 0.49);
+});
+
+import { validate } from '../lib/transform.mjs';
+
+function boardWith(rows) { return { metricLabel: 'x', rows }; }
+function dataWith(eogRows, accRows, expRows) {
+  return { eog: boardWith(eogRows), evaAccuracy: boardWith(accRows), evaExperience: boardWith(expRows) };
+}
+
+test('validate passes clean data', () => {
+  const good = dataWith(
+    [{ rank: 1, model: 'A', score: 46 }, { rank: 2, model: 'B', score: 40 }, { rank: 3, model: 'C', score: 30 }],
+    [{ rank: 1, name: 'X', score: 0.4 }, { rank: 2, name: 'Y', score: 0.3 }, { rank: 3, name: 'Z', score: 0.2 }],
+    [{ rank: 1, name: 'X', score: 0.5 }, { rank: 2, name: 'Y', score: 0.4 }, { rank: 3, name: 'Z', score: 0.3 }],
+  );
+  assert.deepEqual(validate(good, null), []);
+});
+
+test('validate flags too-few rows', () => {
+  const bad = dataWith([{ rank: 1, model: 'A', score: 46 }], [], []);
+  const issues = validate(bad, null);
+  assert.ok(issues.some((i) => i.includes('eog') && i.includes('<3')));
+});
+
+test('validate flags out-of-range scores', () => {
+  const bad = dataWith(
+    [{ rank: 1, model: 'A', score: 150 }, { rank: 2, model: 'B', score: 40 }, { rank: 3, model: 'C', score: 30 }],
+    [{ rank: 1, name: 'X', score: 0.4 }, { rank: 2, name: 'Y', score: 0.3 }, { rank: 3, name: 'Z', score: 0.2 }],
+    [{ rank: 1, name: 'X', score: 0.5 }, { rank: 2, name: 'Y', score: 0.4 }, { rank: 3, name: 'Z', score: 0.3 }],
+  );
+  assert.ok(validate(bad, null).some((i) => i.includes('out of')));
+});
+
+test('validate flags suspicious top-score swing vs previous', () => {
+  const prev = dataWith(
+    [{ rank: 1, model: 'A', score: 46 }, { rank: 2, model: 'B', score: 40 }, { rank: 3, model: 'C', score: 30 }],
+    [{ rank: 1, name: 'X', score: 0.4 }, { rank: 2, name: 'Y', score: 0.3 }, { rank: 3, name: 'Z', score: 0.2 }],
+    [{ rank: 1, name: 'X', score: 0.5 }, { rank: 2, name: 'Y', score: 0.4 }, { rank: 3, name: 'Z', score: 0.3 }],
+  );
+  const now = dataWith(
+    [{ rank: 1, model: 'A', score: 10 }, { rank: 2, model: 'B', score: 8 }, { rank: 3, model: 'C', score: 6 }],
+    [{ rank: 1, name: 'X', score: 0.4 }, { rank: 2, name: 'Y', score: 0.3 }, { rank: 3, name: 'Z', score: 0.2 }],
+    [{ rank: 1, name: 'X', score: 0.5 }, { rank: 2, name: 'Y', score: 0.4 }, { rank: 3, name: 'Z', score: 0.3 }],
+  );
+  assert.ok(validate(now, prev).some((i) => i.includes('swing')));
+});
+
+import { buildLeaderboard, hasChanges, diffBoards } from '../lib/transform.mjs';
+
+test('buildLeaderboard assembles all three boards plus meta', () => {
+  const out = buildLeaderboard({
+    eogScript: EOG_SAMPLE,
+    evaJson: EVA_SAMPLE,
+    sources: { eog: { url: 'u1', hash: 'h1' }, eva: { url: 'u2', hash: 'h2' } },
+    now: '2026-06-02T00:00:00Z',
+  });
+  assert.equal(out.updated, '2026-06-02T00:00:00Z');
+  assert.equal(out.sources.eog.hash, 'h1');
+  assert.equal(out.eog.rows[0].model, 'Gemini 3.5 Flash (High)');
+  assert.equal(out.evaAccuracy.rows[0].name, 'Nova + GPT + Sonic');
+  assert.equal(out.evaExperience.rows[0].name, 'Gemini Live');
+});
+
+test('hasChanges ignores meta, detects board changes', () => {
+  const base = buildLeaderboard({ eogScript: EOG_SAMPLE, evaJson: EVA_SAMPLE, sources: {}, now: 'A' });
+  const sameDataNewMeta = buildLeaderboard({ eogScript: EOG_SAMPLE, evaJson: EVA_SAMPLE, sources: { eog: { hash: 'x' } }, now: 'B' });
+  assert.equal(hasChanges(base, sameDataNewMeta), false);
+
+  const changedEog = EOG_SAMPLE.replace('avg: 46.0', 'avg: 99.0');
+  const changed = buildLeaderboard({ eogScript: changedEog, evaJson: EVA_SAMPLE, sources: {}, now: 'A' });
+  assert.equal(hasChanges(base, changed), true);
+});
+
+test('diffBoards produces human-readable change lines', () => {
+  const base = buildLeaderboard({ eogScript: EOG_SAMPLE, evaJson: EVA_SAMPLE, sources: {}, now: 'A' });
+  const changedEog = EOG_SAMPLE.replace('avg: 46.0', 'avg: 41.0');
+  const changed = buildLeaderboard({ eogScript: changedEog, evaJson: EVA_SAMPLE, sources: {}, now: 'A' });
+  const lines = diffBoards(base, changed);
+  assert.ok(lines.some((l) => l.startsWith('eog:')));
+});

AGENTS.md

@@ -0,0 +1,135 @@
+# AGENTS.md
+
+## Purpose
+
+This repo powers the NowAI-Bench static site and its leaderboard sync workflow.
+The main site files are `index.html`, `style.css`, and `data/leaderboard.json`.
+
+## Instruction Order
+
+Read this file first.
+
+If a root `CLAUDE.md` exists, read it next and treat it as additional
+repo-specific user guidance.
+
+When the task is about leaderboard syncing or leaderboard data, also read:
+
+- `.claude/skills/sync-nowai-leaderboard/SKILL.md`
+- `docs/superpowers/specs/2026-06-02-leaderboard-sync-design.md`
+- `docs/superpowers/plans/2026-06-02-leaderboard-sync.md`
+
+## Leaderboard Sync Support
+
+This repo includes a repo-local skill/workflow at
+`.claude/skills/sync-nowai-leaderboard/`. Codex should support that workflow
+directly even if skill auto-discovery is unavailable.
+
+Use this run order:
+
+1. `node .claude/skills/sync-nowai-leaderboard/sync.mjs --check`
+2. Inspect the diff and sanity-gate output.
+3. Only if the result is sane, run
+   `node .claude/skills/sync-nowai-leaderboard/sync.mjs --push`
+
+## Deployment Guardrails
+
+- `origin` is staging: `ServiceNow-AI/NowAI-Bench-Staging`
+- `prod` is production: `ServiceNow-AI/NowAI-Bench`
+- Never push to `prod` as part of automated or routine sync work.
+- For sync tasks, prefer the scripted guardrails in `sync.mjs` over manual edits.
+
+## Verification
+
+If you change the sync workflow or transform logic, run:
+
+- `cd .claude/skills/sync-nowai-leaderboard && node --test`
+
+If you change site rendering, verify the static page still loads cleanly and
+that leaderboard data remains consistent with `data/leaderboard.json`.
+
+
+<claude-mem-context>
+# Memory Context
+
+# [NowAI-Bench] recent context, 2026-06-30 12:47pm PDT
+
+Legend: 🎯session 🔴bugfix 🟣feature 🔄refactor ✅change 🔵discovery ⚖️decision 🚨security_alert 🔐security_note
+Format: ID TIME TYPE TITLE
+Fetch details: get_observations([IDs]) | Search: mem-search skill
+
+Stats: 50 obs (14,815t read) | 128,192t work | 88% savings
+
+### Jun 12, 2026
+1761 3:07p ✅ Created project memory link for sync automation system
+1762 " ✅ Added sync automation entry to project memory index
+1764 5:32p ✅ Teams webhook integration configured in environment
+1765 5:33p 🔵 Teams MessageCard webhook format identified from plato chat pattern
+1766 5:34p 🔵 Teams webhook URL configured securely in NowAI-Bench
+1767 5:35p 🟣 Teams Adaptive Card webhook module implemented for NowAI-Bench
+1768 5:36p 🔵 Teams Adaptive Card successfully posted to Power Automate webhook
+1769 " ✅ Teams module integrated into leaderboard sync orchestrator
+1770 " 🔄 Notification system refactored to use Adaptive Cards
+1771 5:37p ✅ Error handling migrated to structured Teams notifications
+1772 " ✅ All sync error paths migrated to structured Teams notifications
+1773 5:38p 🟣 Success notification implemented for staging deployment
+1774 " ✅ TEAMS_WEBHOOK_URL integrated into cron sync script
+1775 " 🔵 Comprehensive validation confirms Teams notification integration working
+1776 5:39p 🔵 Error Adaptive Card successfully posted to Teams
+1777 " ✅ Teams notification feature documented as complete and deployed
+1778 " ✅ Teams notification feature integrated into automation design documentation
+1779 " ✅ Teams notification support integrated into project memory
+1780 5:43p 🔄 Adaptive Card layout refined with header band styling for better visual hierarchy
+1781 " 🔵 Refined Adaptive Card layout validated with both success and error styles
+1782 5:54p 🟣 Teams webhook integration for leaderboard sync notifications
+1783 " ✅ Git artifact exclusion: nowai-bench-v0.zip
+1784 " ✅ Documented Teams webhook configuration in .env.example
+1785 " 🟣 Teams Adaptive Card notifications committed to leaderboard sync
+1786 " ✅ Teams Adaptive Card feature deployed to staging
+1787 5:55p ✅ Launchd job restricted to weekdays (Mon-Fri) at 14:00
+1788 " 🔵 Launchd weekday scheduling verified and activated
+1789 " ✅ Updated DESIGN.md to reflect weekday-only scheduling
+1790 " ✅ Documented launchd weekday scheduling implementation detail
+1791 5:56p ✅ Updated project memory with weekday scheduling detail
+### Jun 23, 2026
+S1124 Diagnose why NowAI leaderboard sync hasn't been running; implement observability improvement during quiet data periods (Jun 23 at 10:23 AM)
+S1125 Investigate sync inactivity concern and improve observability for background automation running during quiet data periods (Jun 23 at 10:27 AM)
+S1126 Diagnose why NowAI leaderboard sync hasn't been running for a week; implement observability improvement to confirm automation health during stable-data periods (Jun 23 at 10:29 AM)
+S1129 Confirmed logging mechanism for sync checks — verifying that automated sync runs are being captured and persisted (Jun 23 at 10:30 AM)
+S1130 Update leaderboard sync skill documentation — clarify notification behavior (Friday heartbeat) and add logging guidance (Jun 23 at 2:03 PM)
+2219 2:04p 🔵 Leaderboard Auto-Sync Architecture — Approved Design Specification
+2220 " 🔵 Leaderboard Sync Skill — Operational Implementation & Guardrails
+S1131 Understand and improve leaderboard sync logging and operational tooling — document behavior and create convenience shortcuts (Jun 23 at 2:04 PM)
+2221 2:05p ✅ Makefile for Leaderboard Sync — Convenience Targets Added
+S1132 Commit and push Makefile and SKILL.md documentation updates to staging and production (Jun 23 at 2:05 PM)
+2222 2:14p ✅ Makefile and SKILL.md committed with launchd sync documentation
+2223 " ✅ Makefile and SKILL.md deployed to staging and production
+S1154 Extend NowAI-Bench quarterly header sync: add "Last Updated" text detection and update logic to the leaderboard sync process, with Teams notification indicating if that's the only change (Jun 23 at 2:15 PM)
+### Jun 24, 2026
+2264 12:02p 🔵 Located "Last Updated" Header in NowAI-Bench UI and Identified Sync Script Gap
+2265 " 🔵 Located Quarter Indicator and "Last Updated" Label in index.html
+2266 " 🔵 Identified Data Source and Existing Scripts in Project Structure
+2267 12:03p 🔵 Located Existing NowAI Sync Infrastructure and Teams Notification Module
+2268 " 🔵 Existing Sync Workflow: 7-Step Process with Change Detection and Teams Notifications
+2269 " 🔵 Located Quarterly Header Markup and Transformation Module
+S1172 Implement quarter label detection and application in sync.mjs with distinct handling for quarter-only vs. leaderboard-only changes (Jun 24 at 12:05 PM)
+2273 2:28p ⚖️ Quarter label detection implementation approach for sync.mjs
+2274 " ✅ Added synchronous file I/O imports to sync.mjs
+2275 2:29p 🟣 Implemented quarter label detection and application helpers
+2276 " 🔄 Restructured main() to support independent quarter label and leaderboard syncs
+2277 2:30p 🔵 sync.mjs syntax and runtime validation successful
+2278 " 🔵 Quarter label detection implementation task completed
+2279 2:33p ✅ Quarter label detection feature committed to repository
+2280 2:35p ✅ Quarter label detection feature deployed to staging HuggingFace Space
+2281 " ✅ Quarter label detection feature deployed to production HuggingFace Space
+S1173 Implement quarter label detection and auto-update in NowAI-Bench leaderboard sync with distinct notifications for quarter-only vs. leaderboard-only changes (Jun 24 at 2:36 PM)
+**Investigated**: Examined sync.mjs orchestration flow (fetch → build → validate → commit → push), identified quarter label location in index.html regex pattern, reviewed staging/prod deployment architecture with two separate HF Space remotes
+
+**Learned**: Sync workflow has three independent concerns that must be orchestrated separately: upstream data fetching, leaderboard validation gates, and UI header updates. Sanity validation gates should apply only to leaderboard data changes, allowing quarter-only updates to bypass validation. Teams notifications must signal what changed (rankings, header, or both) to enable appropriate reviewer workflows. The staging/prod split uses force-push and metadata patching on staging to isolate metadata changes from code.
+
+**Completed**: Added synchronous file I/O imports (readFileSync, writeFileSync); implemented three helper functions (quarterForDate, detectQuarterLabel, applyQuarterLabel) with regex-based quarter label extraction and in-place HTML patching; restructured main() orchestration with step 3.5 quarter detection, independent change flags (leaderboardChanged and quarterInfo.needsUpdate), conditional validation/writes/commits; implemented dynamic commit messages and differentiated Teams notification titles (📅 for quarter-only, ✅ for rankings); validated via syntax check and smoke test; committed to main (SHA 5661ed6); deployed to both staging and production HuggingFace Spaces
+
+**Next Steps**: Feature is complete and live in both environments. The quarter label auto-update will activate automatically next time the leaderboard sync script runs after Q3 begins (July 1). No further development work planned in this session.
+
+
+Access 128k tokens of past work via get_observations([IDs]) or mem-search skill.
+</claude-mem-context>