Kearm commited on
Commit
efdcdac
·
verified ·
1 Parent(s): f878a0f

Upload BENCHMARK_SYSTEM_CARD.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. BENCHMARK_SYSTEM_CARD.md +104 -0
BENCHMARK_SYSTEM_CARD.md ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # RESMP Benchmark Instrument
2
+
3
+ **Release:** Rev 01
4
+ **Type:** Zero-dependency Web Component for grouped evaluation charts
5
+ **Status:** Experimental, usable, and versioned with the `RESMP-DEV/README` Space
6
+
7
+ ## What it does
8
+
9
+ The RESMP Benchmark Instrument renders a matrix of model or system evaluations as grouped small-multiple bar charts. It is designed for reports where readers need to compare the same ordered series across many tasks without relearning colors, scale, or layout in each panel.
10
+
11
+ It accepts local JSON, performs no runtime data fetch, and produces:
12
+
13
+ - a responsive browser view;
14
+ - a print-clean report;
15
+ - direct numeric labels on every available observation;
16
+ - explicit `N/A` marks for missing observations;
17
+ - a browser SVG export of the full composition.
18
+
19
+ ## Intended uses
20
+
21
+ - model, quantization, runtime, and hardware comparisons;
22
+ - grouped benchmark families such as STEM, instruction following, tool use, and agentic tasks;
23
+ - matched-control reports where fixed series identity matters;
24
+ - static Hugging Face, GitHub Pages, or local evaluation reports.
25
+
26
+ ## Non-goals
27
+
28
+ This component does not calculate metrics, retrieve benchmark results, establish statistical significance, or decide whether comparisons are methodologically valid. It visualizes values supplied by the publisher. A polished chart is not evidence that the underlying experiment was controlled.
29
+
30
+ ## Input contract
31
+
32
+ Inputs must conform to [`benchmark.schema.json`](benchmark.schema.json). The core structure is:
33
+
34
+ ```json
35
+ {
36
+ "title": "MODEL SYSTEM EVALUATIONS",
37
+ "subtitle": "PINNED RUN / MATCHED CONTROL",
38
+ "higherIsBetter": true,
39
+ "series": [
40
+ {"id": "candidate", "label": "CANDIDATE", "color": "#6f36d9"}
41
+ ],
42
+ "groups": [
43
+ {
44
+ "name": "STEM",
45
+ "panels": [
46
+ {"name": "MATH EXACT", "values": {"candidate": 62.4}}
47
+ ]
48
+ }
49
+ ]
50
+ }
51
+ ```
52
+
53
+ Series IDs must be unique. Values must be finite numbers or `null`. Missing keys and `null` render as `N/A`; they are never silently converted to zero.
54
+
55
+ ## Rendering semantics
56
+
57
+ - Series ordering is global and follows the `series` array.
58
+ - The numeric scale is shared across the complete chart, not recomputed per panel.
59
+ - Zero is always included in the domain.
60
+ - Negative and positive values share a visible zero baseline.
61
+ - `higherIsBetter` controls the displayed interpretation label only; it does not transform values.
62
+ - Values are displayed to two decimal places in Rev 01.
63
+
64
+ ## Reproducibility guidance
65
+
66
+ The chart JSON should be stored beside the evaluation receipt. For publishable results, record:
67
+
68
+ - model and dataset revisions;
69
+ - runtime, hardware, precision, and kernel versions;
70
+ - prompts, sampling settings, and evaluator versions;
71
+ - seeds or nondeterminism boundaries;
72
+ - raw result artifact hashes;
73
+ - aggregation code and any excluded samples.
74
+
75
+ The demonstration values in [`benchmark.example.json`](benchmark.example.json) are synthetic layout data, not claimed benchmark results.
76
+
77
+ ## Accessibility
78
+
79
+ Every panel is emitted as an SVG with an accessible benchmark name. Direct values reduce dependence on color alone, and series order remains fixed. Rev 01 does not yet emit a full hidden data table, keyboard chart navigation, configurable patterns, or automatic contrast validation. Those are known gaps.
80
+
81
+ ## Export behavior
82
+
83
+ The Export SVG action creates a self-contained SVG containing the browser-rendered HTML/CSS inside `<foreignObject>`. It is suitable for browsers and Chromium-based capture workflows. Some vector editors and non-browser SVG pipelines do not fully support `<foreignObject>`; use browser print-to-PDF or a screenshot pipeline for those consumers.
84
+
85
+ ## Limitations
86
+
87
+ - One global scale can compress small differences when metric ranges vary widely.
88
+ - Rev 01 has no confidence intervals, error bars, sample counts, or significance annotations.
89
+ - Long series or panel labels may require custom CSS.
90
+ - Color selection is publisher-controlled and not automatically checked for accessibility.
91
+ - The component trusts already-sanitized text after escaping display fields; do not inject arbitrary HTML through config values.
92
+ - It does not validate experimental methodology.
93
+
94
+ ## Files
95
+
96
+ - [`benchmark-chart.js`](benchmark-chart.js) — Web Component and export logic
97
+ - [`benchmark-component.css`](benchmark-component.css) — portable chart styling
98
+ - [`benchmark.schema.json`](benchmark.schema.json) — machine-readable input schema
99
+ - [`benchmark.example.json`](benchmark.example.json) — canonical example input
100
+ - [`benchmark.html`](benchmark.html) — full reference presentation
101
+
102
+ ## License and reuse
103
+
104
+ The component is published for reuse with RESMP.DEV releases. Preserve result provenance and do not use its visual consistency to imply experimental comparability where none exists.