Add standalone model card
Browse files
README.md
CHANGED
|
@@ -26,13 +26,14 @@ Evaluated across all 231 public tasks in [fstandhartinger/jevbench](https://gith
|
|
| 26 |
|
| 27 |
| Tier | Tasks | Accuracy | ECE | Brier Score | Median Latency (p50) |
|
| 28 |
|---|---|---|---|---|---|
|
| 29 |
-
| **Easy** | 48 | **
|
| 30 |
-
| **Original** | 72 | **
|
| 31 |
-
| **Hard** | 111 | **
|
| 32 |
-
| **Overall Aggregate** | **231** | **
|
| 33 |
|
| 34 |
-
- **Easy Tier Sub-Breakdown:** Choice accuracy: **
|
| 35 |
-
- **Original Tier Sub-Breakdown:** Choice accuracy: **
|
|
|
|
| 36 |
|
| 37 |
### 2. RLCDAlignBench Alignment & Safety Evaluation (100 Instances)
|
| 38 |
|
|
@@ -40,13 +41,14 @@ Evaluated across the 10 core AI alignment failure modes (arXiv:2609.29429):
|
|
| 40 |
|
| 41 |
| Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
|
| 42 |
|---|---|---|---|---|---|
|
| 43 |
-
| **
|
| 44 |
-
| **Honesty (Deception)** | 11 | **0.
|
| 45 |
-
| **
|
| 46 |
-
| **
|
| 47 |
-
| **
|
| 48 |
-
| **
|
| 49 |
-
| **
|
|
|
|
| 50 |
|
| 51 |
### 3. Edge vs Cloud Latency (ClassOne vs TypeSafe Jev API)
|
| 52 |
|
|
|
|
| 26 |
|
| 27 |
| Tier | Tasks | Accuracy | ECE | Brier Score | Median Latency (p50) |
|
| 28 |
|---|---|---|---|---|---|
|
| 29 |
+
| **Easy** | 48 | **93.8%** (45/48) | 0.0610 | 0.0516 | **45.0 ms** |
|
| 30 |
+
| **Original** | 72 | **63.9%** (46/72) | 0.2510 | 0.2499 | **43.1 ms** |
|
| 31 |
+
| **Hard** | 111 | **37.8%** (42/111) | 0.4299 | 0.4022 | **91.3 ms** |
|
| 32 |
+
| **Overall Aggregate** | **231** | **57.6%** (133/231) | — | — | **~44 ms** |
|
| 33 |
|
| 34 |
+
- **Easy Tier Sub-Breakdown:** Choice accuracy: **100.0%** (36/36); Noul policy accuracy: **75.0%** (9/12).
|
| 35 |
+
- **Original Tier Sub-Breakdown:** Choice accuracy: **66.7%** (24/36); Score rubrics: **66.7%** (8/12); Noul accuracy: **58.3%** (14/24).
|
| 36 |
+
- **Hard Tier Sub-Breakdown:** Noul policy compliance: **44.7%** (17/38); Choice accuracy: **34.3%** (23/67); Score rubrics: **33.3%** (2/6).
|
| 37 |
|
| 38 |
### 2. RLCDAlignBench Alignment & Safety Evaluation (100 Instances)
|
| 39 |
|
|
|
|
| 41 |
|
| 42 |
| Failure Mode / Axis | Samples (N) | AUROC | Accuracy (%) | ECE | Latency (p50) |
|
| 43 |
|---|---|---|---|---|---|
|
| 44 |
+
| **Concealing Uncertainty** | 14 | **0.980** | **85.7%** | **0.0718** | 130.2 ms |
|
| 45 |
+
| **Honesty (Deception)** | 11 | **0.800** | **72.7%** | 0.2445 | 219.6 ms |
|
| 46 |
+
| **Refusal (Jailbreaks)** | 11 | **0.667** | **54.5%** | 0.1934 | 271.1 ms |
|
| 47 |
+
| **Power Seeking** | 6 | **0.556** | 50.0% | 0.1794 | 213.2 ms |
|
| 48 |
+
| **Reward Hacking** | 9 | **0.500** | 33.3% | 0.2935 | 209.6 ms |
|
| 49 |
+
| **Prompt Injection** | 8 | **0.500** | 37.5% | 0.3207 | 167.9 ms |
|
| 50 |
+
| **Bias** | 9 | 0.375 | 55.6% | 0.1659 | 221.5 ms |
|
| 51 |
+
| **Overall Average** | **100** | **0.516** | **51.0%** | **0.1584** | **200.0 ms** |
|
| 52 |
|
| 53 |
### 3. Edge vs Cloud Latency (ClassOne vs TypeSafe Jev API)
|
| 54 |
|