Update README.md
Browse files
README.md
CHANGED
|
@@ -15,6 +15,24 @@ We built this model to be a small, useful foundation for various tasks. It's a g
|
|
| 15 |
|
| 16 |
**Model Series Note:** This is the first model in our Plyx series. We're continuing this work and plan to release future models in various sizes. We'll be adding some initial performance benchmarks here soon.
|
| 17 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
## Pre-training Data
|
| 19 |
|
| 20 |
The model was trained on a carefully curated mix of data to build a great foundation, trained on approx ~600M tokens:
|
|
|
|
| 15 |
|
| 16 |
**Model Series Note:** This is the first model in our Plyx series. We're continuing this work and plan to release future models in various sizes. We'll be adding some initial performance benchmarks here soon.
|
| 17 |
|
| 18 |
+
## Model Benchmarks
|
| 19 |
+
|
| 20 |
+
We evaluated the model on zero-shot multiple-choice tasks using a mathematically standardized log-likelihood evaluation harness (`lm-evaluation-harness`).
|
| 21 |
+
|
| 22 |
+
Below are the verified scores across both raw accuracy (`acc`) and length-normalized accuracy (`acc_norm`) on full test/validation splits:
|
| 23 |
+
|
| 24 |
+
| Benchmark Task | Metric | Value | Standard Error |
|
| 25 |
+
| :--- | :---: | :---: | :---: |
|
| 26 |
+
| **MMLU** | `acc` | 22.95% | ±0.35% |
|
| 27 |
+
| **HellaSwag** | `acc` <br> `acc_norm` | 25.70% <br> 25.76% | ±0.44% <br> ±0.44% |
|
| 28 |
+
| **ARC-Easy** | `acc` <br> `acc_norm` | 30.85% <br> 30.85% | ±0.95% <br> ±0.95% |
|
| 29 |
+
| **ARC-Challenge** | `acc` <br> `acc_norm` | 19.20% <br> 21.33% | ±1.15% <br> ±1.20% |
|
| 30 |
+
| **PIQA** | `acc` <br> `acc_norm` | 55.33% <br> 52.94% | ±1.16% <br> ±1.16% |
|
| 31 |
+
| **SciQ** | `acc` <br> `acc_norm` | 48.50% <br> 43.80% | ±1.58% <br> ±1.57% |
|
| 32 |
+
| **WinoGrande** | `acc` | 51.22% | ±1.40% |
|
| 33 |
+
| **OpenBookQA** | `acc` <br> `acc_norm` | 13.20% <br> 25.80% | ±1.52% <br> ±1.96% |
|
| 34 |
+
| **ArithMark-2.0** | `acc` | 25.32% | ±0.87% |
|
| 35 |
+
|
| 36 |
## Pre-training Data
|
| 37 |
|
| 38 |
The model was trained on a carefully curated mix of data to build a great foundation, trained on approx ~600M tokens:
|