Update README.md
Browse files
README.md
CHANGED
|
@@ -15,10 +15,10 @@ tags:
|
|
| 15 |
- hybrid
|
| 16 |
---
|
| 17 |
|
| 18 |
-
# Pebble-10M
|
| 19 |

|
| 20 |
|
| 21 |
-
Pebble-10M
|
| 22 |
|
| 23 |
## Model Details
|
| 24 |
|
|
@@ -48,7 +48,7 @@ The model was trained on a 25B token subset of the following datasets:
|
|
| 48 |
|
| 49 |
## Benchmarks
|
| 50 |
|
| 51 |
-
Pebble-10M
|
| 52 |
|
| 53 |
| Benchmark | Accuracy | Random Baseline |
|
| 54 |
|-----------|----------|------------------|
|
|
@@ -84,7 +84,7 @@ Here is a simple Python script to load the model and generate text interactively
|
|
| 84 |
import torch
|
| 85 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 86 |
|
| 87 |
-
MODEL_ID = "basically-ai/Pebble-10M
|
| 88 |
|
| 89 |
def main():
|
| 90 |
print("Loading Pebble 10M...")
|
|
|
|
| 15 |
- hybrid
|
| 16 |
---
|
| 17 |
|
| 18 |
+
# Pebble-10M
|
| 19 |

|
| 20 |
|
| 21 |
+
Pebble-10M is a compact, hybrid autoregressive language model. It combines the efficiency of state-space models with the proven performance of attention layers, optimized using a custom Muon + AdamW optimizer split.
|
| 22 |
|
| 23 |
## Model Details
|
| 24 |
|
|
|
|
| 48 |
|
| 49 |
## Benchmarks
|
| 50 |
|
| 51 |
+
Pebble-10M performs above random chance on several commonsense and arithmetic benchmarks.
|
| 52 |
|
| 53 |
| Benchmark | Accuracy | Random Baseline |
|
| 54 |
|-----------|----------|------------------|
|
|
|
|
| 84 |
import torch
|
| 85 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 86 |
|
| 87 |
+
MODEL_ID = "basically-ai/Pebble-10M"
|
| 88 |
|
| 89 |
def main():
|
| 90 |
print("Loading Pebble 10M...")
|