---
language: en
license: mit
tags:
  - pretrained
  - causal-lm
  - fineweb-edu
  - custom-architecture
---

# tiny-edu-166m (ParchmentLM)

A 166M parameter transformer pretrained from scratch on 4B tokens of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).

## Architecture (ParchmentLM)

Custom decoder-only transformer:
- **Parameters:** 166M
- **Layers:** 12
- **Hidden size:** 768
- **Attention heads:** 12
- **FFN:** SwiGLU (hidden=2048)
- **Context length:** 1024
- **Positional encoding:** RoPE (base=10000)
- **Normalization:** RMSNorm
- **Tokenizer:** cl100k_base (100277 tokens)

## Training

- **Dataset:** FineWeb-Edu 10BT sample
- **Tokens seen:** ~4B
- **Steps:** 30,000
- **Optimizer:** AdamW (lr=3e-4, cosine decay to 3e-5)
- **Hardware:** Single A100 80GB

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("SlitherCode/tiny-edu-166m", trust_remote_code=True)
model     = AutoModelForCausalLM.from_pretrained("SlitherCode/tiny-edu-166m", trust_remote_code=True)

inputs = tokenizer("The history of mathematics", return_tensors="pt")
out    = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

## License

Model weights: MIT. Training data: ODC-By 1.0.