README.md

---
license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
  - tiny
  - tiny-lm
  - tiny-model
  - slm
  - small-language-model
  - from-scratch
  - tinystories
  - gpt
  - bpe
datasets:
  - ronendagan/TinyStories
metrics:
  - perplexity
---

# tinystories-50m

A **54,804,992-parameter** transformer language model trained **from scratch** on
[TinyStories](https://huggingface.co/datasets/ronendagan/TinyStories), a corpus of
simple, repetitive children's stories. It is the 50M scale-up in the
`tinystories-24m` → `tinystories-50m` lineage (the 24M sibling was coherent at
18.2 tok/param; this one trains at 8.17 tok/param on the same narrow domain).

It writes fluent, on-domain children's stories. It is **not** a general
language model — out-of-domain generation degrades, and it should not be used
for anything beyond the story domain it was trained on.

## Architecture

| Field | Value |
|---|---|
| Parameters | **54,804,992** (exact; verified against the safetensors header) |
| Layers (L) | 16 |
| d_model (D) | 512 |
| Heads (H) | 8 (head dim 64) |
| FFN dim | 2048 (4× D) |
| Vocab | 8192 (BPE) |
| Max seq len | 512 |
| Embeddings | **weight-tied** (lm_head = tok) |
| Norm | RMSNorm (pre-norm, 2 per block + final) |
| Activation | GELU |
| Attention | causal, no bias in linear layers |
| Dtype | float32 |

Parameter breakdown (sums exactly to 54,804,992):

- token embedding: 8192 × 512 = 4,194,304
- position embedding: 512 × 512 = 262,144
- 16 blocks × 3,146,752 = 50,348,032
  - 2 × RMSNorm (512) + qkv (512×1536) + proj (512×512) + fc1 (512×2048) + fc2 (2048×512)
- final RMSNorm: 512

## Training

- **Data:** TinyStories (ronendagan/TinyStories), ~447.86M tokens after BPE-8192
  re-tokenization, 8.17 tokens/param.
- **Optimizer:** AdamW, cosine LR decay with warmup (peak 6e-4).
- **Batch:** 64, seq 512 → 32,768 tokens/step.
- **Steps:** 13,668 (one full epoch). Best checkpoint at step 13,250.
- **Hardware:** single NVIDIA RTX 5090 (32 GB), peak ~15 GB.
- **Final val loss:** 1.6371 (best ckpt 1.6566 @ step 13,250).

## Evaluated numbers

- **Held-out perplexity (TinyStories val split):** **5.24** (best ckpt,
  val cross-entropy 1.6566 → exp = 5.2412). This is the honest metric for a
  narrow-domain model; standard general benchmarks (BLiMP/ARC/PIQA) are not
  meaningful here and are deliberately not reported.
- **Coherence:** 9/9 seeded generations (3 story-start prompts × 3 seeds) are
  fluent, on-domain, with consistent characters and correct punctuation. Minor
  artifacts expected at this scale (occasional garbled quote char, a couple of
  logical slips).

## Files

| File | What |
|---|---|
| `model.safetensors` | weights (210 MB, 99 tensors, float32) |
| `tokenizer.json` | BPE-8192 tokenizer (`tokenizers` format) |
| `config.json` | architecture config |
| `load_model.py` | self-contained loader + `TinyStoriesGPT` class |

## Usage

```python
from load_model import load
model, tok = load()
ids = tok.encode("Once upon a time,")
out = model.generate(torch.tensor([ids]).cuda(), 100, temp=0.8, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))
```

## What it is and is not

- **Is:** a small, from-scratch, on-domain story generator. Good for studying
  how a ~55M transformer learns a narrow, repetitive domain.
- **Is not:** a general-purpose LM. Do not expect coherent output on code,
  math, or open-domain text. The low perplexity is domain-specific.