README.md

metadata

license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
  - tiny
  - tiny-lm
  - tiny-model
  - slm
  - small-language-model
  - from-scratch
  - tinystories
  - gpt
  - bpe
datasets:
  - ronendagan/TinyStories
metrics:
  - perplexity

tinystories-50m

A 54,804,992-parameter transformer language model trained from scratch on TinyStories, a corpus of simple, repetitive children's stories. It is the 50M scale-up in the tinystories-24m → tinystories-50m lineage (the 24M sibling was coherent at 18.2 tok/param; this one trains at 8.17 tok/param on the same narrow domain).

It writes fluent, on-domain children's stories. It is not a general language model — out-of-domain generation degrades, and it should not be used for anything beyond the story domain it was trained on.

Architecture

Field	Value
Parameters	54,804,992 (exact; verified against the safetensors header)
Layers (L)	16
d_model (D)	512
Heads (H)	8 (head dim 64)
FFN dim	2048 (4× D)
Vocab	8192 (BPE)
Max seq len	512
Embeddings	weight-tied (lm_head = tok)
Norm	RMSNorm (pre-norm, 2 per block + final)
Activation	GELU
Attention	causal, no bias in linear layers
Dtype	float32

Parameter breakdown (sums exactly to 54,804,992):

• token embedding: 8192 × 512 = 4,194,304
• position embedding: 512 × 512 = 262,144
• 16 blocks × 3,146,752 = 50,348,032
  • 2 × RMSNorm (512) + qkv (512×1536) + proj (512×512) + fc1 (512×2048) + fc2 (2048×512)
• final RMSNorm: 512

Training

• Data: TinyStories (ronendagan/TinyStories), ~447.86M tokens after BPE-8192 re-tokenization, 8.17 tokens/param.
• Optimizer: AdamW, cosine LR decay with warmup (peak 6e-4).
• Batch: 64, seq 512 → 32,768 tokens/step.
• Steps: 13,668 (one full epoch). Best checkpoint at step 13,250.
• Hardware: single NVIDIA RTX 5090 (32 GB), peak ~15 GB.
• Final val loss: 1.6371 (best ckpt 1.6566 @ step 13,250).

Evaluated numbers

• Held-out perplexity (TinyStories val split): 5.24 (best ckpt, val cross-entropy 1.6566 → exp = 5.2412). This is the honest metric for a narrow-domain model; standard general benchmarks (BLiMP/ARC/PIQA) are not meaningful here and are deliberately not reported.
• Coherence: 9/9 seeded generations (3 story-start prompts × 3 seeds) are fluent, on-domain, with consistent characters and correct punctuation. Minor artifacts expected at this scale (occasional garbled quote char, a couple of logical slips).

Files

File	What
model.safetensors	weights (210 MB, 99 tensors, float32)
tokenizer.json	BPE-8192 tokenizer (tokenizers format)
config.json	architecture config
load_model.py	self-contained loader + TinyStoriesGPT class

Usage

from load_model import load
model, tok = load()
ids = tok.encode("Once upon a time,")
out = model.generate(torch.tensor([ids]).cuda(), 100, temp=0.8, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

What it is and is not

• Is: a small, from-scratch, on-domain story generator. Good for studying how a ~55M transformer learns a narrow, repetitive domain.
• Is not: a general-purpose LM. Do not expect coherent output on code, math, or open-domain text. The low perplexity is domain-specific.