Add model card, config, and loader for tinystories-50m

1372fb6 verified 4 days ago

2.86 kB

license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
  - tiny
  - tiny-lm
  - tiny-model
  - slm
  - SLM
  - small-language-model
  - from-scratch
  - tinystories
metrics:
  - perplexity
  - val-loss

tinystories-50m

A 54,804,992-parameter from-scratch BPE GPT trained on TinyStories. This is the scale-up of my tinystories-24m — same narrow children's-story domain, ~2.3x the parameters.

What it is

Architecture: custom GPT — 16 layers, d=512, 8 heads, FFN 2048, seq 512, vocab 8192 (BPE). Weight-tied embeddings, RMSNorm, GELU, fused qkv. Not a stock transformers architecture — load it with the included model_loader.py.
Data: 447.9M TinyStories tokens, 1 full epoch (13,668 steps, batch 64, seq 512).
Compute: RTX 5090 (32 GB), ~8 steps/s, ~28 min.
Params: 54,804,992 exactly (verified against the safetensors header: 99 tensors, weight-tied so tok.weight is not double-counted).

Training

LR 3e-4, cosine decay, warmup 500. No divergence (unlike the 24m at 6e-4).
Best val loss 1.6566 @ step 13,250; final 1.6371 @ step 13,668.
In-domain perplexity (held-out TinyStories): ~5.24 (best ckpt).
~8.2 tokens/param.

Quality (honest)

Coherent, on-theme children's stories with proper punctuation and narrative structure. Sample (seed 0):

Once upon a time, there was a little girl named Lily. She loved to play in the garden with her dog, Buddy. One day, Lily's mom taught her a new game called wash. Lily was very excited and asked her mom if she could bring her favorite toy back. Lily went outside and filled a bucket with water. As she was catching the bucket, she noticed that it was wet and her clothes were wet. She quickly dried herself off and her clothes were all wet. She ran inside and asked her mom for help.

Occasional artifacts are expected at this scale on a narrow domain: a garbled word here and there, pronoun slips, and short-range repetition. It is a story generator, not a general model — do not expect general English, reasoning, or instruction following.

What it is NOT good at

General-domain text (out of distribution; expect degradation).
Reasoning, math, multi-turn dialogue, instruction following.
Long coherence beyond ~140 generated tokens (trained at seq 512 but quality degrades with length).

Usage

from model_loader import load, generate
model, tok = load(".")          # needs torch, safetensors, tokenizers
print(generate(model, tok, "Once upon a time, there was a little girl named Lily.", seed=0))

Lineage

tinystories-24m (24.59M, coherent at 18.2 tok/param) → tinystories-50m (54.80M, 8.2 tok/param).