tinyllm — instruction-tuned

A 15.7M-parameter Llama-architecture language model trained from random initialization on TinyStories.

Built as a complete walk through the model lifecycle — tokenizer, architecture, pretraining, evaluation, instruction tuning, packaging, quantization, and local serving. It is small enough to train in about 45 minutes on a free Colab T4 and to run on a 2-core laptop CPU with no GPU.

Architecture

Parameters	15,735,168 (12,589,440 non-embedding)
Layers	8
Hidden size	384
Attention heads	6 query / 2 key-value (GQA)
Head dim	64
MLP	SwiGLU, intermediate 1024
Normalization	RMSNorm (eps 1e-05)
Position encoding	RoPE (theta 10000)
Context length	512
Vocabulary	8192 (SentencePiece BPE, byte fallback)
Embeddings	tied input/output

Training

Tokens	164M (~10 per parameter)
Steps	2,500 at 65,536 tokens/step
Optimizer	AdamW (betas 0.9/0.95, wd 0.1 on matrices only)
Schedule	cosine, 200 warmup steps, peak LR 0.0006
Precision	fp16 AMP with loss scaling
Hardware	1x NVIDIA T4 (Colab free tier)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("pythonstudentiam/tinyllm")
model = AutoModelForCausalLM.from_pretrained("pythonstudentiam/tinyllm")

messages = [{"role": "user", "content": "Write a story about a lost puppy."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=250, do_sample=True, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

With llama.cpp

GGUF conversions are included in this repo.

llama-server -m tinyllm-Q8_0.gguf -c 512 --host 127.0.0.1 --port 8080

Limitations

This model has 15.7M parameters and a 8192-token vocabulary, trained exclusively on synthetic children's stories. Be concrete about what that means:

• It only does one thing. It writes simple short stories in the TinyStories style. Anything else — code, arithmetic, factual questions, translation, summarization of arbitrary text — produces confident nonsense.
• Its vocabulary is small. Words outside a children's-story vocabulary fall back to individual bytes, which it handles poorly.
• Context is 512 tokens. There is no long-range coherence to be had.
• No safety tuning of any kind. It has had no alignment work beyond instruction tuning on story prompts.
• Quantization hurts more than usual. Small models have less parameter redundancy to absorb rounding error; Q4_K_M is measurably worse here than the usual "negligible loss" guidance for 7B+ models would suggest.

Not suitable for any production use. It is a teaching artifact.

Training data

TinyStories — synthetic short stories generated by GPT-3.5/GPT-4, constrained to the vocabulary of a 3-4 year old. Licensed CDLA-Sharing-1.0.

Instruction tuning used TinyStoriesInstruct.