| --- |
| license: cdla-sharing-1.0 |
| datasets: |
| - roneneldan/TinyStories |
| - roneneldan/TinyStoriesInstruct |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - llama |
| - tiny |
| - educational |
| - gguf |
| --- |
| |
| # tinyllm — instruction-tuned |
|
|
| A 15.7M-parameter Llama-architecture language model trained from |
| random initialization on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). |
|
|
| Built as a complete walk through the model lifecycle — tokenizer, architecture, |
| pretraining, evaluation, instruction tuning, packaging, quantization, and local |
| serving. It is small enough to train in about 45 minutes on a free Colab T4 and |
| to run on a 2-core laptop CPU with no GPU. |
|
|
| ## Architecture |
|
|
| | | | |
| |---|---| |
| | Parameters | 15,735,168 (12,589,440 non-embedding) | |
| | Layers | 8 | |
| | Hidden size | 384 | |
| | Attention heads | 6 query / 2 key-value (GQA) | |
| | Head dim | 64 | |
| | MLP | SwiGLU, intermediate 1024 | |
| | Normalization | RMSNorm (eps 1e-05) | |
| | Position encoding | RoPE (theta 10000) | |
| | Context length | 512 | |
| | Vocabulary | 8192 (SentencePiece BPE, byte fallback) | |
| | Embeddings | tied input/output | |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Tokens | 164M (~10 per parameter) | |
| | Steps | 2,500 at 65,536 tokens/step | |
| | Optimizer | AdamW (betas 0.9/0.95, wd 0.1 on matrices only) | |
| | Schedule | cosine, 200 warmup steps, peak LR 0.0006 | |
| | Precision | fp16 AMP with loss scaling | |
| | Hardware | 1x NVIDIA T4 (Colab free tier) | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("pythonstudentiam/tinyllm") |
| model = AutoModelForCausalLM.from_pretrained("pythonstudentiam/tinyllm") |
| |
| messages = [{"role": "user", "content": "Write a story about a lost puppy."}] |
| prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| ids = tok(prompt, return_tensors="pt") |
| out = model.generate(**ids, max_new_tokens=250, do_sample=True, temperature=0.8) |
| print(tok.decode(out[0], skip_special_tokens=True)) |
| ``` |
|
|
| ### With llama.cpp |
|
|
| GGUF conversions are included in this repo. |
|
|
| ```bash |
| llama-server -m tinyllm-Q8_0.gguf -c 512 --host 127.0.0.1 --port 8080 |
| ``` |
|
|
| ## Limitations |
|
|
| This model has 15.7M parameters and a 8192-token vocabulary, |
| trained exclusively on synthetic children's stories. Be concrete about what that means: |
|
|
| - **It only does one thing.** It writes simple short stories in the TinyStories |
| style. Anything else — code, arithmetic, factual questions, translation, |
| summarization of arbitrary text — produces confident nonsense. |
| - **Its vocabulary is small.** Words outside a children's-story vocabulary fall |
| back to individual bytes, which it handles poorly. |
| - **Context is 512 tokens.** There is no long-range coherence to be had. |
| - **No safety tuning of any kind.** It has had no alignment work beyond |
| instruction tuning on story prompts. |
| - **Quantization hurts more than usual.** Small models have less parameter |
| redundancy to absorb rounding error; Q4_K_M is measurably worse here than the |
| usual "negligible loss" guidance for 7B+ models would suggest. |
|
|
| Not suitable for any production use. It is a teaching artifact. |
|
|
| ## Training data |
|
|
| [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) — synthetic |
| short stories generated by GPT-3.5/GPT-4, constrained to the vocabulary of a |
| 3-4 year old. Licensed CDLA-Sharing-1.0. |
|
|
| Instruction tuning used [TinyStoriesInstruct](https://huggingface.co/datasets/roneneldan/TinyStoriesInstruct). |
|
|
|
|
