File size: 3,468 Bytes
a1fe5be | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | ---
license: cdla-sharing-1.0
datasets:
- roneneldan/TinyStories
- roneneldan/TinyStoriesInstruct
language:
- en
pipeline_tag: text-generation
tags:
- llama
- tiny
- educational
- gguf
---
# tinyllm — instruction-tuned
A 15.7M-parameter Llama-architecture language model trained from
random initialization on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).
Built as a complete walk through the model lifecycle — tokenizer, architecture,
pretraining, evaluation, instruction tuning, packaging, quantization, and local
serving. It is small enough to train in about 45 minutes on a free Colab T4 and
to run on a 2-core laptop CPU with no GPU.
## Architecture
| | |
|---|---|
| Parameters | 15,735,168 (12,589,440 non-embedding) |
| Layers | 8 |
| Hidden size | 384 |
| Attention heads | 6 query / 2 key-value (GQA) |
| Head dim | 64 |
| MLP | SwiGLU, intermediate 1024 |
| Normalization | RMSNorm (eps 1e-05) |
| Position encoding | RoPE (theta 10000) |
| Context length | 512 |
| Vocabulary | 8192 (SentencePiece BPE, byte fallback) |
| Embeddings | tied input/output |
## Training
| | |
|---|---|
| Tokens | 164M (~10 per parameter) |
| Steps | 2,500 at 65,536 tokens/step |
| Optimizer | AdamW (betas 0.9/0.95, wd 0.1 on matrices only) |
| Schedule | cosine, 200 warmup steps, peak LR 0.0006 |
| Precision | fp16 AMP with loss scaling |
| Hardware | 1x NVIDIA T4 (Colab free tier) |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("pythonstudentiam/tinyllm")
model = AutoModelForCausalLM.from_pretrained("pythonstudentiam/tinyllm")
messages = [{"role": "user", "content": "Write a story about a lost puppy."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=250, do_sample=True, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))
```
### With llama.cpp
GGUF conversions are included in this repo.
```bash
llama-server -m tinyllm-Q8_0.gguf -c 512 --host 127.0.0.1 --port 8080
```
## Limitations
This model has 15.7M parameters and a 8192-token vocabulary,
trained exclusively on synthetic children's stories. Be concrete about what that means:
- **It only does one thing.** It writes simple short stories in the TinyStories
style. Anything else — code, arithmetic, factual questions, translation,
summarization of arbitrary text — produces confident nonsense.
- **Its vocabulary is small.** Words outside a children's-story vocabulary fall
back to individual bytes, which it handles poorly.
- **Context is 512 tokens.** There is no long-range coherence to be had.
- **No safety tuning of any kind.** It has had no alignment work beyond
instruction tuning on story prompts.
- **Quantization hurts more than usual.** Small models have less parameter
redundancy to absorb rounding error; Q4_K_M is measurably worse here than the
usual "negligible loss" guidance for 7B+ models would suggest.
Not suitable for any production use. It is a teaching artifact.
## Training data
[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) — synthetic
short stories generated by GPT-3.5/GPT-4, constrained to the vocabulary of a
3-4 year old. Licensed CDLA-Sharing-1.0.
Instruction tuning used [TinyStoriesInstruct](https://huggingface.co/datasets/roneneldan/TinyStoriesInstruct).
|