---
license: cdla-sharing-1.0
datasets:
  - roneneldan/TinyStories
  - roneneldan/TinyStoriesInstruct
language:
  - en
pipeline_tag: text-generation
tags:
  - llama
  - tiny
  - educational
  - gguf
---

# tinyllm — instruction-tuned

A 15.7M-parameter Llama-architecture language model trained from
random initialization on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).

Built as a complete walk through the model lifecycle — tokenizer, architecture,
pretraining, evaluation, instruction tuning, packaging, quantization, and local
serving. It is small enough to train in about 45 minutes on a free Colab T4 and
to run on a 2-core laptop CPU with no GPU.

## Architecture

| | |
|---|---|
| Parameters | 15,735,168 (12,589,440 non-embedding) |
| Layers | 8 |
| Hidden size | 384 |
| Attention heads | 6 query / 2 key-value (GQA) |
| Head dim | 64 |
| MLP | SwiGLU, intermediate 1024 |
| Normalization | RMSNorm (eps 1e-05) |
| Position encoding | RoPE (theta 10000) |
| Context length | 512 |
| Vocabulary | 8192 (SentencePiece BPE, byte fallback) |
| Embeddings | tied input/output |

## Training

| | |
|---|---|
| Tokens | 164M (~10 per parameter) |
| Steps | 2,500 at 65,536 tokens/step |
| Optimizer | AdamW (betas 0.9/0.95, wd 0.1 on matrices only) |
| Schedule | cosine, 200 warmup steps, peak LR 0.0006 |
| Precision | fp16 AMP with loss scaling |
| Hardware | 1x NVIDIA T4 (Colab free tier) |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("pythonstudentiam/tinyllm")
model = AutoModelForCausalLM.from_pretrained("pythonstudentiam/tinyllm")

messages = [{"role": "user", "content": "Write a story about a lost puppy."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=250, do_sample=True, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))
```

### With llama.cpp

GGUF conversions are included in this repo.

```bash
llama-server -m tinyllm-Q8_0.gguf -c 512 --host 127.0.0.1 --port 8080
```

## Limitations

This model has 15.7M parameters and a 8192-token vocabulary,
trained exclusively on synthetic children's stories. Be concrete about what that means:

- **It only does one thing.** It writes simple short stories in the TinyStories
  style. Anything else — code, arithmetic, factual questions, translation,
  summarization of arbitrary text — produces confident nonsense.
- **Its vocabulary is small.** Words outside a children's-story vocabulary fall
  back to individual bytes, which it handles poorly.
- **Context is 512 tokens.** There is no long-range coherence to be had.
- **No safety tuning of any kind.** It has had no alignment work beyond
  instruction tuning on story prompts.
- **Quantization hurts more than usual.** Small models have less parameter
  redundancy to absorb rounding error; Q4_K_M is measurably worse here than the
  usual "negligible loss" guidance for 7B+ models would suggest.

Not suitable for any production use. It is a teaching artifact.

## Training data

[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) — synthetic
short stories generated by GPT-3.5/GPT-4, constrained to the vocabulary of a
3-4 year old. Licensed CDLA-Sharing-1.0.

Instruction tuning used [TinyStoriesInstruct](https://huggingface.co/datasets/roneneldan/TinyStoriesInstruct).