tinyllm: instruction-tuned

README.md

@@ -0,0 +1,102 @@
+---
+license: cdla-sharing-1.0
+datasets:
+  - roneneldan/TinyStories
+  - roneneldan/TinyStoriesInstruct
+language:
+  - en
+pipeline_tag: text-generation
+tags:
+  - llama
+  - tiny
+  - educational
+  - gguf
+---
+
+# tinyllm — instruction-tuned
+
+A 15.7M-parameter Llama-architecture language model trained from
+random initialization on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).
+
+Built as a complete walk through the model lifecycle — tokenizer, architecture,
+pretraining, evaluation, instruction tuning, packaging, quantization, and local
+serving. It is small enough to train in about 45 minutes on a free Colab T4 and
+to run on a 2-core laptop CPU with no GPU.
+
+## Architecture
+
+| | |
+|---|---|
+| Parameters | 15,735,168 (12,589,440 non-embedding) |
+| Layers | 8 |
+| Hidden size | 384 |
+| Attention heads | 6 query / 2 key-value (GQA) |
+| Head dim | 64 |
+| MLP | SwiGLU, intermediate 1024 |
+| Normalization | RMSNorm (eps 1e-05) |
+| Position encoding | RoPE (theta 10000) |
+| Context length | 512 |
+| Vocabulary | 8192 (SentencePiece BPE, byte fallback) |
+| Embeddings | tied input/output |
+
+## Training
+
+| | |
+|---|---|
+| Tokens | 164M (~10 per parameter) |
+| Steps | 2,500 at 65,536 tokens/step |
+| Optimizer | AdamW (betas 0.9/0.95, wd 0.1 on matrices only) |
+| Schedule | cosine, 200 warmup steps, peak LR 0.0006 |
+| Precision | fp16 AMP with loss scaling |
+| Hardware | 1x NVIDIA T4 (Colab free tier) |
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("pythonstudentiam/tinyllm")
+model = AutoModelForCausalLM.from_pretrained("pythonstudentiam/tinyllm")
+
+messages = [{"role": "user", "content": "Write a story about a lost puppy."}]
+prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+ids = tok(prompt, return_tensors="pt")
+out = model.generate(**ids, max_new_tokens=250, do_sample=True, temperature=0.8)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+### With llama.cpp
+
+GGUF conversions are included in this repo.
+
+```bash
+llama-server -m tinyllm-Q8_0.gguf -c 512 --host 127.0.0.1 --port 8080
+```
+
+## Limitations
+
+This model has 15.7M parameters and a 8192-token vocabulary,
+trained exclusively on synthetic children's stories. Be concrete about what that means:
+
+- **It only does one thing.** It writes simple short stories in the TinyStories
+  style. Anything else — code, arithmetic, factual questions, translation,
+  summarization of arbitrary text — produces confident nonsense.
+- **Its vocabulary is small.** Words outside a children's-story vocabulary fall
+  back to individual bytes, which it handles poorly.
+- **Context is 512 tokens.** There is no long-range coherence to be had.
+- **No safety tuning of any kind.** It has had no alignment work beyond
+  instruction tuning on story prompts.
+- **Quantization hurts more than usual.** Small models have less parameter
+  redundancy to absorb rounding error; Q4_K_M is measurably worse here than the
+  usual "negligible loss" guidance for 7B+ models would suggest.
+
+Not suitable for any production use. It is a teaching artifact.
+
+## Training data
+
+[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) — synthetic
+short stories generated by GPT-3.5/GPT-4, constrained to the vocabulary of a
+3-4 year old. Licensed CDLA-Sharing-1.0.
+
+Instruction tuning used [TinyStoriesInstruct](https://huggingface.co/datasets/roneneldan/TinyStoriesInstruct).
+

chat_template.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{{ '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n' }}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "max_position_embeddings": 512,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 6,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 2,
+  "pad_token_id": 3,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.13.1",
+  "use_cache": true,
+  "vocab_size": 8192
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 3,
+  "do_sample": true,
+  "temperature": 0.8,
+  "top_p": 0.95,
+  "top_k": 40,
+  "repetition_penalty": 1.1,
+  "max_new_tokens": 256,
+  "transformers_version": "5.13.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb1646ce3e70f0b72a5329948d8c02de5939999ef33fa322f29a5304387b700a
+size 62948736

tokenizer.json

@@ -0,0 +1,144 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<unk>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<|im_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<pad>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Metaspace",
+    "replacement": "▁",
+    "prepend_scheme": "first",
+    "split": false
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "SpecialToken": {
+          "id": "<s>",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "SpecialToken": {
+          "id": "<s>",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "<s>",
+          "type_id": 1
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {
+      "<s>": {
+        "id": "<s>",
+        "ids": [
+          1
+        ],
+        "tokens": [
+          "<s>"
+        ]
+      }
+    }
+  },
+  "decoder": {
+    "type": "Sequence",
+    "decoders": [
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "▁"
+        },
+        "content": " "
+      },
+      {
+        "type": "ByteFallback"
+      },
+      {
+        "type": "Fuse"
+      },
+      {
+        "type": "Strip",
+        "content": " ",
+        "start": 1,
+        "stop": 0
+      }
+    ]
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": true,
+    "byte_fallback": true,
+    "ignore_merges": false,
+    "vocab": {
+      "<unk>": 0,
+      "<s>": 1,
+      "<|im_end|>": 2
+    },
+    "merges": []
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c89d5b8716d7431c6e3c0b2ba8cab7dfbac48af8e0a6303c93d5d66dee3c6d7b
+size 134480

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "add_prefix_space": null,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

training_metadata.json

@@ -0,0 +1,164 @@
+{
+  "config": {
+    "project": "tinyllm",
+    "hub": {
+      "user": "pythonstudentiam",
+      "model_repo_suffix": "tinyllm",
+      "ckpt_repo_suffix": "tinyllm-checkpoints",
+      "model_repo": "pythonstudentiam/tinyllm",
+      "ckpt_repo": "pythonstudentiam/tinyllm-checkpoints"
+    },
+    "tokenizer": {
+      "vocab_size": 8192,
+      "model_type": "bpe",
+      "character_coverage": 1.0,
+      "train_sentences": 400000,
+      "max_sentence_length": 8192,
+      "unk_id": 0,
+      "bos_id": 1,
+      "eos_id": 2,
+      "pad_id": 3,
+      "unk_piece": "<unk>",
+      "bos_piece": "<s>",
+      "eos_piece": "</s>",
+      "pad_piece": "<pad>",
+      "im_start": "<|im_start|>",
+      "im_end": "<|im_end|>",
+      "user_defined_symbols": [
+        "<|im_start|>",
+        "<|im_end|>"
+      ],
+      "chat_template": "{% for message in messages %}{{ '<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n' }}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}"
+    },
+    "model": {
+      "hidden_size": 384,
+      "num_hidden_layers": 8,
+      "num_attention_heads": 6,
+      "num_key_value_heads": 2,
+      "intermediate_size": 1024,
+      "vocab_size": 8192,
+      "max_position_embeddings": 512,
+      "rope_theta": 10000.0,
+      "rms_norm_eps": 1e-05,
+      "tie_word_embeddings": true,
+      "attention_bias": false,
+      "mlp_bias": false,
+      "initializer_range": 0.02,
+      "head_dim": 64,
+      "kv_dim": 128,
+      "n_rep": 3,
+      "n_params": 15735168
+    },
+    "data": {
+      "dataset_id": "roneneldan/TinyStories",
+      "instruct_dataset_id": "roneneldan/TinyStoriesInstruct",
+      "train_split": "train",
+      "val_split": "validation",
+      "seq_len": 512,
+      "val_tokens": 1000000,
+      "shard_tokens": 25000000,
+      "seed": 1337
+    },
+    "train": {
+      "micro_batch_size": 32,
+      "grad_accum_steps": 4,
+      "max_steps": 2500,
+      "learning_rate": 0.0006,
+      "min_lr_ratio": 0.1,
+      "warmup_steps": 200,
+      "weight_decay": 0.1,
+      "beta1": 0.9,
+      "beta2": 0.95,
+      "grad_clip": 1.0,
+      "dtype": "fp16",
+      "compile_model": false,
+      "eval_every": 250,
+      "eval_batches": 40,
+      "sample_every": 500,
+      "log_every": 10,
+      "checkpoint_every": 500,
+      "keep_last_n_checkpoints": 2,
+      "seed": 1337,
+      "smoke_max_steps": 50,
+      "smoke_stories": 2000,
+      "tokens_per_step": 65536,
+      "total_tokens": 163840000,
+      "min_lr": 5.9999999999999995e-05
+    },
+    "sft": {
+      "micro_batch_size": 16,
+      "grad_accum_steps": 4,
+      "max_steps": 1500,
+      "learning_rate": 0.0001,
+      "min_lr_ratio": 0.1,
+      "warmup_steps": 50,
+      "weight_decay": 0.0,
+      "beta1": 0.9,
+      "beta2": 0.95,
+      "grad_clip": 1.0,
+      "seq_len": 512,
+      "ignore_index": -100,
+      "eval_every": 200,
+      "log_every": 10,
+      "checkpoint_every": 500,
+      "seed": 1337
+    },
+    "gen": {
+      "max_new_tokens": 256,
+      "temperature": 0.8,
+      "top_p": 0.95,
+      "top_k": 40,
+      "repetition_penalty": 1.1,
+      "eval_prompts": [
+        "Once upon a time, there was a little girl named Lily.",
+        "Tom and Sara went to the park. They saw a big",
+        "The cat was very hungry, so it"
+      ],
+      "eval_instructions": [
+        "Write a story about a lost puppy who finds its way home.",
+        "Write a short story using the words: ball, tree, happy.",
+        "Tell me a story about a brave little boat."
+      ]
+    },
+    "quant": {
+      "levels": [
+        "Q8_0",
+        "Q5_K_M",
+        "Q4_K_M"
+      ],
+      "perplexity_ctx": 512,
+      "perplexity_chunks": 40
+    },
+    "serve": {
+      "host": "127.0.0.1",
+      "port": 8080,
+      "threads": 4,
+      "ctx_size": 512,
+      "served_model_name": "tinyllm",
+      "default_quant": "Q8_0",
+      "llamacpp_build": "b10107",
+      "llamacpp_asset": "llama-b10107-bin-win-cpu-x64.zip",
+      "base_url": "http://127.0.0.1:8080/v1",
+      "llamacpp_url": "https://github.com/ggml-org/llama.cpp/releases/download/b10107/llama-b10107-bin-win-cpu-x64.zip"
+    },
+    "derived": {
+      "head_dim": 64,
+      "kv_dim": 128,
+      "n_params": 15735168,
+      "param_breakdown": {
+        "embedding": 3145728,
+        "attention": 3145728,
+        "mlp": 9437184,
+        "layernorms": 6528,
+        "lm_head": 0,
+        "per_layer": 1573632,
+        "blocks_total": 12589056,
+        "non_embedding": 12589440,
+        "total": 15735168
+      },
+      "tokens_per_step": 65536,
+      "total_tokens": 163840000,
+      "flops_per_token": 94411008
+    }
+  }
+}