| # 5M Text Generator | |
| A small 5M parameter text generation model trained on educational/scientific text data. | |
| ## Model Details | |
| - **Parameters**: 4,983,808 (4.98M) | |
| - **Architecture**: Transformer Decoder with GQA, RoPE, SwiGLU | |
| - **Training**: 1000 steps on 5000 text samples | |
| - **Vocabulary**: 8000 tokens (BPE) | |
| ## Usage | |
| ```python | |
| from transformers import PreTrainedTokenizerFast | |
| import torch | |
| import sys | |
| sys.path.insert(0, ".") # if needed | |
| from model import TextDecoder | |
| # Load model | |
| model = TextDecoder.from_pretrained("CodeDevX/5m-text-generator") | |
| tokenizer = PreTrainedTokenizerFast.from_pretrained("CodeDevX/5m-text-generator") | |
| # Generate text | |
| input_ids = tokenizer("The quick brown fox", return_tensors="pt").input_ids | |
| output = model.generate(input_ids, max_new_tokens=50, temperature=0.8) | |
| print(tokenizer.decode(output[0])) | |
| ``` | |
| ## Architecture | |
| - Hidden size: 256 | |
| - Layers: 6 | |
| - Attention heads: 8 (KV heads: 2) | |
| - Intermediate size: 768 | |
| - Max sequence length: 512 | |
| ## Training Results | |
| | Step | Loss | | |
| |------|------| | |
| | 100 | 12.68 | | |
| | 200 | 2.17 | | |
| | 500 | 0.15 | | |
| | 1000 | 0.13 | | |
| ## License | |
| MIT | |