| tags: | |
| - gpt | |
| - from-scratch | |
| - code-generation | |
| - rope | |
| - gqa | |
| - swiglu | |
| # gpt-python-modern-114m | |
| A ~114M parameter LLaMA/Mistral-style decoder-only transformer (RMSNorm, RoPE, SwiGLU, | |
| Grouped Query Attention), trained from scratch on | |
| [devoppro/Python](https://huggingface.co/datasets/devoppro/Python). | |
| - Layers: 12, Query heads: 12, KV heads: 4 | |
| - Embedding dim: 768 | |
| - Context length: 512 | |
| - Trained to step 0, val loss 10.3440 | |
| - Tokenizer: GPT-2 BPE (tiktoken 'gpt2') | |