---
tags:
- gpt
- from-scratch
- code-generation
- rope
- gqa
- swiglu
---

# gpt-python-modern-114m

A ~114M parameter LLaMA/Mistral-style decoder-only transformer (RMSNorm, RoPE, SwiGLU,
Grouped Query Attention), trained from scratch on
[devoppro/Python](https://huggingface.co/datasets/devoppro/Python).

- Layers: 12, Query heads: 12, KV heads: 4
- Embedding dim: 768
- Context length: 512
- Trained to step 0, val loss 10.3440
- Tokenizer: GPT-2 BPE (tiktoken 'gpt2')