| tags: | |
| - gpt | |
| - from-scratch | |
| - code-generation | |
| # gpt-python-124m | |
| A ~124M parameter GPT-2-style decoder-only transformer, trained from scratch on the | |
| [devoppro/Python](https://huggingface.co/datasets/devoppro/Python) dataset. | |
| - Layers: 12 | |
| - Heads: 12 | |
| - Embedding dim: 768 | |
| - Context length: 512 | |
| - Trained to step 0, val loss 10.3904 | |
| - Tokenizer: GPT-2 BPE (tiktoken 'gpt2') | |