File size: 381 Bytes
439e641 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | ---
tags:
- gpt
- from-scratch
- code-generation
---
# gpt-python-124m
A ~124M parameter GPT-2-style decoder-only transformer, trained from scratch on the
[devoppro/Python](https://huggingface.co/datasets/devoppro/Python) dataset.
- Layers: 12
- Heads: 12
- Embedding dim: 768
- Context length: 512
- Trained to step 0, val loss 10.3904
- Tokenizer: GPT-2 BPE (tiktoken 'gpt2')
|