Upload README.md with huggingface_hub

bd3e147 verified 1 day ago

473 Bytes

tags:
  - gpt
  - from-scratch
  - code-generation
  - rope
  - gqa
  - swiglu

gpt-python-modern-114m

A ~114M parameter LLaMA/Mistral-style decoder-only transformer (RMSNorm, RoPE, SwiGLU, Grouped Query Attention), trained from scratch on devoppro/Python.

Layers: 12, Query heads: 12, KV heads: 4
Embedding dim: 768
Context length: 512
Trained to step 0, val loss 10.3440
Tokenizer: GPT-2 BPE (tiktoken 'gpt2')