Slayer1B 1.09 — training checkpoints
Arkadiusz Słota, Anna Sieczka, Piotr Styła, Aleksander Ogrodzki, Paweł Puzio, Marcin Kuśmierski / Fabryka AI (SlayerLab)
Native training checkpoints of the run that will produce Slayer1B 1.09: continued pretraining from the final weights of Slayer1B 1.08 with a fresh optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a Transformers/safetensors package and not a finished model.
Status: training in progress. This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 35,000 and one full-state checkpoint at update 35,000 (uploaded 12 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on Track. Every file listed below was compared after upload with the checksum written when it was copied from the training machine: the LFS object id (sha256) and the size match.
| Start | final weights of Slayer1B 1.08 (revision 5f854a0c), fresh optimizer state, update 0 |
| Updates so far | 1 → 35,000 in this repository (35,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
| Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then constant; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
| Data | see the data card; new data 20% Polish / 80% English by target allocation |
| Hardware | 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once |
| Trainer | train_gpt_ref r24.2 (sha256 46f83b51…); model definition train_gpt_ref_r23.py (sha256 db36c150…) |
| Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
| Tokenizer | v6, vocabulary 32,768 (sha256 0640d3bd…), as in Slayer1B 1.08 |
Files
| file | contents | size |
|---|---|---|
wagi_<update>.pt |
model weights (float32), config, step |
4.3 GB |
ckpt_35000.pt |
model weights, optimizer state (Muon and AdamW), data-queue position, config, step |
8.9 GB |
wagi is Polish for “weights”. Load with torch.load(path, map_location="cpu", weights_only=True); the model definition is in
train_gpt_ref_r23.py in the 1.08 repository.
Tokens per checkpoint
Two different counts, kept apart on purpose. Positions in this run = update × 1,048,576 (training positions processed by the 1.09 run itself, counting each update once). Lineage lower bound = positions inherited from Slayer1B 1.00–1.08 (18,863,257,682, source- and file-disjoint) + positions in this run − 114,627,128 positions identified as copies in the first data segment. The lineage figure is a lower bound on source- and file-disjoint positions seen by the model line, not a count of globally deduplicated unique tokens.
| update | positions in this run (billions) | lineage lower bound (billions) |
|---|---|---|
| 5000 | 5.24 | 23.99 |
| 10000 | 10.49 | 29.23 |
| 15000 | 15.73 | 34.48 |
| 20000 | 20.97 | 39.72 |
| 25000 | 26.21 | 44.96 |
| 30000 | 31.46 | 50.21 |
| 35000 | 36.70 | 55.45 |
SHA-256
| update | file | sha256 |
|---|---|---|
| 5000 | wagi_5000.pt |
bc289e2afe7e40d37c60a23c75bd10480ac903e29a0f5d8d184be6bcdf640e91 |
| 10000 | wagi_10000.pt |
fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643 |
| 15000 | wagi_15000.pt |
1cc151d20cd1abf3b02a9dfbedb5c5b458861105307498acccfc15283d951ae4 |
| 20000 | wagi_20000.pt |
41a457dcdf1d60011fe576fe938d2d61e80ce71028c8f1b484face9ee3dc73d5 |
| 25000 | wagi_25000.pt |
3bd8e411635eb1a18fc6e96fcfee38b68850dbd20171f9cc73897187277ceda3 |
| 30000 | wagi_30000.pt |
74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d |
| 35000 | wagi_35000.pt |
75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43 |
| 35000 | ckpt_35000.pt |
f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e |
The same list is in SHA256SUMS.
Limitations
These checkpoints are intermediate states of a constant-learning-rate run. Training loss and evaluation results at a given update are on Track; they are not a quality verdict on a finished model. Changes of the data segment during the run (for example at update 10,000) change the training-loss distribution, so a drop across such a boundary does not by itself show a quality gain.