allenai/Dolci-Think-SFT-Clean-v2
Viewer • Updated • 2.05M • 3.76k • 9
Penny Alpha (59M-parameter mixture-of-experts, pretrained on 1B tokens of Ultra-FineWeb-L3) fine-tuned for chat with reasoning on Dolci-Think-SFT conversations (8K context).
| Model | Penny-Alpha-SFT-dev1 |
| Family | Penny |
| Version | 1.0 |
| Author | Hanz |
| Organization | LocalWisdom |
| Parameters | 0.06B total (mixture of experts) |
| Architecture | DeepSeek-V4.1-Flash (DeepseekV41ForCausalLM) |
| Context length | 8,192 tokens |
| Initialized from | ./init/penny-alpha-base |
| Tokens this stage | 0.200B |
| Peak LR / schedule | 0.0003 / cosine |
| Tokens per step | 131,072 |
| Final validation loss | 3.9876 |
| Weights | model.safetensors (bf16; router bias, attention sinks and hyper-connection mixes in f32) |
| Source | Dataset | Tokens |
|---|---|---|
| dolci_think | allenai/Dolci-Think-SFT-Clean-v2 | 0.200B |
<|end▁of▁sentence|>{system}<|User|>{user}<|Assistant|><think>...</think>{reply}<|end▁of▁sentence|> (the tokenizer's chat_template writes this).