Penny-Alpha-SFT-dev1

Penny Alpha (59M-parameter mixture-of-experts, pretrained on 1B tokens of Ultra-FineWeb-L3) fine-tuned for chat with reasoning on Dolci-Think-SFT conversations (8K context).

Model Penny-Alpha-SFT-dev1
Family Penny
Version 1.0
Author Hanz
Organization LocalWisdom
Parameters 0.06B total (mixture of experts)
Architecture DeepSeek-V4.1-Flash (DeepseekV41ForCausalLM)
Context length 8,192 tokens
Initialized from ./init/penny-alpha-base
Tokens this stage 0.200B
Peak LR / schedule 0.0003 / cosine
Tokens per step 131,072
Final validation loss 3.9876
Weights model.safetensors (bf16; router bias, attention sinks and hyper-connection mixes in f32)

Training data

Source Dataset Tokens
dolci_think allenai/Dolci-Think-SFT-Clean-v2 0.200B

Chat format

<|end▁of▁sentence|>{system}<|User|>{user}<|Assistant|><think>...</think>{reply}<|end▁of▁sentence|> (the tokenizer's chat_template writes this).

Downloads last month
14
Safetensors
Model size
58.9M params
Tensor type
F32
·
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train LocalWisdom/Penny-Alpha-SFT-dev1