DynamicMind-MoE-Instruct

An instruction-tuned version of DynamicMind-MoE, a sparse mixture-of-experts model with 30.2M total parameters and 8.9M active per token. It was fully fine-tuned for two epochs on smol-smoltalk with the loss computed only on assistant tokens (end-of-turn tokens weighted 5×, so the model learns to stop).

At this size it is a small research model. It follows the chat format and answers simple requests, but factual knowledge, reasoning and arithmetic are weak. See the limitations below.

Quickstart

This model uses custom architecture code, so load it with trust_remote_code=True. Use the recommended sampling settings (stored in generation_config.json). With near-greedy decoding this model can fall into repetition loops.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "DedeProGames/DynamicMind-MoE-Instruct"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, torch_dtype=torch.float32).eval()

messages = [{"role": "user", "content": "What is photosynthesis?"}]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=False)
out = model.generate(input_ids, max_new_tokens=256)  # temperature 0.7, top-p 0.9, repetition penalty 1.1 from generation_config.json
print(tokenizer.decode(out[0, input_ids.shape[1]:], skip_special_tokens=True))

Keep prompt + answer within the 1,024-token context.

Chat format

This vocabulary has no chat tokens, so the format is plain text. apply_chat_template renders it:

<|endoftext|>### User:
{user message}
### Assistant:
{assistant reply}
<|endoftext|>### User:
...

<|endoftext|> (id 0) is the BOS, the end-of-turn token and the EOS.

Architecture

Same as DynamicMind-MoE: 9 layers, hidden size 256, 8 attention heads with 2 KV heads, one always-on shared expert plus 14 routed experts per layer with the top 2 selected per token, auxiliary-loss-free balancing, tied embeddings, 8,192-token digit-aware vocabulary, 1,024-token context. 30,150,912 total parameters, 8,917,248 active per token. Weights in BF16 (model.safetensors, 60 MB).

Training

Base	DedeProGames/DynamicMind-MoE
Data	smol-smoltalk train split, conversations of up to 1,024 tokens (221,811 with this tokenizer; 68.9M supervised tokens)
Epochs	2 (13,864 steps, batch 32 = 16 × 2 gradient accumulation)
Optimizer	AdamW (β = 0.9, 0.95), weight decay 0.1 on 2D matrices, grad clip 1.0
Schedule	lr 1e-4 (the base model's pretraining peak), 300 warmup steps, cosine to 1e-5
Loss	assistant tokens; `<
Precision	bf16 autocast, fp32 master weights
Hardware	1× NVIDIA RTX 3060 12 GB, 239 minutes

Held-out loss on assistant tokens (smol-smoltalk test split): 1.422 at the end of training.

Evaluation

Benchmarks follow the protocol of the Kiyo-230M-Instruct card: FP16, the model's own chat template, no system prompt, new-token limits IFEval 1,024, GPQA 384, LiveBench 512. The context is 1,024 tokens, so the generation limit is capped at 1,024 minus the prompt length. The main setting uses temperature 0.1, top-k 50 and repetition penalty 1.05, as on the card (GPQA is sampled at T = 0.6, as on the card). The second setting is the recommended one shipped in generation_config.json.

Benchmark	Metric	DynamicMind-MoE-Instruct, protocol	DynamicMind-MoE-Instruct, recommended sampling
GPQA Diamond	pass@1 (198 × 5 samples, T = 0.6)	12.32	not run
IFEval	prompt-level strict	18.85	14.97
LiveBench instruction following	strict, 200 questions	18.66	not run
CaseReportBench	token F1	not run (not on the Hub)	not run
EOS rate (IFEval)	share of answers that stop	0.88	0.99

For reference, the Kiyo card reports (not re-run here): Kiyo-230M-Instruct GPQA 21.11, IFEval 17.38, LiveBench 18.77; SmolLM2-135M-Instruct GPQA 16.57, IFEval 20.89, LiveBench 24.23; BananaMind-2-Pro GPQA 15.05, IFEval 20.15, LiveBench 19.75; Supra2-100M-Instruct GPQA 5.56, IFEval 9.06, LiveBench 11.28.

Notes:

• With the protocol setting, about 12% of IFEval answers never stop and loop. Under the recommended sampling this drops to about 1%, at a cost of about 4 IFEval points.
• GPQA: only 46% of answers contain an identifiable option letter, and 40 prompts were truncated to fit the 1,024-token context. Among answered questions, accuracy is 26.6%, close to the 25% chance level.
• The GPQA prompt was written for this run, not taken from the Kiyo card, so the number is not strictly comparable.
• Single run, single seed. Differences of a point or two are within noise.

Per-sample outputs and summaries are in eval/.

Limitations

• It is small (8.9M active parameters per token). Expect factual errors, invented details, arithmetic mistakes, weak multi-step reasoning, and repetitive text in long answers.
• Under near-greedy decoding it can loop. Keep the recommended sampling settings.
• Context is 1,024 tokens.
• English only.
• Requires trust_remote_code=True, since the architecture is custom.
• No safety tuning and no safety evaluation. Do not use it for medical, legal, financial or other high-stakes decisions.

License

Apache-2.0, as for the base model. The SFT data (smol-smoltalk) is also Apache-2.0.