DynamicMind-MoE

DynamicMind-MoE is a sparse mixture-of-experts decoder-only model trained on FineWeb-Edu, SmolLM-Corpus and FineMath

The model has 30.2M total parameters but activates only 8.9M per token, and was initialised by sparse-upcycling DynamicMind-Mini. It keeps the same custom 8k-token byte-level BPE tokenizer with digit-aware tokenization, so inference cost per token is unchanged from the dense model while total capacity is 3.4x larger.

Model Details

Field	Value
Total parameters	30,150,912
Active parameters per token	8,917,248
Architecture	Sparse MoE Llama-style decoder
Layers	9
Hidden size	256
Routed experts per layer	14
Shared experts per layer	1
Experts activated per token	2 routed + 1 shared
Expert intermediate size	256
Attention heads	8
KV heads	2
Vocabulary size	8,192
Context length	1,024
Embeddings	Tied input/output embeddings
Weight format	safetensors

Mixture of Experts

Each block's dense MLP is replaced by one always-on shared expert plus 14 fine-grained routed experts, of which the router selects the top 2 per token. Shared + top-2 at intermediate size 256 reproduces the dense model's exact active parameter count.

Load balancing uses an auxiliary-loss-free bias (DeepSeek-V3 style): a per-expert bias steers selection toward idle experts while the combining weights come from the unbiased softmax, so balancing costs no gradient interference.

Routing is per token, per layer — a single sequence touches many different experts, and the 9 routers are independent of each other.

Tokenizer

DynamicMind-MoE uses the same digit-aware 8k tokenizer as DynamicMind-Mini.

Digits are kept as separate tokens so numbers do not collapse into large number tokens during tokenization.

Digit IDs:

Token	ID
1	9
2	10
3	11
4	12
5	13
6	14
7	15
8	16
9	17
0	18

Training

Field	Value
Initialisation	Sparse upcycling from DynamicMind-Mini
Tokens seen	10,008,133,632
Optimizer steps	8,484
Sequence length	1,024
Tokens per optimizer step	1,179,648
Peak learning rate	1e-4
Min learning rate	1e-5
LR schedule	Cosine decay after 200 warmup steps
Optimizer	AdamW (betas 0.9, 0.95)
Weight decay	0.1
Gradient clipping	1.0
Seed	1337
Hardware	1x RTX 3060 12GB, 39.5h

Data mixture: FineWeb-Edu sample-10BT 50.0%, Cosmopedia-v2 27.8%, FineMath-4plus 22.2%.

Embeddings, attention and norms transferred 1:1 from the dense model. The dense MLP (intermediate size 768) splits exactly into three 256-wide slices — because down_proj sums over the intermediate axis — and each expert was seeded from one slice plus small noise to break router symmetry.

Benchmarks

Self-reported results from the official BananaMind Base Bench 1.1 script, all measured with the same runner, dtype (bfloat16) and GPU.

Model	Total params	Active params	Overall Elo
BananaMind-2-Medium	55.9M	55.9M	1,037
GPT-2	124M	124M	990
BananaMind-2-Nano	12.1M	12.1M	915
DynamicMind-MoE	30.2M	8.9M	912
Pythia-14m-deduped	14M	14M	909
BananaMind-2-MoE	26.1M	—	903
DynamicMind-Mini	8.9M	8.9M	868

Detailed DynamicMind-MoE result

Category	Passed	Elo
Overall	137 / 350	912
Language completion	38 / 50	1,115
Logical reasoning	17 / 50	970
World knowledge	25 / 50	951
Commonsense	23 / 50	923
Context tracking	15 / 50	843
Quantitative	11 / 50	822
Code completion	8 / 50	793

Against the dense DynamicMind-Mini the MoE gains +44 Elo overall at identical inference cost, improving in six of seven categories. Quantitative is the exception (822 vs 837), despite FineMath being 22% of the corpus.

Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings. Cross-tokenizer comparisons (e.g. against GPT-2's 50k vocabulary) carry a residual tokenizer effect that mean-per-token log-probability does not fully remove.

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run inference:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DedeProGames/DynamicMind-MoE"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()

prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is a base model, not instruction-tuned — it continues text rather than following instructions. At 8.9M active parameters it reproduces register and structure well (encyclopedic text reads encyclopedic, code keeps valid indentation) but is frequently wrong on facts and arithmetic. Keep a finite generation limit and do not use it for high-stakes decisions.

License

Apache 2.0