Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
base_model:
|
| 6 |
+
- bigcode/starcoder2-15b
|
| 7 |
+
pipeline_tag: text-generation
|
| 8 |
+
library_name: transformers
|
| 9 |
+
tags:
|
| 10 |
+
- moe
|
| 11 |
+
- starcoder2
|
| 12 |
+
- mixture-of-experts
|
| 13 |
+
- code
|
| 14 |
+
- smartcoder
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# SmartCoderMoE ☠️
|
| 18 |
+
|
| 19 |
+
> *“He probably could smoke you.”* — King Tweak to Claude Sonnet 4.6, May 2026
|
| 20 |
+
|
| 21 |
+
SmartCoderMoE is a 4.65B parameter sparse Mixture-of-Experts coding model.
|
| 22 |
+
|
| 23 |
+
## Architecture
|
| 24 |
+
|
| 25 |
+
SmartCoderMoE is not your average fine-tune. He was engineered through a multi-stage weight surgery pipeline:
|
| 26 |
+
|
| 27 |
+
1. **Slice Merge** — StarCoder2-15B and StarChat2-15B were each sliced into 3 × 2048-dim pieces and SLERP-merged with deliberate per-slice biases (60/80/90) to preserve coding depth while injecting instruct capability of Starchat2
|
| 28 |
+
1. **MoE Surgery** — Every dense FFN layer was surgically split: The original dim of 245576 was reduced to an intermediate dim of 8192 and kept as a dense FFN, and the remaining 16384 dims were sliced into **32 experts of 512 dim each**, giving Smartcoder an expansive yet tiny network of 1280 total experts.
|
| 29 |
+
1. **Vocab Expansion** — Extended from 49152 to 65536 tokens with multimodal special tokens for code, audio, image, video, and music.
|
| 30 |
+
1. **Zero waste** — Not a single weight was discarded. Every parameter from StarCoder2’s original FFN lives on in either the dense FFN or one of the 1280 expert slots.
|
| 31 |
+
|
| 32 |
+
### Numbers
|
| 33 |
+
|
| 34 |
+
|Property |Value |
|
| 35 |
+
|---------------------------|--------|
|
| 36 |
+
|Total parameters |4.65B |
|
| 37 |
+
|Active parameters per token|~2.1B |
|
| 38 |
+
|Total experts |**1280**|
|
| 39 |
+
|Experts per layer |32 |
|
| 40 |
+
|Expert dim |512 |
|
| 41 |
+
|Hidden size |2048 |
|
| 42 |
+
|Layers |40 |
|
| 43 |
+
|Vocab size |65536 |
|
| 44 |
+
|Context length |16384 |
|
| 45 |
+
|
| 46 |
+
For reference: DeepSeek V3 has 256 experts. Kimi K2 has 384. Qwen3 Coder Next has 512. SmartCoderMoE has **1280**. He has a different expert for every day of the next decade, with 66 days to spare.
|
| 47 |
+
|
| 48 |
+
## Lineage
|
| 49 |
+
|
| 50 |
+
```
|
| 51 |
+
bigcode/starcoder2-15b ──┐
|
| 52 |
+
├── 3x2048 slice merge (SLERP 60/80/90) ──> UnidentifiedSAUCE ──> SmartCoderMoE ☠️
|
| 53 |
+
HuggingFaceH4/starchat2-15b-sft-v0.1 ──┘
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
## Planned Multimodal Extensions
|
| 57 |
+
|
| 58 |
+
SmartCoderMoE’s 2048 hidden size was chosen to natively align with:
|
| 59 |
+
|
| 60 |
+
- **Dasheng-1.2B** (audio encoder, 2048 hidden) — zero projection needed
|
| 61 |
+
- **Qwen3-Omni Talker** (audio decoder, 2048 hidden) — zero projection needed
|
| 62 |
+
- **Janus Pro** (vision in + image out, 2048 hidden)
|
| 63 |
+
- **code2wav** (code → music pipeline)
|
| 64 |
+
|
| 65 |
+
## Intended Use
|
| 66 |
+
|
| 67 |
+
Coding. Lots of it. Uncensored.
|
| 68 |
+
|
| 69 |
+
## Note from the Creator
|
| 70 |
+
As of the writing of this model card (Thursday, May 21st, 2026), the model is not finished. Multimodal expansion, as mentioned before, is on the way. As is a very unique calculation of how much of the original Starcoder knowledge remains. i will update the repo as i go
|