---
license: apache-2.0
language:
- en
tags:
- small-lm
- gemma4-attention
- muon
- swiglu
- experimental
library_name: pytorch
---

# Shard-40m-v1

A 54.5M parameter dense transformer trained on consumer-grade compute (Thunder Compute pretrain + Colab anneal). Released as a research artifact and pipeline-validation reference. Not a deployable model.

This is the first checkpoint in the Shard series of small experimental transformers.

## Architecture

```
Total params:        54,538,752 (~54.5M)
Hidden dim:          512
Layers:              12
Attention heads:     8 (MHA, no GQA)
Head dim:            64
MLP intermediate:    2048 (SwiGLU)
Vocab size:          8192
Max sequence:        8192
Attention pattern:   Gemma 4 alternating sliding window (window=1024) + global, last layer global
Norm:                RMSNorm, pre-norm
Position encoding:   RoPE on Q and K
Embeddings:          tied input/output
Activation:          SwiGLU
MoE:                 none
Engram:              none
```

## Training

```
Phase 1 (pretrain):
  Compute:           Thunder Compute single GPU
  Steps:             48,220 of a 100,000 step target (paused early)
  Throughput:        86,800 tokens per second
  Optimizer:         Muon for hidden 2D weights, AdamW for embeddings and norms
  LR schedule:       WSD (warmup-stable-decay)
  Stabilizers:       lm_head logit cap 30, z-loss coefficient 1e-4

Phase 2 (anneal):
  Compute:           Colab A100
  Steps:             20,000 (full anneal complete)
  Final cross-entropy: 3.27
  Mix:               OpenWebMath, FineWeb-Edu carryover, NuminaMath, MetaMathQA, ArXiv, Cosmopedia, AI2 ARC
```

## Files

- `models/model.pt` — anneal final checkpoint (model state only, 105 MB bf16)
- `models/pretrain.pt` — pretrain step 47,500 (with optimizer state, 217 MB)
- `models/tokenizer.json` — custom 8192-vocab BPE
- `code/` — minimum loading code (model.py, config.py, tokenizer.py, muon.py)

## How to load

```python
import sys, torch
sys.path.insert(0, 'code')
from config import Config
from model import ToyLM
from tokenizer import load_tokenizer

ck = torch.load('models/model.pt', map_location='cpu', weights_only=False)
cfg = Config(**ck['cfg']) if isinstance(ck['cfg'], dict) else ck['cfg']
model = ToyLM(cfg).cuda().to(torch.bfloat16)
model.load_state_dict(ck['model'])
model.eval()

tok = load_tokenizer('models/tokenizer.json')
ids = torch.tensor([tok.encode('The capital of France is').ids], device='cuda')
with torch.no_grad():
    for _ in range(40):
        logits, _ = model(ids)
        nxt = logits[:, -1].argmax(-1, keepdim=True)
        ids = torch.cat([ids, nxt], 1)
print(tok.decode(ids[0].tolist()))
```

## Benchmark

Greedy decode at 47 tokens per second on a single CUDA GPU. Model footprint 109 MB in bf16, 16 MB peak inference memory.

Sampled outputs at temperature 0.7, top_p 0.9:

| Prompt | Output |
|---|---|
| `The capital of France is` | `"covered by the Crown" (for example, the Great Seal of France...)` |
| `To compute 12 plus 7, we can` | `now use the first 6 as a reversible input...` |
| `Question: What is 23 + 19? Answer:` | `The answer is 23. Answer: 23. Answer: 23` (loops) |
| `def fibonacci(n):` | `// Appendix A. - S. B. V. Shanker. - S. M. P. Gerber...` |
| `Once upon a time, in a small village,` | `a woman is a gentleman in a village with an infinite wealth...` |
| `Solve: 17 * 23 = ?` | `?????\n*****` (breakdown) |

## What this artifact proves

The training pipeline runs end to end on consumer-grade hardware. Muon + AdamW dual optimizer, WSD schedule, Gemma 4 alternating attention, anneal phase mixing math, code, and prose all stable. Loss decreases monotonically through pretrain. No NaN events, no divergence, no rank loss flagged by the Muon min-singular-value sentinel.

## What this artifact cannot do

Math (broken, hallucinates digits or loops). Code generation (gibberish). Factual grounding (hallucinates with grammatical confidence). Long-context retrieval (max sequence 8192 with sliding window 1024 means effective context is much shorter for non-global layers).

## Why release it

To document a reproducible recipe at this scale. The next iteration in this line moves to a 412M MoE with 3 routed experts, vocabulary 262144, distillation pretraining from frontier teachers, and a token budget that crosses the Chinchilla line. This artifact is the baseline against which that next model will be measured.

# Notes
As this model was trained by  [Crownelius](https://huggingface.co/Crownelius), it does not adhere to the required specifications and therefore cannot be integrated into the inference script

## License

Apache 2.0. Use freely. Attribution appreciated but not required.

## Citation

```
@misc{shard40mv1,
  author = {Shane (Crownelius)},
  title  = {Shard-40m-v1: a 54.5M dense transformer trained on consumer compute},
  year   = {2026},
  publisher = {HuggingFace},
  url    = {https://huggingface.co/CompactAI-O/Shard-40m-v1}
}
```