README.md

---
library_name: transformers
pipeline_tag: text-generation
language:
  - en
tags:
  - spark-gpt
  - qwen3-moe
  - from-scratch
  - stories
---

# Lil Bard 172M MoE

Lil Bard is a small English story language model pretrained from scratch. It is
a base model, not an instruction-tuned or chat model.

The model has 172,052,992 total parameters and 58,806,784 active parameters per
token. It uses 16 transformer layers, width 512, 8 feed-forward experts with
top-2 routing, and a maximum exported context length of 32,768 tokens.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "N8Programs/lil-bard"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Once upon a time", return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

The tokenizer automatically prepends BOS. Its special-token IDs are EOS 0,
BOS 8190, and PAD 8191.

## Architecture

| Property | Value |
|---|---:|
| Total parameters | 172,052,992 |
| Active parameters/token | 58,806,784 |
| Layers | 16 |
| Hidden size | 512 |
| Attention heads / KV heads | 4 / 2 |
| Head dimension | 128 |
| Experts / selected experts | 8 / 2 |
| Dense MLP size | 1,536 |
| Expert MLP size | 768 |
| Vocabulary | 8,192 |
| Maximum exported context | 32,768 |
| Published weight dtype | BF16 |

The checkpoint uses the stock Transformers `Qwen3MoeForCausalLM` layout. MoE
expert weights are stored as per-expert `gate_proj`, `up_proj`, and `down_proj`
tensors for compatibility across Transformers releases; loading has been tested
with Transformers 4.57.1 and 5.11.0.

## Tokenizer

The 8,192-entry tokenizer is a byte-level BPE tokenizer trained on a balanced
1.5-million-document sample of the corpus. It does not use regex, whitespace,
or word pretokenization. BOS, EOS, PAD, and UNK are distinct tokens.

## Training data

The corpus contained 8,732,634 documents drawn from:

- [`klusai/ds-tf1-en-3m`](https://huggingface.co/datasets/klusai/ds-tf1-en-3m)
- [`karpathy/tinystories-gpt4-clean`](https://huggingface.co/datasets/karpathy/tinystories-gpt4-clean)
- A deterministic 3-million-row sample of
  [`littlelearner/LittleCurriculum`](https://huggingface.co/datasets/littlelearner/LittleCurriculum)

A canonical validation set excluded 1,000 DS-TF1 test rows and 1,000
TinyStories test rows from training.

The model trained for exactly 2,492,032,616 real loss tokens over 25,485
distributed steps on two NVIDIA GB10 systems. Whole-document packing achieved
99.4713% utilization. Training used a local adaptation of
[`N8python/spark-gpt`](https://github.com/N8python/spark-gpt).

The complete training trace is available in the
[`lil_bard_moe_8x2_full` W&B run](https://wandb.ai/n8programs/sparkgpt/runs/mxk8gln1).

## Evaluation

| Evaluation | Result |
|---|---:|
| Canonical validation loss | 1.42228 nats/token |
| ARC-Easy zero-shot accuracy | 32.15% |
| ARC-Easy zero-shot normalized accuracy | 32.79% |

ARC-Easy was evaluated on all 2,376 test questions with lm-eval 0.4.12 in
BF16, using the base-model prompt format and an explicit BOS token.

## Historical checkpoints

To keep ordinary downloads of this repository small, the 25 periodic
checkpoints are published separately in
[`N8Programs/lil-bard-checkpts`](https://huggingface.co/N8Programs/lil-bard-checkpts).
They span step 1,000 through step 25,000 in increments of 1,000.

## Limitations

This model was trained primarily on simple synthetic stories. It has limited
world knowledge and reasoning ability, may produce repetitive or incoherent
text, and has not been safety-tuned. Do not use it for factual, medical, legal,
financial, or other high-stakes decisions.