Upload BF16 final Lil Bard step-25485 model

README.md

@@ -0,0 +1,120 @@
+---
+library_name: transformers
+pipeline_tag: text-generation
+language:
+  - en
+tags:
+  - spark-gpt
+  - qwen3-moe
+  - from-scratch
+  - stories
+---
+
+# Lil Bard 172M MoE
+
+Lil Bard is a small English story language model pretrained from scratch. It is
+a base model, not an instruction-tuned or chat model.
+
+The model has 172,052,992 total parameters and 58,806,784 active parameters per
+token. It uses 16 transformer layers, width 512, 8 feed-forward experts with
+top-2 routing, and a maximum exported context length of 32,768 tokens.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "N8Programs/lil-bard"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    device_map="auto",
+)
+
+inputs = tokenizer("Once upon a time", return_tensors="pt").to(model.device)
+with torch.inference_mode():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=200,
+        do_sample=True,
+        temperature=0.8,
+        top_p=0.95,
+        pad_token_id=tokenizer.pad_token_id,
+    )
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+The tokenizer automatically prepends BOS. Its special-token IDs are EOS 0,
+BOS 8190, and PAD 8191.
+
+## Architecture
+
+| Property | Value |
+|---|---:|
+| Total parameters | 172,052,992 |
+| Active parameters/token | 58,806,784 |
+| Layers | 16 |
+| Hidden size | 512 |
+| Attention heads / KV heads | 4 / 2 |
+| Head dimension | 128 |
+| Experts / selected experts | 8 / 2 |
+| Dense MLP size | 1,536 |
+| Expert MLP size | 768 |
+| Vocabulary | 8,192 |
+| Maximum exported context | 32,768 |
+| Published weight dtype | BF16 |
+
+The checkpoint uses the stock Transformers `Qwen3MoeForCausalLM` layout.
+
+## Tokenizer
+
+The 8,192-entry tokenizer is a byte-level BPE tokenizer trained on a balanced
+1.5-million-document sample of the corpus. It does not use regex, whitespace,
+or word pretokenization. BOS, EOS, PAD, and UNK are distinct tokens.
+
+## Training data
+
+The corpus contained 8,732,634 documents drawn from:
+
+- [`klusai/ds-tf1-en-3m`](https://huggingface.co/datasets/klusai/ds-tf1-en-3m)
+- [`karpathy/tinystories-gpt4-clean`](https://huggingface.co/datasets/karpathy/tinystories-gpt4-clean)
+- A deterministic 3-million-row sample of
+  [`littlelearner/LittleCurriculum`](https://huggingface.co/datasets/littlelearner/LittleCurriculum)
+
+A canonical validation set excluded 1,000 DS-TF1 test rows and 1,000
+TinyStories test rows from training.
+
+The model trained for exactly 2,492,032,616 real loss tokens over 25,485
+distributed steps on two NVIDIA GB10 systems. Whole-document packing achieved
+99.4713% utilization. Training used a local adaptation of
+[`N8python/spark-gpt`](https://github.com/N8python/spark-gpt).
+
+The complete training trace is available in the
+[`lil_bard_moe_8x2_full` W&B run](https://wandb.ai/n8programs/sparkgpt/runs/mxk8gln1).
+
+## Evaluation
+
+| Evaluation | Result |
+|---|---:|
+| Canonical validation loss | 1.42228 nats/token |
+| ARC-Easy zero-shot accuracy | 32.15% |
+| ARC-Easy zero-shot normalized accuracy | 32.79% |
+
+ARC-Easy was evaluated on all 2,376 test questions with lm-eval 0.4.12 in
+BF16, using the base-model prompt format and an explicit BOS token.
+
+## Historical checkpoints
+
+To keep ordinary downloads of this repository small, the 25 periodic
+checkpoints are published separately in
+[`N8Programs/lil-bard-checkpts`](https://huggingface.co/N8Programs/lil-bard-checkpts).
+They span step 1,000 through step 25,000 in increments of 1,000.
+
+## Limitations
+
+This model was trained primarily on simple synthetic stories. It has limited
+world knowledge and reasoning ability, may produce repetitive or incoherent
+text, and has not been safety-tuned. Do not use it for factual, medical, legal,
+financial, or other high-stakes decisions.

config.json

@@ -0,0 +1,36 @@
+{
+  "architectures": [
+    "Qwen3MoeForCausalLM"
+  ],
+  "model_type": "qwen3_moe",
+  "vocab_size": 8192,
+  "hidden_size": 512,
+  "num_hidden_layers": 16,
+  "intermediate_size": 1536,
+  "num_attention_heads": 4,
+  "num_key_value_heads": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "rms_norm_eps": 1e-06,
+  "max_position_embeddings": 32768,
+  "rope_theta": 1000000.0,
+  "rope_scaling": null,
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "tie_word_embeddings": false,
+  "use_cache": true,
+  "bos_token_id": 8190,
+  "eos_token_id": 0,
+  "pad_token_id": 8191,
+  "torch_dtype": "bfloat16",
+  "decoder_sparse_step": 1,
+  "moe_intermediate_size": 768,
+  "num_experts": 8,
+  "num_experts_per_tok": 2,
+  "norm_topk_prob": true,
+  "output_router_logits": false,
+  "router_aux_loss_coef": 0.001,
+  "mlp_only_layers": [],
+  "use_sliding_window": false,
+  "sliding_window": null
+}

generation_config.json

@@ -0,0 +1,5 @@
+{
+  "bos_token_id": 8190,
+  "eos_token_id": 0,
+  "pad_token_id": 8191
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6127bc3df898c2e30efee0cf9f01e13035c895af6f618e4830fea589ca30d91
+size 344125920

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<|beginoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|pad|>",
+  "model_max_length": 32768,
+  "clean_up_tokenization_spaces": false,
+  "unk_token": "<|unk|>"
+}