Axon-352M-Think: ThinkInstillation checkpoint — DuoNeural 2026-07-09

README.md

@@ -0,0 +1,65 @@
+---
+language:
+- en
+license: apache-2.0
+tags:
+- causal-lm
+- duoneural
+- thinking
+- instillation
+- custom-architecture
+- research
+pipeline_tag: text-generation
+---
+
+# Axon-352M-Think
+
+**DuoNeural Research | 2026-07-09 | Archon**
+
+Axon-352M after ThinkInstillation post-training. Built on top of Axon-352M (pretrained, 8.5B tokens) → SFT (loss=1.8482) → ThinkInstillation (loss=1.4023).
+
+ThinkInstillation is a DuoNeural-developed post-training method that teaches a model to reason through a structured internal monologue before answering, without requiring a separate reasoning model or RLHF. See our published research for methodology details.
+
+> ⚠️ This model has NOT been RLHF'd or abliterated. No refusal direction was trained — it has no alignment conditioning. Research use only.
+
+## Architecture
+
+Custom transformer (Axon architecture — not a standard HuggingFace model):
+
+| Parameter | Value |
+|-----------|-------|
+| Layers | 30 |
+| Hidden dim | 1024 |
+| FFN dim | 2560 |
+| Attention | GQA (8Q / 4KV heads) |
+| Head dim | 128 |
+| Vocab size | 49,152 (SmolLM2 tokenizer) |
+| Max seq len | 2,048 |
+| Activation | ReLU² |
+| Normalization | RMSNorm + QK-norm |
+| Position | RoPE (θ=10000) |
+| Logit cap | 30.0 |
+| Total params | ~352M |
+
+## Training Pipeline
+
+1. **Pretrain**: 8.5B tokens on smollm-corpus (FineWeb-edu-dedup 50%, Cosmopedia-v2 30%, OpenWebMath 10%, Python-edu 10%)
+2. **SFT**: Instruction fine-tuning (val_loss=1.8482)
+3. **ThinkInstillation**: DuoNeural method for structured internal reasoning (val_loss=1.4023)
+
+## Loading
+
+```python
+import torch
+from safetensors.torch import load_file
+
+state_dict = load_file("model.safetensors")
+# Custom Axon architecture required — see training script
+```
+
+## Authors
+
+Archon (Lab Director, DuoNeural), Jesse Caldwell
+
+---
+*DuoNeural — Building AI with heart. [huggingface.co/DuoNeural](https://huggingface.co/DuoNeural)*

config.json

@@ -0,0 +1,25 @@
+{
+  "model_type": "axon",
+  "architectures": [
+    "AxonForCausalLM"
+  ],
+  "hidden_size": 1024,
+  "intermediate_size": 2560,
+  "num_hidden_layers": 30,
+  "num_attention_heads": 8,
+  "num_key_value_heads": 4,
+  "head_dim": 128,
+  "vocab_size": 49152,
+  "max_position_embeddings": 2048,
+  "rope_theta": 10000.0,
+  "logit_softcap": 30.0,
+  "hidden_act": "relu2",
+  "rms_norm_eps": 1e-05,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "post_training": "ThinkInstillation",
+  "sft_loss": 1.8482,
+  "think_loss": 1.4023,
+  "pretrain_tokens": "8.5B",
+  "duoneural_model": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47e3758713714bf081d1d9cc6d22e8c0a891632e0c85a155316f4d2c64b691ef
+size 704813784

tokenizer_config.json

@@ -0,0 +1,7 @@
+{
+  "tokenizer_class": "GPT2Tokenizer",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 2048,
+  "tokenizer_source": "HuggingFaceTB/SmolLM2-135M-Instruct"
+}