NanoMath v1.1

NanoMath is a compact 113.27M-parameter decoder-only language model for maths reasoning. Its architecture and inference code are implemented from scratch in PyTorch rather than through Transformers. Despite its size, v1.1 produces readable multi-step solutions and solved four of six mixed prompts in the current spot check.

This repository contains two files that must be used together:

• model_weights.pth is the v1.1 inference checkpoint.
• token.model is its 16,384-piece SentencePiece tokenizer.

The NanoMath source repository contains the matching model code, training pipeline, and local chat CLI.

Architecture

Setting	Value
Parameters	113.27M
Decoder layers	16
Model width	768
Query heads	12
Key-value heads	4
Context length	1,024
Vocabulary	16,384
Position encoding	RoPE
Normalization	RMSNorm
MLP	SwiGLU

The model uses grouped-query attention, tied token and output embeddings, and bias-free projections. The checkpoint stores the full model configuration and completed optimizer step.

Training record

The v1.1 checkpoint completed optimizer step 5,999 on two NVIDIA T4 GPUs. Its packed training build contains 641,276,817 train tokens and 13,076,780 validation tokens from GSM8K, NuminaMath, and generated maths problems.

Training weights prompt tokens at 0, reasoning tokens at 1, and final-answer tokens at 3. This changes the loss calculation only. It does not add a calculator or symbolic solver at inference time.

Checkpoint SHA-256:

f13f762b64ec710221575b6f5d4dae05f6105d8234b8a53a35ea7b40680f6dca

Run it

git clone https://github.com/agmada-asa/NanoMath.git
cd NanoMath
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
hf download agmadaasa/NanoMath model_weights.pth token.model --local-dir build
python chat.py "What is 84 / 7?"

The model requires the custom classes in the source repository. AutoModel and the standard Transformers text-generation pipeline cannot load this checkpoint.

Observed outputs

The following results came from greedy decoding with a 200-token limit. The model answered four of the six prompts correctly. This is a small, reproducible behavior sample, not an accuracy score.

Prompt	Output answer	Expected	Result
What is 84 / 7?	12	12	Correct
Calculate 57 + 68.	89	125	Incorrect
What is 17 * 24?	408	408	Correct
A box has 15 pencils. Six are removed and four are added. How many pencils are in the box?	13	13	Correct
Solve 3x + 5 = 20.	5	5	Correct
Which is larger, 5/8 or 2/3?	Did not finish a valid comparison	2/3	Incorrect

Producing coherent solution structure across division, multiplication, algebra, and a word problem is a solid result for a model of this size. NanoMath can still produce plausible reasoning with a wrong answer, so its arithmetic should be checked before use.

Versions

main and the v1.1 tag contain the current model. The original 136.18M checkpoint and its 32,768-piece tokenizer remain available under the v1.0 tag.

hf download agmadaasa/NanoMath \
  model_weights.pth token.model \
  --revision v1.0 \
  --local-dir build

Do not mix a checkpoint from one version with the tokenizer from another.