README.md

---
license: cc-by-sa-4.0
library_name: transformers
pipeline_tag: text-generation
tags:
- oeis
- integer-sequences
- qwen3
- causal-lm
- mlx
datasets:
- N8Programs/oeis-massive
---

# NextTerm-440M

[![Benchmark radar](radar_chart.png)](radar_chart.png)

## Model Summary

NextTerm-440M is a 440M parameter causal transformer trained to continue integer
sequences. It uses a Qwen3 architecture with a compact 16-token digit
vocabulary: decimal digits, negative sign, comma separator, BOS, EOS, PAD, and one unused token.

The model was trained on an extended OEIS corpus that enhanced many OEIS sequences with additional terms from b-files (supplemental appendices provided with OEIS) and then further augmented the data w/ a variety of prefix-preserving transforms empirically selected via small pilot experiments. The model was trained for 14B tokens w/ preserved sequence prefixes rather than concatenating distinct documents (as this was found to improve performance in pilot experiments).


NextTerm-440M improves dramatically over NextTerm-47M on long-context sequence continuation (as it was trained w/ a context length of 4096), innate OEIS knowledge, and long-range in context learning. The 47M model, however, remains ahead on very short prefixes that require simple rule induction without much context, which may be due to the 440M model's training on longer contexts and more complex sequences.

The tokenizer accepts integer sequences formatted as comma-separated values, for
example:

```text
1,-2,3,-4,
```

The tokenizer ignores characters other than digits, commas, and `-`. Digits are
tokenized individually, so there is no fixed integer-magnitude limit, but large
integers consume more context. The model was not trained on numbers with leading
zeros, so strings like `01,02,03,` should be treated as out of distribution.

## Training Details

| Field | Value |
| --- | --- |
| Parameters | 440,500,224 |
| Architecture | Qwen3-style causal LM |
| Layers | 28 |
| Hidden size | 1024 |
| FFN size | 3072 |
| Attention heads | 16 |
| KV heads | 8 |
| Vocabulary size | 16 |
| Training tokens | 13,999,999,995 |
| Sequence length cap | 4096 training tokens per sequence |
| Batch mode | Length-bucketed sequence batches |
| Optimizer | Muon/AdamW hybrid |
| LR schedule | Linear warmup to `1e-2` for Muon, `1e-4` for AdamW, cosine decay to 0.1x, final cooldown to 0 |
| Training hardware | Single H100 |
| Export dtype | bfloat16 |

A classic Muon/AdamW hybrid was used: Muon for 2D weight matrices and AdamW for 1D parameters and embedding matrices.

The model was trained on the following files in the `N8Programs/oeis-massive` dataset, randomly mixed:

- `oeis_train_bfile_prefix4096.packed`
- `oeis_synth_aug0_inv_len_13245370099_seed0.packed`

## Evaluation Results

### Main Benchmarks

| Model | OEIS-Eval-Neo | Ryskina & Knight | M1 Competition 111 MAPE |
| --- | ---: | ---: | ---: |
| **NextTerm-440M** | **34.43%** | 52.63% | **17.6239** |
| NextTerm-47M | 29.49% | **70.18%** | 18.7621 |
| Qwen3-0.6B | 18.44% | 33.33% | 22.7984 |
| Qwen3-1.7B | 20.77% | 49.12% | 22.2411 |
| Qwen3-4B | 23.74% | 63.16% | 19.1731 |
| Qwen3-8B | 24.62% | 57.89% | 18.4027 |
| Qwen3-14B | 26.00% | 59.65% | 17.9837 |

OEIS-Eval-Neo is a decontaminated held-out OEIS next-term evaluation. M1
Competition 111 reports macro MAPE, where lower is better. Ryskina & Knight
(2021) is a 57-sequence next-term benchmark based on psychometrics and puzzles. Note that the 47M model's strong performance on Ryskina & Knight is indicative of its strength on short-prefix sequences and rule induction. 

### Polynomial Continuation

The polynomial continuation evaluation samples integer sequences from
polynomials of degree 1 through 4 and asks for the next term. Accuracy is exact
match across 200 samples for each prompt length `k`.

| Model | Arithmetic | Quadratic | Cubic | Quartic |
| --- | ---: | ---: | ---: | ---: |
| **NextTerm-440M** | 94.38% | **86.39%** | **75.20%** | **67.83%** |
| NextTerm-47M | 94.15% | 81.07% | 37.43% | 15.17% |
| Qwen3-0.6B | 90.31% | 8.72% | 0.30% | 0.02% |
| Qwen3-1.7B | 93.10% | 41.57% | 5.36% | 0.71% |
| Qwen3-4B | 93.90% | 77.26% | 28.18% | 5.98% |
| Qwen3-8B | **96.10%** | 80.59% | 32.93% | 7.95% |
| Qwen3-14B | 95.60% | 84.61% | 49.16% | 14.98% |

## Usage

### MLX

```bash
mlx_lm.generate --model N8Programs/NextTerm-440M --prompt "1,2,3,"
```

### Hugging Face Transformers

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "N8Programs/NextTerm-440M"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "1,2,3,4,5,"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
    eos_token_id=[tokenizer.convert_tokens_to_ids(","), tokenizer.eos_token_id],
    pad_token_id=tokenizer.pad_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

For strict next-term evaluation, stop generation on *comma or EOS* and parse the
text before the first comma as the predicted integer.

## Reproducibility

This repository contains the local evaluation scripts and artifacts used for the
results above, including the small evaluation datasets needed to rerun them:

- `oeis_eval_mlx_neo.py` for OEIS-Eval-Neo with MLX batch generation.
- `arithmetic_eval.py` for arithmetic/quadratic/cubic/quartic continuation.
- `eval_m1_competition_mape_mlx.py` for M1 Competition 111 MAPE.
- `oeis_val_neo.jsonl` for OEIS-Eval-Neo.
- `m1_competition_111.jsonl` for M1 Competition 111.
- `eval_results.txt` for the compact result table.


The last three training checkpoints are available separately at
[N8Programs/NextTerm-440M-Checkpoints](https://huggingface.co/N8Programs/NextTerm-440M-Checkpoints).
The released `final_latest` checkpoint was trained for 14B tokens. Additionally, the checkpoint corresponding to the best val loss is available as well (although it is not included in the main results table as it was inferior on downstream eval performance).

The `.packed` files used for training are binary files containing the tokenized and augmented OEIS data - w/ tokens encoded as nibbles. A dedicate decoder is provided in this repo as `decode_packed_oeis.py`.

## Citation

```bibtex
@misc{nextterm440m2026,
  author       = {Nathan Breslow},
  title        = {NextTerm-440M: A Pretrained Transformer for Integer Sequence Prediction},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/N8Programs/NextTerm-440M}},
  note         = {440.5M parameter model trained on augmented OEIS data}
}
```

## Attribution

This model and dataset were trained and created using data from the
**On-Line Encyclopedia of Integer Sequences (OEIS)**.

- Source: https://oeis.org/
- License: Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0)
- OEIS End-User License Agreement: https://oeis.org/wiki/The_OEIS_End-User_License_Agreement