File size: 2,459 Bytes
d47573d 451a228 d47573d 451a228 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 | ---
license: cc-by-sa-4.0
library_name: transformers
tags:
- oeis
- qwen3
- causal-lm
- checkpoint
---
# NextTerm-440M Checkpoints
Transformers-compatible checkpoints from the OEIS NextTerm-440M run.
These checkpoints use a Qwen3-style causal LM architecture with a 16-token OEIS digit vocabulary. They were converted from the training checkpoints by remapping the custom interleaved RoPE basis into the Hugging Face / Qwen split-half RoPE basis, so they can be loaded directly with `AutoModelForCausalLM`.
## Checkpoints
| Folder | Tokens trained | Notes |
| --- | ---: | --- |
| `checkpoints/final_latest` | 13,999,999,995 | Final checkpoint; recommended default |
| `checkpoints/best_val` | 9,500,200,875 | Best validation-loss checkpoint |
| `checkpoints/checkpoint_tokens_012000258345` | 12,000,258,345 | Historical checkpoint |
| `checkpoints/checkpoint_tokens_012500265837` | 12,500,265,837 | Historical checkpoint |
| `checkpoints/checkpoint_tokens_013000266889` | 13,000,266,889 | Historical checkpoint |
| `checkpoints/checkpoint_tokens_013500289737` | 13,500,289,737 | Historical checkpoint |
## OEIS Vocab
The model is token-ID based; no text tokenizer is included.
| Token ID | Meaning |
| ---: | --- |
| `0`-`9` | decimal digits |
| `10` | negative sign |
| `11` | term separator |
| `12` | BOS |
| `13` | EOS |
| `14` | PAD |
| `15` | reserved |
For next-term generation, stop on any of `[11, 13, 14]`.
## Loading
```python
import torch
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"N8Programs/NextTerm-440M-Checkpoints",
subfolder="checkpoints/final_latest",
dtype=torch.bfloat16,
device_map="auto",
)
```
Example input IDs for the prefix `1, 2, 3, ...`:
```python
input_ids = torch.tensor([[12, 1, 11, 2, 11, 3, 11]], device=model.device)
out = model.generate(
input_ids,
max_new_tokens=192,
do_sample=False,
eos_token_id=[11, 13, 14],
pad_token_id=14,
)
```
## Evaluation Notes
OEIS Eval Neo excludes exact packed-sequence overlaps with the training data and uses `max_new_tokens=192`, which is sufficient for every answer in that eval set.
Known OEIS Eval Neo results:
| Checkpoint | Accuracy |
| --- | ---: |
| `final_latest` | 6545 / 19034 = 34.39% |
| `best_val` | 6477 / 19034 = 34.03% |
Each checkpoint folder includes an `oeis_checkpoint_meta.json` file with training tokens, source checkpoint path, and conversion details.
|