license: mit
library_name: pytorch
pipeline_tag: text-generation
tags:
- nested-learning
- hope
- test-time-learning
- test-time-training
- continual-learning
- byte-level
- attention-free
- recurrent
- arxiv:2512.24695
HOPE — Nested Learning in PyTorch
The index card for nested-learning-hope, an independent PyTorch reproduction of HOPE from "Nested Learning: The Illusion of Deep Learning Architectures" (arXiv:2512.24695, Behrouz et al., Google Research). No official code or weights exist for the paper; this is an unofficial reproduction at reduced scale.
HOPE is an attention-free language model whose weights change during the forward pass (test-time learning): Self-Modifying Titans (Eq. 83-93, gated delta rule, self-generated targets) plus a Continuum Memory System (Eq. 70-71, MLP levels updated every 1/2/4 chunks by the accumulated task-loss gradient).
The released recipe (reproducible in ~7 GPU-hours)
| Parameters | 30.6M (4 layers, d=512, chunk 512, no positional embedding) |
| Tokenizer | none — raw bytes (vocab 256) |
| Training data | FineWeb-Edu sample, 0.3B bytes, sequence length 2048 |
| Held-out validation | 1.721 bits/byte |
| Needle recall (1K / 6.6K bytes) | +2.59 / +2.10 bpb vs control (n=40) |
| 8K extrapolation (trained at 2K) | flat, no horizon (1.66 → 1.96 bpb) |
Every number is reproducible from the GitHub repository (committed training log, evaluation scripts, and an exactness test verifying the chunk-parallel path equals the sequential recurrence to fp64 machine precision).
Reproducing the model
No weights are hosted here (yet) — the checkpoint reproduces from the GitHub repository in about 7 hours on one 16 GB GPU, and the committed training log lets you verify your run against ours at step 100/200 before committing the full budget:
git clone https://github.com/smallhours19/nested-learning-hope && cd nested-learning-hope
pip install -r requirements.txt
python scripts/download_data.py --out data/fineweb-edu --shards 3
python scripts/train_byte.py --data data/fineweb-edu --out runs/hope-byte
Where this implementation had to choose beyond the paper's text, every choice is documented in DEVIATIONS.md; stability devices the paper does not mention are cataloged in STABILITY.md.
Citation
@article{behrouz2025nested,
title = {Nested Learning: The Illusion of Deep Learning Architectures},
author = {Behrouz, Ali and Razaviyayn, Meisam and Zhong, Peilin and
Mirrokni, Vahab},
journal = {arXiv preprint arXiv:2512.24695},
year = {2025}
}