| --- |
| license: mit |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - nested-learning |
| - hope |
| - test-time-learning |
| - test-time-training |
| - continual-learning |
| - byte-level |
| - attention-free |
| - recurrent |
| - arxiv:2512.24695 |
| --- |
| |
| # HOPE — Nested Learning in PyTorch |
|
|
| The index card for [nested-learning-hope](https://github.com/smallhours19/nested-learning-hope), |
| an independent PyTorch reproduction of HOPE from **"Nested Learning: The Illusion |
| of Deep Learning Architectures"** ([arXiv:2512.24695](https://arxiv.org/abs/2512.24695), |
| Behrouz et al., Google Research). No official code or weights exist for the paper; |
| this is an unofficial reproduction at reduced scale. |
|
|
| HOPE is an attention-free language model whose weights change during the forward |
| pass (test-time learning): Self-Modifying Titans (Eq. 83-93, gated delta rule, |
| self-generated targets) plus a Continuum Memory System (Eq. 70-71, MLP levels |
| updated every 1/2/4 chunks by the accumulated task-loss gradient). |
|
|
| ## The released recipe (reproducible in ~7 GPU-hours) |
|
|
| | | | |
| |---|---| |
| | Parameters | 30.6M (4 layers, d=512, chunk 512, no positional embedding) | |
| | Tokenizer | none — raw bytes (vocab 256) | |
| | Training data | FineWeb-Edu sample, 0.3B bytes, sequence length 2048 | |
| | Held-out validation | **1.721 bits/byte** | |
| | Needle recall (1K / 6.6K bytes) | **+2.59 / +2.10 bpb** vs control (n=40) | |
| | 8K extrapolation (trained at 2K) | flat, no horizon (1.66 → 1.96 bpb) | |
|
|
| Every number is reproducible from the GitHub repository (committed training |
| log, evaluation scripts, and an exactness test verifying the chunk-parallel |
| path equals the sequential recurrence to fp64 machine precision). |
|
|
| ## Reproducing the model |
|
|
| No weights are hosted here (yet) — the checkpoint reproduces from the GitHub |
| repository in about 7 hours on one 16 GB GPU, and the committed training log |
| lets you verify your run against ours at step 100/200 before committing the |
| full budget: |
|
|
| ```bash |
| git clone https://github.com/smallhours19/nested-learning-hope && cd nested-learning-hope |
| pip install -r requirements.txt |
| python scripts/download_data.py --out data/fineweb-edu --shards 3 |
| python scripts/train_byte.py --data data/fineweb-edu --out runs/hope-byte |
| ``` |
|
|
| Where this implementation had to choose beyond the paper's text, every choice |
| is documented in |
| [DEVIATIONS.md](https://github.com/smallhours19/nested-learning-hope/blob/main/docs/DEVIATIONS.md); |
| stability devices the paper does not mention are cataloged in |
| [STABILITY.md](https://github.com/smallhours19/nested-learning-hope/blob/main/docs/STABILITY.md). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{behrouz2025nested, |
| title = {Nested Learning: The Illusion of Deep Learning Architectures}, |
| author = {Behrouz, Ali and Razaviyayn, Meisam and Zhong, Peilin and |
| Mirrokni, Vahab}, |
| journal = {arXiv preprint arXiv:2512.24695}, |
| year = {2025} |
| } |
| ``` |
|
|
