---
license: mit
library_name: pytorch
pipeline_tag: text-generation
tags:
- nested-learning
- hope
- test-time-learning
- test-time-training
- continual-learning
- byte-level
- attention-free
- recurrent
- arxiv:2512.24695
---

# HOPE — Nested Learning in PyTorch

The index card for [nested-learning-hope](https://github.com/smallhours19/nested-learning-hope),
an independent PyTorch reproduction of HOPE from **"Nested Learning: The Illusion
of Deep Learning Architectures"** ([arXiv:2512.24695](https://arxiv.org/abs/2512.24695),
Behrouz et al., Google Research). No official code or weights exist for the paper;
this is an unofficial reproduction at reduced scale.

HOPE is an attention-free language model whose weights change during the forward
pass (test-time learning): Self-Modifying Titans (Eq. 83-93, gated delta rule,
self-generated targets) plus a Continuum Memory System (Eq. 70-71, MLP levels
updated every 1/2/4 chunks by the accumulated task-loss gradient).

## The released recipe (reproducible in ~7 GPU-hours)

| | |
|---|---|
| Parameters | 30.6M (4 layers, d=512, chunk 512, no positional embedding) |
| Tokenizer | none — raw bytes (vocab 256) |
| Training data | FineWeb-Edu sample, 0.3B bytes, sequence length 2048 |
| Held-out validation | **1.721 bits/byte** |
| Needle recall (1K / 6.6K bytes) | **+2.59 / +2.10 bpb** vs control (n=40) |
| 8K extrapolation (trained at 2K) | flat, no horizon (1.66 → 1.96 bpb) |

Every number is reproducible from the GitHub repository (committed training
log, evaluation scripts, and an exactness test verifying the chunk-parallel
path equals the sequential recurrence to fp64 machine precision).

## Reproducing the model

No weights are hosted here (yet) — the checkpoint reproduces from the GitHub
repository in about 7 hours on one 16 GB GPU, and the committed training log
lets you verify your run against ours at step 100/200 before committing the
full budget:

```bash
git clone https://github.com/smallhours19/nested-learning-hope && cd nested-learning-hope
pip install -r requirements.txt
python scripts/download_data.py --out data/fineweb-edu --shards 3
python scripts/train_byte.py --data data/fineweb-edu --out runs/hope-byte
```

Where this implementation had to choose beyond the paper's text, every choice
is documented in
[DEVIATIONS.md](https://github.com/smallhours19/nested-learning-hope/blob/main/docs/DEVIATIONS.md);
stability devices the paper does not mention are cataloged in
[STABILITY.md](https://github.com/smallhours19/nested-learning-hope/blob/main/docs/STABILITY.md).

## Citation

```bibtex
@article{behrouz2025nested,
  title   = {Nested Learning: The Illusion of Deep Learning Architectures},
  author  = {Behrouz, Ali and Razaviyayn, Meisam and Zhong, Peilin and
             Mirrokni, Vahab},
  journal = {arXiv preprint arXiv:2512.24695},
  year    = {2025}
}
```