Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,75 @@
+---
+license: mit
+library_name: pytorch
+pipeline_tag: text-generation
+tags:
+- nested-learning
+- hope
+- test-time-learning
+- test-time-training
+- continual-learning
+- byte-level
+- attention-free
+- recurrent
+- arxiv:2512.24695
+---
+
+# HOPE — Nested Learning in PyTorch
+
+The index card for [nested-learning-hope](https://github.com/smallhours19/nested-learning-hope),
+an independent PyTorch reproduction of HOPE from **"Nested Learning: The Illusion
+of Deep Learning Architectures"** ([arXiv:2512.24695](https://arxiv.org/abs/2512.24695),
+Behrouz et al., Google Research). No official code or weights exist for the paper;
+this is an unofficial reproduction at reduced scale.
+
+HOPE is an attention-free language model whose weights change during the forward
+pass (test-time learning): Self-Modifying Titans (Eq. 83-93, gated delta rule,
+self-generated targets) plus a Continuum Memory System (Eq. 70-71, MLP levels
+updated every 1/2/4 chunks by the accumulated task-loss gradient).
+
+## The released recipe (reproducible in ~7 GPU-hours)
+
+| | |
+|---|---|
+| Parameters | 30.6M (4 layers, d=512, chunk 512, no positional embedding) |
+| Tokenizer | none — raw bytes (vocab 256) |
+| Training data | FineWeb-Edu sample, 0.3B bytes, sequence length 2048 |
+| Held-out validation | **1.721 bits/byte** |
+| Needle recall (1K / 6.6K bytes) | **+2.59 / +2.10 bpb** vs control (n=40) |
+| 8K extrapolation (trained at 2K) | flat, no horizon (1.66 → 1.96 bpb) |
+
+Every number is reproducible from the GitHub repository (committed training
+log, evaluation scripts, and an exactness test verifying the chunk-parallel
+path equals the sequential recurrence to fp64 machine precision).
+
+## Reproducing the model
+
+No weights are hosted here (yet) — the checkpoint reproduces from the GitHub
+repository in about 7 hours on one 16 GB GPU, and the committed training log
+lets you verify your run against ours at step 100/200 before committing the
+full budget:
+
+```bash
+git clone https://github.com/smallhours19/nested-learning-hope && cd nested-learning-hope
+pip install -r requirements.txt
+python scripts/download_data.py --out data/fineweb-edu --shards 3
+python scripts/train_byte.py --data data/fineweb-edu --out runs/hope-byte
+```
+
+Where this implementation had to choose beyond the paper's text, every choice
+is documented in
+[DEVIATIONS.md](https://github.com/smallhours19/nested-learning-hope/blob/main/docs/DEVIATIONS.md);
+stability devices the paper does not mention are cataloged in
+[STABILITY.md](https://github.com/smallhours19/nested-learning-hope/blob/main/docs/STABILITY.md).
+
+## Citation
+
+```bibtex
+@article{behrouz2025nested,
+  title   = {Nested Learning: The Illusion of Deep Learning Architectures},
+  author  = {Behrouz, Ali and Razaviyayn, Meisam and Zhong, Peilin and
+             Mirrokni, Vahab},
+  journal = {arXiv preprint arXiv:2512.24695},
+  year    = {2025}
+}
+```