model card (frozen)

README.md

@@ -0,0 +1,195 @@
+---
+language: en
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+inference: false
+tags:
+- babylm
+- babylm-2026
+- strict-small
+- linear-attention
+- state-tracking
+- delta-rule
+- custom_code
+---
+
+# bind2_0
+
+> ## ⚠️ The official benchmark badge is NOT the point of this repo
+>
+> On the official BabyLM 2026 strict-small zero-shot surface this model is **statistically tied** with its
+> matched monolithic and bind1 controls (mean-4 excl. entity: **56.92** vs 56.60/56.63 at 23.9M; slightly
+> above the GPT-2 baseline on BLiMP, **66.11** vs 65.08). It wins nothing there, and that is part of the
+> finding.
+>
+> **What this stage actually shows** (three results, all kept):
+> 1. **The mechanism is trainable**: with direct-task training on a purpose-built synthetic swap-tracking
+>    task (n=800 per eval, 5-way, chance 0.20), the forced-bottleneck architecture reached **0.9988
+>    accuracy** — with a sharp grokking transition between 5M and 10M training tokens
+>    (0.179 → 0.969 → 0.996) — while matched monolithic, bind1-style, and no-binding controls stayed at
+>    chance (0.2125 / 0.1938 / 0.1938).
+> 2. **It does not emerge from plain LM pretraining**: after standard BabyLM strict-small pretraining,
+>    zero-shot give-verb state-tracking probes (n=60, chance 0.50) show **no model above chance**; this
+>    build (23.9M) scored 0.200 — significantly below chance, i.e. strong recency capture. The synthetic
+>    grok did not transfer.
+> 3. **The architecture costs ~nothing on general language** ("no tax, no win"): official zero-shot scores
+>    tied across architectures; raw LM perplexity is slightly worse than the monolithic control
+>    (11.0 vs 9.7 at 23.9M), as expected for a forced bottleneck.
+>
+> The full falsification-timeline context (what came before this stage and what it forced next) lives at
+> the hub: [`SecludedCorner/bind-evolution`](https://huggingface.co/datasets/SecludedCorner/bind-evolution).
+
+## Model description (family level)
+
+bind2_0 is a small causal LM (main branch = **23.9M params**; branch `27m` = **27M params**) combining:
+
+- **Delta-rule fast-weight memory**: gated delta-rule recurrent value dynamics (GatedDeltaNet), using the
+  third-party MIT-licensed [`flash-linear-attention` (fla)](https://github.com/fla-org/flash-linear-attention)
+  implementation as the core recurrent layer. The delta-rule/fast-weight design follows Yang, Kautz &
+  Hatamizadeh, *Gated Delta Networks: Improving Mamba2 with Delta Rule* (ICLR 2025, arXiv:2412.06464);
+  only the permissively licensed fla implementation is used here.
+- **A forced bottleneck**: attention is chunk-local; information can cross chunk boundaries **only**
+  through the recurrent state S. This makes the recurrent state the sole carrier of long-range bindings —
+  the design hypothesis under test at this stage.
+
+Later family members are not described here; see the hub for the family narrative.
+
+## Training data
+
+Official **BabyLM 2026 Strict-Small** corpus (the provided ~10M-word text-only corpus; no custom data).
+Training: 150M tokens over the 16.3M-token encoding (SEQ256, batch 16, vocab 16k), recurrent state reset
+per block. Final training perplexity: 11.0 (23.9M build), 10.8 (27M build); matched monolithic control: 9.7.
+
+## Results (official pipeline, strict-small zero-shot, single seed)
+
+Main branch (23.9M):
+
+| task | bind2_0 | mono control | bind1 control | GPT-2 baseline |
+|---|---:|---:|---:|---:|
+| BLiMP | 66.11 | 65.35 | 65.50 | 65.08 |
+| BLiMP supplement | 58.11 | 58.17 | 58.35 | 57.25 |
+| EWoK | 51.95 | 51.32 | 51.57 | — |
+| entity_tracking (filtered) | 19.02 | 21.16 | 19.22 | 21.07 |
+| COMPS | 51.49 | 51.55 | 51.11 | 51.81 |
+| mean(4, excl. entity) | **56.92** | 56.60 | 56.63 | — |
+
+Branch `27m` (27M; mono control at this tier is 27.4M):
+
+| task | bind2_0 | mono control | bind1 control | GPT-2 baseline |
+|---|---:|---:|---:|---:|
+| BLiMP | 65.14 | 64.35 | 66.68 | 65.08 |
+| BLiMP supplement | 60.81 | 58.55 | 60.90 | 57.25 |
+| EWoK | 51.16 | 50.70 | 51.90 | — |
+| entity_tracking (filtered) | 20.53 | 19.24 | 20.00 | 21.07 |
+| COMPS | 50.89 | 51.00 | 51.36 | 51.81 |
+| mean(4, excl. entity) | 57.00 | 56.15 | 57.71 | — |
+
+entity_tracking is ~chance for every model under the current filtered standard (non-discriminative), hence
+excluded from the mean. The mean-of-4 is NOT the official leaderboard "Overall" (which also weights GLUE,
+reading, AoA, and more). All numbers single-seed; the across-architecture spread (~1.5pp) is within seed
+noise.
+
+## Honest limitations — what this stage cannot do
+
+- **It does not track state zero-shot.** After plain LM pretraining, give-verb state-tracking probes are at
+  or below chance (0.200 at 23.9M = strong recency capture). Do not use this model expecting emergent
+  entity/state tracking.
+- **It does not beat its controls on the official benchmark.** Tied within noise; that is the honest
+  reading, not modesty.
+- **The synthetic grok required direct-task training** — it is evidence the bottleneck can force state into
+  the recurrent path, not evidence of a general capability.
+- Single seed per build; raw LM perplexity pays a small bottleneck tax (11.0 vs 9.7).
+
+## What this stage forced next
+
+The gap between "trainable in principle (0.9988 synthetic grok)" and "does not emerge from LM pretraining
+(chance zero-shot)" forced the next question on the ladder: split the confound — first prove the mechanism
+is *causally real at depth* under a pre-registered gate, separately from transfer. That question —
+including a preregistered NULL we report as NULL, and the causal evidence around it — is answered on the
+hub: [`SecludedCorner/bind-evolution`](https://huggingface.co/datasets/SecludedCorner/bind-evolution).
+
+## How to load
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+tok = AutoTokenizer.from_pretrained("SecludedCorner/bind2_0",
+                                    revision="<40-char commit SHA>")
+model = AutoModelForCausalLM.from_pretrained("SecludedCorner/bind2_0",
+                                             revision="<40-char commit SHA>",
+                                             trust_remote_code=True)
+```
+
+- `trust_remote_code=True` is **required** for the model (the config's `auto_map` points at the inlined
+  `modeling_babylm.py` shipped in this repo). The tokenizer loads without it.
+- **A CUDA GPU with a working Triton is required for inference.** The fla 0.5.1 GatedDeltaNet path
+  dispatches to Triton kernels; on CPU-only environments the forward pass fails at kernel launch
+  (`RuntimeError: 0 active drivers`). The checkpoint itself loads fine on CPU; only the forward pass needs
+  a GPU.
+
+## Dependency pins
+
+Exact versions the checkpoints were trained/exported/validated with (Python 3.11.15; also shipped as
+`requirements_pins.txt` in this repo):
+
+```
+torch==2.12.1+cu126
+transformers==5.13.0
+triton-windows==3.7.1.post27
+flash-linear-attention==0.5.1
+fla-core==0.5.1
+safetensors==0.8.0
+numpy==2.4.6
+```
+
+Notes:
+
+- **fla** is required at runtime: the inlined modeling code lazily imports `fla.layers.GatedDeltaNet`,
+  which this architecture instantiates. Installed from PyPI as release **0.5.1** (no commit pin — the
+  environment records the plain PyPI release; `fla-core` 0.5.1 comes with it).
+- **triton**: the training/export environment is Windows and uses the `triton-windows` fork
+  (3.7.1.post27); on Linux use the upstream `triton` matching your torch build.
+- **torch** build is CUDA 12.6 (`+cu126`); pick the equivalent build for your platform.
+
+## Export fidelity & known-defect disclosure
+
+**Export fidelity (verified twice):**
+
+- At grid-evaluation time (2026-07-12) the HF export was verified logit-identical to the training model
+  (wrapper-vs-original logit diff = 0.00e+00).
+- On 2026-07-15 a dedicated round-trip validation re-checked both exports, as they sit on disk, against the
+  original training checkpoints: **all 186 weight tensors bitwise identical** (max abs diff 0.0, no
+  missing/extra keys) and **logits bitwise identical** (max abs diff 0.00e+00 across 4 deterministic
+  batches of 8×128 tokens, fp32, passing at both atol 1e-4 and atol 1e-5) for **both** the 23.9M and 27M
+  builds. Caveat, disclosed: that re-check ran on CPU, where fla's Triton kernels cannot execute, so three
+  fla components were replaced by math-equivalent pure-PyTorch implementations applied identically to both
+  sides. It therefore validates **export fidelity** (weights and module wiring survive
+  `.pt → safetensors → AutoModelForCausalLM` exactly), not Triton-kernel numerics; a GPU re-run with stock
+  kernels remains the gold check.
+
+**Known defect — `attention_mask` is accepted but ignored:**
+
+- The exported wrapper accepts `attention_mask` in `forward()` and never uses it — on the causal-LM path
+  and, for this architecture, on the AutoModel (sequence-classification) path as well. Empirically,
+  `attention_mask=ones`, `=zeros`, and omitted all produce bitwise-identical logits on both builds.
+- **Consequence:** in a batch, right-padding is silently attended over as real tokens — **batched padded
+  inference gives wrong results.** Run unbatched, or length-sorted/unpadded. Per-example inference is
+  unaffected; the published zero-shot numbers above came from the per-example evaluation setting and are
+  unaffected by this defect.
+
+## How to cite this model
+
+Always cite at a pinned revision: pass `revision="<40-char commit SHA>"` to `from_pretrained`, or use the
+`/tree/<sha>` URL form. Authoritative per-branch SHAs are recorded at push time in the project
+`PUBLISH_LEDGER`; the final SHAs are noted in a dated addendum below after publication.
+
+## Card freeze policy
+
+**The body of this card is frozen at publish.** Any later information (including the final commit SHAs and
+resolved links) is added only as clearly dated addendum sections below this line — the text above is never
+silently edited.
+
+## Branches
+
+- `main` — 23.9M-parameter build (the primary artifact)
+- `27m` — 27M-parameter build (same architecture and recipe, wider)