| # Full precision Mamba2-8B Resurface control |
|
|
| Declared 2026-09-28 before fitting this repository's adapter. The purpose is to |
| measure the effect of the same readout adapter and training budget on NVIDIA's |
| uncompressed pure Mamba2-8B. The E8/W5 experiment is a historical comparator, |
| not a training dependency. All metrics are reported at the actually measured |
| precision and protocol, without assuming equality to native Megatron BF16. |
|
|
| ## Frozen model and intervention |
|
|
| Load `nvidia/mamba2-8b-3t-4k` revision |
| `b915550c63ba9359f88f44d1f6a600d85af27302`, checkpoint SHA256 |
| `47c2766f6aad89d73beafbeaecb334aab902d7370906d081764a90bb7a8bbbcb` |
| and tokenizer SHA256 |
| `5862e2f71caf762bc9845662be5fec2867deb58d874568235a02a36c5111cd09`. |
| Cast the 507 original BF16 tensors to FP16 in the native state-spaces Mamba2 |
| runtime; freeze all 8,236,999,680 base parameters. Verify strict source tensor |
| mapping, model geometry and no shared embedding/output storage. |
|
|
| Use the same post-D adapter as the compressed experiment at all 56 native |
| gated RMSNorm inputs: `y + sigmoid(w dot u+b) * g*(V@y)` across 128 heads. |
| Its 224 tensors contain 1,154,104 FP32 trainable master parameters, exported |
| as FP16 for inference. Initialize V=0, g=1, w=0, b=-4; use soft sigmoid at |
| train and eval, no task switch, EMA, or new recurrent cache. The adapter is |
| external to the frozen base; check its identities and absence of base grads. |
|
|
| ## Training |
|
|
| Reuse the *same exact data generator contract and seeds* as the compressed |
| adapter: 1,536 TRAIN numeric bindings over three templates and N=16/64, |
| disjoint six-digit key/value intervals; full 256K CE on every answer suffix |
| token. Use seed 2026092803 and `torch.randperm(1536)` once for example order. |
| Each step pairs one MK example with a 512-token segment from the already |
| prepared 448 WikiText-2 TRAIN windows. Their historical manifest and file |
| SHA256 are pinned by the train command. Heldout and validation windows are |
| not included in training. Reusing this fixed TRAIN text gives the control the |
| same training exposures as the compressed arm. |
|
|
| Use a second, independently loaded **uncompressed FP16** model as the frozen |
| prose teacher. This is the direct counterpart to the compressed experiment's |
| teacher, which was its own unadapted compressed base. The fixed per-step loss is |
| `MK answer CE + 0.5 prose CE + 0.5 KL(own unadapted teacher || student) + 3 C`, |
| where C is the same prose-only router closure, with budget 0.006 and excess |
| coefficient 10. Temperature 1, 511 prose targets per step, 64-token staged |
| head chunks, and one optimizer update after both task gradients. |
|
|
| AdamW FP32 masters: V/g LR 1e-4, w/b LR 3e-4, betas (.9,.999), eps 1e-8, |
| weight decay 0, clip norm 1. Multiplicative schedule |
| `0.1+0.9*(1+cos(pi*j/1535))/2` at successful index j. Native FP16 forward, |
| block checkpointing, gradient scale 1024 with growth interval 2000. Overflow |
| retries the exact same pair without optimizer/master update, at most 8 retries |
| and 1,544 total attempts. Run 1,536 **successful** updates; the final step is |
| the only candidate. Small GPU smokes are discarded before formal training. |
|
|
| ## Evaluation and interpretation |
|
|
| After training, restore the actual serialized FP16 adapter onto the same |
| uncompressed base. Evaluate baseline, adapter enabled and restored baseline |
| in a single process, with identical tokenizer, MK prompts and full WikiText-2 |
| validation windows. For MK use full 256K greedy generation up to 12 tokens, |
| first standalone six-digit number, normal and target-removed controls, native |
| prefill plus recurrent decode with fresh FP16 cache. For PPL use all 130 |
| nonoverlapping reset windows and all 264,764 next-token targets. Record all |
| raw scores, differences and file hashes; no early selection or task switch. |
|
|
| The earlier compressed arm's independent CONFIRM set has now been observed and |
| uses the same three template families. Running this source control on it gives |
| a matched historical comparison, **not a newly untouched holdout**. The |
| WikiText-2 validation text also informed earlier development. We will not |
| promote a source+adapter result to a general recall conclusion without new |
| templates, longer distances and a truly untouched corpus. |
|
|
| Report all four observed arms explicitly: source FP16, source+adapter, |
| compressed/readapted, compressed/readapted+adapter. The source and compressed |
| teachers differ by construction. Report PPL and recall together; any cross-arm |
| claim remains limited by shared test protocols and the historical numerical |
| replay discrepancy in the compressed project. |
|
|
