UniRes

Universal Residual Predictor for Neural Network Weight Reconstruction

UniRes is a universal residual network that reconstructs approximate higher-precision (FP32 / FP64) weights from low-precision (BF16 / FP16) tensors of arbitrary shape and size. It operates completely architecture-agnostically and generalises across model families never seen during training (Llama, BERT, T5, Gemma, VAE, CLIP, etc.).

Model Description

Given a low-precision tensor (W_{\text{low}}) UniRes predicts a residual (R) such that

[ \hat{W} = W_{\text{low}} + R ]

The final reconstructed weights are emitted in the user-selected precision (FP32 or FP64). The network accepts tensors of any rank and dimension; no architecture-specific adapters are required.

Evaluation (Completely Out-of-Distribution)

Ground-truth vs BF16 cast

• Tensors: 194
• Elements: 126 892 531
• Total absolute error: (6.6856248102 \times 10^{2})
• Maximum absolute error: (3.4952163696 \times 10^{-3})

Ground-truth vs UniRes prediction

• Total absolute error: (1.4362314089 \times 10^{3})
• Maximum absolute error: (3.4807920456 \times 10^{-3})

Per-element comparison

• UniRes better than BF16: 10 539 994 elements (8.306 %)
• BF16 better than UniRes: 116 350 729 elements (91.692 %)
• Equal: 1 808 elements (0.001 %)

Ratio of total absolute errors: UniRes / BF16 = 2.148×

While UniRes does not improve aggregate absolute error, it produces a strictly lower absolute deviation on more than ten million individual elements.

Interpretation

The minority of element-wise wins is consistent with the persistence of correlated Gaussian structure in neural-network weight matrices well beyond initialisation (Hirst & Ramgoolam, 2026). These residual correlations—local spatial structure, higher-order moments and permutation-invariant dependencies—transfer to a limited degree across architectures, allowing a universal residual predictor to capture useful signal on a non-negligible fraction of elements.

Citation

If you use UniRes, please cite:

@article{unires2026,
  title={Persistent Correlated Structure in Neural Network Weights Enables Partial Out-of-Distribution Residual Prediction from Low-Precision Representations},
  author={UniRes Authors},
  year={2026}
}

@article{hirst2026gaussianity,
  title={Approximate Gaussianity Beyond Initialisation in Neural Networks},
  author={Hirst, Edward and Ramgoolam, Sanjaye},
  journal={Machine Learning: Science and Technology},
  volume={7},
  number={3},
  pages={035038},
  year={2026},
  eprint={2510.05218}
}