here comes something even more awesome 👍 thank you very much for all the great work you are giving us
It's still in its early stages; there are some interface or engine/functionality errors. I've been testing and debugging for 3 days, so any feedback is welcome.
Keeping router weights in F32 avoids quantizing those tensors, while upstream quantization can still change the hidden states they receive. In the LFM2.5 MiniPlus card, does 'zero routing drift' mean measured expert-selection agreement with BF16, or that the router tensors are unchanged? That distinction would help interpret the precision map.
The main idea is to preserve the original expert-selection mechanism by keeping the routers and their biases in F32. Given identical hidden states, the preserved routers will select the same experts as the original model.
The quantized experts can still introduce small differences in their outputs, which can affect the final token probabilities and occasionally change the generated token. However, a different token doesn't necessarily mean an incorrect one. Even with aggressive quantization such as IQ3_XXS, the model can still produce valid and coherent outputs.
The goal is to preserve routing fidelity while minimizing the impact of expert quantization on overall generation quality.
