| # Implementation notes | |
| This release implements parallel constrained inference using unchanged LFM2.5-350M weights. No training, optimizer, fine-tuning, or parameter update is performed. It does not reproduce TypeSafe's proprietary Jev training or internals. | |
| ## LFM2.5 compatibility | |
| The pinned [base config](https://huggingface.co/LiquidAI/LFM2.5-350M/blob/9e6c6ccf47cd318696e137d381a7ded8fe4df09f/config.json) has 16 layers: six full-attention layers and ten short-convolution layers, with convolution cache length 3. Merely copying `keys` and `values` would lose convolution history or alias mutable state. | |
| With Transformers 5.17.0, LFM2 uses `DynamicCache` containing both attention and linear-attention cache layer objects. `LinearAttentionLayer.conv_states` stores its short-convolution history. Generic `batch_repeat_interleave` does not handle these convolution layer objects. We deep-copy the cache and call `reorder_cache` with repeated index zero; its layer-specific implementations copy both KV tensors and convolution states along the batch dimension. Convolution states are not shared as writable expanded views. | |
| A shared native chat-template prompt includes the complete JSON schema, all candidate values, the context, and an assistant opening `{\n`. For each field and candidate, we append a field suffix followed by a complete serialized JSON value and a newline delimiter. All such branches are right-padded and evaluated in one causal batch from independent copies of the shared cache. Padded state is never reused. We sum FP32 log-softmax scores over the candidate's complete token sequence, including its quotes and delimiter, and select the largest score per field. The delimiter makes candidate endings explicit; e.g. `express` and `express plus` are separate sequences. Ties choose the first listed candidate. | |
| This uses two model calls regardless of candidate length, but FLOPs, memory, and batch width grow with total candidate count and length. Full-sequence scores have length and tokenization biases; they are not semantic confidence estimates. Tokenization is deliberately segmented at the field/value boundary, so the score is for that specific token sequence, not a sum over all tokenizations of a string. | |
| ## Correctness and benchmark methodology | |
| `tests/test_engine.py` checks hybrid cached branches against full-context forwards, checks that base KV and convolution state remain unchanged, compares every candidate likelihood against an uncached reference (including overlapping multi-token strings), and tests strict JSON/schema evaluation. FP16 scores need numerical tolerances across batching and device kernel order. All actual device validation outputs are retained. | |
| The baseline uses greedy `generate`, the same shared prompt prefix and complete schema, the same model revision, dtype and eager attention implementation, without grammar constraints or output repair. It restores the common prefilled brace before strict parsing. Whitespace formatting is not forced to be verbose. Both methods use the same model and precision settings with deterministic decoding. | |
| The model's field decisions remain independent: other selected fields are not fed back into a branch. Only a flat, closed object with required boolean or string-enum fields is supported. Cross-field constraints, nested objects, optional fields, arbitrary numbers, and free text are rejected or unsupported. Programmatic serialization guarantees syntax and this restricted schema's membership for successful calls, not truth, calibrated probability, relational consistency, or immunity to model errors. | |