Model save

README.md

@@ -1,312 +1,110 @@
 ---
-language: en
-license: apache-2.0
+library_name: transformers
 tags:
-- causal-lm
-- research
-- fp8
-- attention
-- normalization
-- neollm
-- pace
-datasets:
-- HuggingFaceFW/fineweb-edu
+- generated_from_trainer
+model-index:
+- name: NeoLLM
+  results: []
 ---
 
-# NeoLLM
-
-NeoLLM is a **135 M parameter** decoder-only language model trained from scratch on
-[FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) in **FP8**
-precision, completing training in approximately **6 hours** on a single NVIDIA RTX 5090.
-It integrates a collection of recently published attention and normalization techniques
-into a single architecture, with the goal of studying how they interact during
-pretraining. The model is actively being developed and the current checkpoint represents
-an intermediate training state.
-
-> **Author / contact:** [@Kyokopom](https://x.com/Kyokopom) on X
-> **Repository:** [KitsuVp/NeoLLM](https://huggingface.co/KitsuVp/NeoLLM)
-
----
-
-## Architecture
-
-NeoLLM is a decoder-only transformer with the following configuration:
-
-| Parameter | Value |
-|---|---|
-| Hidden size | 512 |
-| Layers | 12 |
-| Attention heads | 8 |
-| KV heads (GQA) | 4 |
-| Head dim | 64 |
-| Intermediate size | 1536 |
-| Vocabulary | Qwen3 tokenizer (64,402 tokens) |
-| Context length | 512 tokens |
-
-### Parameter breakdown
-
-| Parameter bucket | Count |
-|---|---|
-| **Total parameters** | 116.22M (116,216,184) |
-| **Embedding parameters** (tied) | 32.97M (32,973,824) |
-| **Non-embedding parameters** | 83.24M (83,242,360) |
-| **Effective trainable parameters** | 116.22M (116,216,184) |
-
-> Weight tying is **enabled**: the input embedding matrix and the language-model head
-> share the same parameters, so the effective trainable budget is
-> `total − embed = 83.24M`.
-
-### Integrated techniques
-
-NeoLLM combines architecture modules, optional auxiliary objectives, and
-training-time optimizer/stability components from the following papers.
-
-**Embedding and token representation**
-
-- **Learnable Multipliers** ([arXiv:2601.04890](https://arxiv.org/abs/2601.04890)) — Adds
-  per-row and per-column learnable scalar parameters to selected matrix layers and, when
-  enabled, embeddings.
-- **Leviathan** ([arXiv:2601.22040](https://arxiv.org/abs/2601.22040)) — Optional
-  continuous token embedding generator that can replace the discrete input lookup table.
-- **KHRONOS** ([arXiv:2505.13315](https://arxiv.org/abs/2505.13315)) — Kernel/basis
-  reference used by the Leviathan continuous token generator implementation.
-- **JTok / JTok-M** ([arXiv:2602.00800](https://arxiv.org/abs/2602.00800)) — Optional
-  token-indexed self-modulation surfaces over Leviathan coordinates.
-- **Spelling Bee Embeddings** ([arXiv:2601.18030](https://arxiv.org/abs/2601.18030)) —
-  Augments token embeddings with character-level spelling information.
-- **Token Embedding Manifold analysis** ([arXiv:2504.01002](https://arxiv.org/abs/2504.01002)) —
-  Reference motivation for treating token embeddings as structured objects rather than
-  unconstrained lookup rows.
-
-**Attention, positions, and output projection**
-
-- **FAN** ([arXiv:2502.21309](https://arxiv.org/abs/2502.21309)) — Fourier Analysis Networks.
-  A portion of the projection channels are dedicated to periodic cosine/sine features.
-- **MEA** ([arXiv:2601.19611](https://arxiv.org/abs/2601.19611)) — Explicit Multi-head
-  Attention. Adds small learnable interaction matrices between attention heads for K and V.
-- **LUCID** ([arXiv:2602.10410](https://arxiv.org/abs/2602.10410)) — Applies a learned
-  lower-triangular preconditioner to V before attention, decorrelating value representations
-  across positions.
-- **Affine-Scaled Attention** ([arXiv:2602.23057](https://arxiv.org/abs/2602.23057)) — Adds
-  two learnable per-head scalars (α and β) to the softmax weights:
-  `[α·softmax(QKᵀ) + β]·V`.
-- **XSA** ([arXiv:2603.09078](https://arxiv.org/abs/2603.09078)) — Exclusive Self Attention.
-  After computing attention, removes the component of the output aligned with the token's
-  own value vector.
-- **Directional Routing** ([arXiv:2603.14923](https://arxiv.org/abs/2603.14923)) — Each head
-  learns K=4 directions in the output space; a learned router suppresses the attention output
-  along each direction per input.
-- **Gated Attention** ([arXiv:2505.06708](https://arxiv.org/abs/2505.06708)) — A sigmoid gate
-  is applied to the attention output before the output projection, introducing non-linearity
-  and preventing attention sinks.
-- **Momentum Attention** ([arXiv:2411.03884](https://arxiv.org/abs/2411.03884)) — Modifies Q
-  and K by subtracting a fraction of the previous position's Q and K values (causal
-  first-difference).
-- **Interleaved Head Attention / IHA** ([arXiv:2602.21371](https://arxiv.org/abs/2602.21371)) —
-  Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns
-  per original head.
-- **REPO** ([arXiv:2512.14391](https://arxiv.org/abs/2512.14391)) — Context re-positioning
-  module that learns contextual position coordinates above a configurable start layer.
-- **GRAPE** ([arXiv:2512.07805](https://arxiv.org/abs/2512.07805)) — Group representational
-  position encoding used by the REPO-GRAPE positional path.
-- **GOAT priors** ([arXiv:2601.15380](https://arxiv.org/abs/2601.15380)) — Optional
-  factorized attention log-prior channels inspired by trainable attention priors.
-- **Hadamard output projection** ([arXiv:2603.08343](https://arxiv.org/abs/2603.08343)) —
-  Replaces dense attention output projection with a structured Hadamard transform plus
-  lightweight scaling.
-
-**Normalization, residual flow, and MLP**
-
-- **SeeDNorm** ([arXiv:2510.22777](https://arxiv.org/abs/2510.22777)) — Applied to Q and K
-  projections. Dynamically rescales normalization from the input's own statistics.
-- **LayerNorm Scaling / LNS** ([arXiv:2502.05795](https://arxiv.org/abs/2502.05795)) — Each
-  layer's output is scaled by 1/√ℓ where ℓ is the layer index.
-- **GPAS** ([arXiv:2506.22049](https://arxiv.org/abs/2506.22049)) — Gradient-Preserving
-  Activation Scaling for residual junctions.
-- **PolyNorm** ([arXiv:2602.04902](https://arxiv.org/abs/2602.04902)) — Replaces the standard
-  MLP activation with normalized linear, quadratic, and cubic branches.
-- **SimpleGPT** ([arXiv:2602.01212](https://arxiv.org/abs/2602.01212)) — Second-order
-  geometry-inspired normalization strategy applied inside MLP projections.
-- **StackMemory / STACKTRANS** ([NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh)) —
-  Optional differentiable hidden-state stack between decoder layers.
-- **Attention Residuals / AttnRes** ([arXiv:2603.15031](https://arxiv.org/abs/2603.15031)) —
-  Optional learned depth-wise aggregation over previous layer outputs or block summaries.
-- **LAUREL** ([arXiv:2411.07501](https://arxiv.org/abs/2411.07501)) — Optional learned
-  augmented residual layer with residual-weight and low-rank variants.
-
-**Training objectives and training-time regularizers**
-
-- **TWEO** ([arXiv:2511.23225](https://arxiv.org/abs/2511.23225)) — Optional
-  Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly
-  training.
-- **NITP** ([arXiv:2605.24956](https://arxiv.org/abs/2605.24956)) — Optional Next Implicit
-  Token Prediction auxiliary objective using shallow-layer implicit token targets and a
-  cosine loss.
-- **NextLat** ([arXiv:2511.05963](https://arxiv.org/abs/2511.05963)) — Optional next-latent
-  prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL.
-
-**Optimizer and training stability**
-
-- **Conda** ([arXiv:2509.24218](https://arxiv.org/abs/2509.24218)) —
-  Column-Normalized Adam optimizer path used by the training script.
-- **Cautious Weight Decay** ([arXiv:2510.12402](https://arxiv.org/abs/2510.12402)) —
-  Sign-selective weight decay variant used by the custom optimizer logic.
-- **Correction of Decoupled Weight Decay** ([arXiv:2512.08217](https://arxiv.org/abs/2512.08217)) —
-  Adapts decoupled weight decay during learning-rate decay.
-- **AdamHD** ([arXiv:2511.14721](https://arxiv.org/abs/2511.14721)) —
-  Decoupled Huber decay regularization reference used by the optimizer.
-- **GradientStabilizer** ([arXiv:2502.17055](https://arxiv.org/abs/2502.17055)) —
-  Optional threshold-free gradient magnitude stabilizer.
-- **PACE** ([arXiv:2606.25086](https://arxiv.org/abs/2606.25086)) —
-  Optional iterate-average controller that trains for the EMA model returned at evaluation
-  and final serialization. The Conda-basis adaptation and its difference from AdamW are
-  documented below.
-
----
-
-### PACE integration and AdamW-reference differences
-
-PACE follows Au and Block's returned-model objective: the live weights are pulled toward a
-power-law EMA with a clipped per-coordinate gain, and evaluation/final serialization use that
-EMA estimator.
-
-- **Reference AdamW rule:** the gain uses AdamW's original-coordinate diagonal
-  second moment, `eta * c * (1+t)^(-kappa) / (sqrt(v_hat) + eps)`.
-- **NeoLLM Conda rule (`mode=conda`):** for projected 2-D tensors, both the EMA
-  displacement and `v_hat` are represented in Conda's cached SVD basis. The control is
-  projected back after applying the diagonal gain. This is a deliberate change from AdamW
-  required to avoid mixing incompatible coordinate systems.
-- **Optional exact AdamW pullback geometry (`mode=adamw`):** an additional
-  original-coordinate second moment is maintained for projected matrices. The live optimizer
-  step remains Conda.
-- **Conda scale:** in `mode=conda`, Conda's matrix-update scale multiplies the unsaturated
-  gain automatically because it is part of the effective Conda preconditioner. AdamW has no
-  corresponding scale.
-- **Fixed algorithm internals:** the EMA is stored in FP32, the gain is clipped at `1`, and PACE
-  reuses each Conda group’s numerical epsilon. These are not exposed as independent switches.
-- **Minimal modes:** `use_pace=False` is plain Conda; `use_pace=True, c=0` is Conda+EMA;
-  `use_pace=True, c>0` is complete PACE.
-- **Ordering:** PACE runs only after Conda, CWD/CHD, and weight-decay correction have fully
-  updated the live weights.
-- **Disabled guarantee:** with `use_pace=False`, no PACE state is allocated and no existing
-  Conda arithmetic or parameter update is changed.
-- **Checkpoint policy:** resumable internal checkpoints retain live weights and complete optimizer
-  state, while evaluation and the final returned/Hub model always use the EMA when PACE is active.
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
-Current run: **disabled; no EMA state, auxiliary moment, or pullback is allocated**.
-
----
-
-## Training
-
-| Setting | Value |
-|---|---|
-| Dataset | FineWeb-Edu (sample-10BT) |
-| Tokens seen | ~1.54B (46,875 steps × batch 64 × length 512) |
-| Precision | FP8 native (E4M3 weights/activations, E5M2 gradients) + BF16 fallback |
-| Optimizer | Conda (PACE disabled) |
-| PACE | disabled; no EMA state, auxiliary moment, or pullback is allocated |
-| Learning rate | 6e-04 with linear warmup (10 % of steps) |
-| Weight decay | 0.1 |
-| Training time | ~3h 47m |
-| Hardware | NVIDIA RTX 5090 (single GPU) |
-
-### Training curve
-
-| Step | Train Loss | Val Loss |
-|---|---|---|
-| 5,000 | 4.189 | 4.103 |
-| 10,000 | 3.875 | 3.798 |
-| 15,000 | 3.777 | 3.692 |
-| 20,000 | 3.732 | 3.643 |
-| 25,000 | 3.704 | 3.624 |
-| 30,000 | 3.681 | 3.594 |
-| 35,000 | 3.665 | 3.573 |
-| 40,000 | 3.580 | 3.503 |
-| 45,000 | 3.537 | 3.453 |
-| 46,875 | — | 3.446 |
-
----
-
-## Limitations
-
-- **Token budget** — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks
-  will improve with more training.
-- **Gradient spike at step 40k** — Reorganized the attention pattern in layer 9 that
-  previously captured long-range token correlations. A checkpoint from ~step 38k is expected
-  to have better aggregate benchmark scores.
-- **PolyNorm exclusivity** — The quadratic branch has become partially redundant with the
-  linear branch. Will be corrected in the next training run.
-- **Base model only** — Not instruction-tuned or aligned; purely a next-token-prediction
-  base model.
-
----
-
-## References
-
-All papers whose techniques are integrated into NeoLLM's architecture,
-training objective, or training stack:
-
-| Area | Technique | Paper title | Reference |
-|---|---|---|---|
-| Embeddings | Learnable Multipliers | Freeing the Scale of Language Model Matrix Layers | [arXiv:2601.04890](https://arxiv.org/abs/2601.04890) |
-| Embeddings | Leviathan | A Separable Architecture for Continuous Token Representation in Language Models | [arXiv:2601.22040](https://arxiv.org/abs/2601.22040) |
-| Embeddings | KHRONOS | KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation | [arXiv:2505.13315](https://arxiv.org/abs/2505.13315) |
-| Embeddings | JTok / JTok-M | JTok: On Token Embedding as Another Axis of Scaling Law via Joint Token Self-Modulation | [arXiv:2602.00800](https://arxiv.org/abs/2602.00800) |
-| Embeddings | Spelling Bee | Spelling Bee Embeddings for Language Modeling | [arXiv:2601.18030](https://arxiv.org/abs/2601.18030) |
-| Embeddings | Token embedding analysis | Token Embeddings Violate the Manifold Hypothesis | [arXiv:2504.01002](https://arxiv.org/abs/2504.01002) |
-| Attention / positions | FAN | Fourier Analysis Networks | [arXiv:2502.21309](https://arxiv.org/abs/2502.21309) |
-| Attention / positions | MEA | Explicit Multi-head Attention for Inter-head Interaction in Large Language Models | [arXiv:2601.19611](https://arxiv.org/abs/2601.19611) |
-| Attention / positions | LUCID | Attention with Preconditioned Representations | [arXiv:2602.10410](https://arxiv.org/abs/2602.10410) |
-| Attention / positions | Affine-Scaled Attention | Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention | [arXiv:2602.23057](https://arxiv.org/abs/2602.23057) |
-| Attention / positions | XSA | Exclusive Self Attention | [arXiv:2603.09078](https://arxiv.org/abs/2603.09078) |
-| Attention / positions | Directional Routing | Directional Routing in Transformers | [arXiv:2603.14923](https://arxiv.org/abs/2603.14923) |
-| Attention / positions | Gated Attention | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free | [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) |
-| Attention / positions | Momentum Attention | Momentum Attention | [arXiv:2411.03884](https://arxiv.org/abs/2411.03884) |
-| Attention / positions | IHA | Interleaved Head Attention | [arXiv:2602.21371](https://arxiv.org/abs/2602.21371) |
-| Attention / positions | REPO | Language Models with Context Re-Positioning | [arXiv:2512.14391](https://arxiv.org/abs/2512.14391) |
-| Attention / positions | GRAPE | Group Representational Position Encoding | [arXiv:2512.07805](https://arxiv.org/abs/2512.07805) |
-| Attention / positions | GOAT priors | You Need Better Attention Priors | [arXiv:2601.15380](https://arxiv.org/abs/2601.15380) |
-| Attention / positions | Hadamard o_proj | Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers | [arXiv:2603.08343](https://arxiv.org/abs/2603.08343) |
-| Residual / normalization | SeeDNorm | Self-Rescaled Dynamic Normalization | [arXiv:2510.22777](https://arxiv.org/abs/2510.22777) |
-| Residual / normalization | LNS | The Curse of Depth in LLMs | [arXiv:2502.05795](https://arxiv.org/abs/2502.05795) |
-| Residual / normalization | GPAS | Gradient-Preserving Activation Scaling | [arXiv:2506.22049](https://arxiv.org/abs/2506.22049) |
-| Residual / normalization | PolyNorm | PolyNorm / PolyCom | [arXiv:2602.04902](https://arxiv.org/abs/2602.04902) |
-| Residual / normalization | SimpleGPT | SimpleGPT | [arXiv:2602.01212](https://arxiv.org/abs/2602.01212) |
-| Residual / normalization | StackMemory / STACKTRANS | Recursive Transformer: Boosting Reasoning Ability with State Stack | [NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh) |
-| Residual / normalization | Attention Residuals | Attention Residuals | [arXiv:2603.15031](https://arxiv.org/abs/2603.15031) |
-| Residual / normalization | LAUREL | LAUREL: Learned Augmented Residual Layer | [arXiv:2411.07501](https://arxiv.org/abs/2411.07501) |
-| Objectives | TWEO | Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies | [arXiv:2511.23225](https://arxiv.org/abs/2511.23225) |
-| Objectives | NITP | Next Implicit Token Prediction for LLM Pre-training | [arXiv:2605.24956](https://arxiv.org/abs/2605.24956) |
-| Objectives | NextLat | Next-Latent Prediction Transformers Learn Compact World Models | [arXiv:2511.05963](https://arxiv.org/abs/2511.05963) |
-| Optimizer / training | Conda | Column-Normalized Adam for Training Large Language Models Faster | [arXiv:2509.24218](https://arxiv.org/abs/2509.24218) |
-| Optimizer / training | CWD | Cautious Weight Decay | [arXiv:2510.12402](https://arxiv.org/abs/2510.12402) |
-| Optimizer / training | WD correction | Correction of Decoupled Weight Decay | [arXiv:2512.08217](https://arxiv.org/abs/2512.08217) |
-| Optimizer / training | AdamHD | AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training | [arXiv:2511.14721](https://arxiv.org/abs/2511.14721) |
-| Optimizer / training | GradientStabilizer | GradientStabilizer | [arXiv:2502.17055](https://arxiv.org/abs/2502.17055) |
-| Optimizer / training | PACE | Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models | [arXiv:2606.25086](https://arxiv.org/abs/2606.25086) |
-
----
-
-## Citation
-
-```bibtex
-@misc{neollm2026,
-  title  = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
-  author = {KitsuVp},
-  year   = {2026},
-  url    = {https://huggingface.co/KitsuVp/NeoLLM}
-}
-```
-
----
-
-## Author
-
-[@Kyokopom](https://x.com/Kyokopom) on X
-
----
-
-## License
+# NeoLLM
 
-Apache 2.0
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.7278
+- Ntp Loss: 2.9840
+- Tweo Loss: 0.0158
+- Nitp Loss: 0.4387
+- Nitp Temporal Loss: 0.3160
+- Nitp Temporal State Loss: 0.1585
+- Nitp Temporal Identification Loss: 1.5747
+- Nitp Temporal Top1 Accuracy: 0.2544
+- Nitp Temporal Mean Absolute Offset: 1.2972
+- Nitp Temporal Diagonal Cosine: 0.6739
+- Nitp Temporal Off Diagonal Cosine: 0.6723
+- Nitp Temporal Identification Margin: -0.0514
+- Nitp Temporal Last Step State Loss: 0.2352
+- Nitp Temporal Rollout Nitp Cosine: 0.2379
+- Nitp Temporal Rollout Pair Cosine: 0.9479
+- Nitp Temporal Valid Window Fraction: 0.9440
+- Nitp Temporal To Nitp Ratio: 0.7202
+- Nitp Temporal Loss Applied: 0.0
+- Total Model Loss: 3.4229
+- Optimizer Step: 46875.0
+- Optimizer Metrics Due: 0.0
+- Pace Step: 0.0
+- Pace Update Due: 0.0
+- Pace Previous Iterate Active: 1.0
+- Conda Adaptive Scale Active: 1.0
+- Conda Scale Smoothed Metric Exact: 1.0
+- Conda Scale Beta: 0.0046
+- Conda Scale Energy Attenuation: 0.0021
+- Conda Scale Candidate Ratio Mean: 0.9985
+- Conda Scale Smoothed Ratio Mean: 0.9982
+- Conda Scale Applied Ratio Mean: 0.9982
+- Conda Scale Gain Mean: 0.0699
+- Conda Scale Gain Clip Fraction: 0.0
+- Conda Scale Ratio Min: 0.9714
+- Conda Scale Ratio Max: 1.0000
+- Conda Scale Low Limit Fraction: 0.0
+- Conda Scale High Limit Fraction: 0.0
+- Conda Scale Amplified Fraction: 0.0
+- Conda Scale Attenuated Fraction: 1.0
+- Conda Scale Energy Weighted Amplification: 0.0
+- Conda Scale Energy Weighted Attenuation Applied: 0.0018
+- Conda Scale Risk Ema Mean: 0.0026
+- Conda Scale Risk Drop Mean: 0.0006
+- Conda Scale Amplification Bonus Mean: 0.0006
+- Conda Scale Monitored Matrices: 146.0
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0006
+- train_batch_size: 64
+- eval_batch_size: 64
+- seed: 42
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 0.1
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss | Loss   | Temporal Loss | Temporal State Loss | Temporal Identification Loss | Temporal Top1 Accuracy | Temporal Mean Absolute Offset | Temporal Diagonal Cosine | Temporal Off Diagonal Cosine | Temporal Identification Margin | Temporal Last Step State Loss | Temporal Rollout Nitp Cosine | Temporal Rollout Pair Cosine | Temporal Valid Window Fraction | Temporal To Nitp Ratio | Temporal Loss Applied | Model Loss | Metrics Due | Update Due | Previous Iterate Active | Adaptive Scale Active | Scale Smoothed Metric Exact | Scale Beta | Scale Energy Attenuation | Scale Candidate Ratio Mean | Scale Smoothed Ratio Mean | Scale Applied Ratio Mean | Scale Gain Mean | Scale Gain Clip Fraction | Scale Ratio Min | Scale Ratio Max | Scale Low Limit Fraction | Scale High Limit Fraction | Scale Amplified Fraction | Scale Attenuated Fraction | Scale Energy Weighted Amplification | Scale Energy Weighted Attenuation Applied | Scale Risk Ema Mean | Scale Risk Drop Mean | Scale Amplification Bonus Mean | Scale Monitored Matrices |
+|:-------------:|:------:|:----:|:---------------:|:------:|:-------------:|:-------------------:|:----------------------------:|:----------------------:|:-----------------------------:|:------------------------:|:----------------------------:|:------------------------------:|:-----------------------------:|:----------------------------:|:----------------------------:|:------------------------------:|:----------------------:|:---------------------:|:----------:|:-----------:|:----------:|:-----------------------:|:---------------------:|:---------------------------:|:----------:|:------------------------:|:--------------------------:|:-------------------------:|:------------------------:|:---------------:|:------------------------:|:---------------:|:---------------:|:------------------------:|:-------------------------:|:------------------------:|:-------------------------:|:-----------------------------------:|:-----------------------------------------:|:-------------------:|:--------------------:|:------------------------------:|:------------------------:|
+| 4.4850        | 0.1067 | 0.0  | 4.3704          | 0.4853 | 0.6151        | 0.4492              | 1.6595                       | 0.2513                 | 1.3104                        | 0.4216                   | 0.4192                       | -0.1064                        | 0.5258                        | 0.2058                       | 0.9702                       | 0.9440                         | 1.2676                 | 0.0                   | 4.0625     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0141     | 0.0182                   | 0.9878                     | 0.9821                    | 0.9821                   | 0.1899          | 0.1642                   | 0.6819          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0179                                    | 0.0242              | 0.0060               | 0.0060                         | 146.0                    |
+| 4.2509        | 0.2133 | 0.0  | 4.1347          | 0.4695 | 0.4417        | 0.2782              | 1.6357                       | 0.2510                 | 1.3128                        | 0.4574                   | 0.4541                       | -0.0946                        | 0.3550                        | 0.2104                       | 0.9543                       | 0.9440                         | 0.9409                 | 0.0                   | 3.8230     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0100     | 0.0318                   | 0.9688                     | 0.9721                    | 0.9721                   | 0.2002          | 0.1688                   | 0.7255          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0279                                    | 0.0316              | 0.0006               | 0.0006                         | 146.0                    |
+| 4.1353        | 0.32   | 0.0  | 4.0248          | 0.4616 | 0.3790        | 0.2162              | 1.6281                       | 0.2512                 | 1.3107                        | 0.4898                   | 0.4864                       | -0.0841                        | 0.2921                        | 0.2158                       | 0.9447                       | 0.9440                         | 0.8211                 | 0.0                   | 3.7173     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0082     | 0.0207                   | 0.9805                     | 0.9792                    | 0.9792                   | 0.1992          | 0.1673                   | 0.7583          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0208                                    | 0.0219              | 0.0012               | 0.0012                         | 146.0                    |
+| 4.0732        | 0.4267 | 0.0  | 3.9661          | 0.4590 | 0.3523        | 0.1901              | 1.6225                       | 0.2516                 | 1.3108                        | 0.5218                   | 0.5186                       | -0.0751                        | 0.2668                        | 0.2179                       | 0.9397                       | 0.9440                         | 0.7677                 | 0.0                   | 3.6602     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0071     | 0.0238                   | 0.9763                     | 0.9785                    | 0.9785                   | 0.1953          | 0.1676                   | 0.7326          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0215                                    | 0.0234              | 0.0001               | 0.0001                         | 146.0                    |
+| 4.0285        | 0.5333 | 0.0  | 3.9243          | 0.4548 | 0.3392        | 0.1770              | 1.6221                       | 0.2531                 | 1.3061                        | 0.5518                   | 0.5489                       | -0.0689                        | 0.2537                        | 0.2207                       | 0.9381                       | 0.9440                         | 0.7458                 | 0.0                   | 3.6153     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0063     | 0.0148                   | 0.9903                     | 0.9829                    | 0.9829                   | 0.1907          | 0.1643                   | 0.7799          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0171                                    | 0.0199              | 0.0051               | 0.0051                         | 146.0                    |
+| 3.9952        | 0.64   | 0.0  | 3.8911          | 0.4544 | 0.3212        | 0.1601              | 1.6103                       | 0.2529                 | 1.3037                        | 0.5928                   | 0.5903                       | -0.0612                        | 0.2359                        | 0.2239                       | 0.9385                       | 0.9440                         | 0.7068                 | 0.0                   | 3.5908     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0058     | 0.0381                   | 0.9619                     | 0.9806                    | 0.9806                   | 0.1882          | 0.1634                   | 0.8281          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0194                                    | 0.0230              | 0.0000               | 0.0000                         | 146.0                    |
+| 3.9769        | 0.7467 | 0.0  | 3.8689          | 0.4523 | 0.3173        | 0.1564              | 1.6094                       | 0.2539                 | 1.3031                        | 0.6123                   | 0.6100                       | -0.0589                        | 0.2322                        | 0.2251                       | 0.9395                       | 0.9440                         | 0.7016                 | 0.0                   | 3.5642     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0053     | 0.0128                   | 0.9894                     | 0.9890                    | 0.9890                   | 0.1899          | 0.1659                   | 0.8047          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0110                                    | 0.0150              | 0.0022               | 0.0022                         | 146.0                    |
+| 3.8822        | 0.8533 | 0.0  | 3.7924          | 0.4437 | 0.3156        | 0.1569              | 1.5872                       | 0.2539                 | 1.2996                        | 0.6478                   | 0.6458                       | -0.0544                        | 0.2328                        | 0.2320                       | 0.9444                       | 0.9440                         | 0.7113                 | 0.0                   | 3.4815     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0050     | 0.0093                   | 0.9907                     | 0.9930                    | 0.9930                   | 0.1751          | 0.1541                   | 0.9111          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0070                                    | 0.0079              | 0.0000               | 0.0000                         | 146.0                    |
+| 3.8322        | 0.96   | 0.0  | 3.7350          | 0.4384 | 0.3158        | 0.1579              | 1.5788                       | 0.2542                 | 1.2969                        | 0.6692                   | 0.6676                       | -0.0522                        | 0.2345                        | 0.2377                       | 0.9471                       | 0.9440                         | 0.7204                 | 0.0                   | 3.4309     | 1.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0047     | 0.0024                   | 0.9984                     | 0.9978                    | 0.9978                   | 0.1206          | 0.0                      | 0.9626          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0022                                    | 0.0032              | 0.0008               | 0.0008                         | 146.0                    |
+| 3.8232        | 1.0    | 0.0  | 3.7278          | 0.4387 | 0.3160        | 0.1585              | 1.5747                       | 0.2544                 | 1.2972                        | 0.6739                   | 0.6723                       | -0.0514                        | 0.2352                        | 0.2379                       | 0.9479                       | 0.9440                         | 0.7202                 | 0.0                   | 3.4229     | 0.0         | 0.0        | 1.0                     | 1.0                   | 1.0                         | 0.0046     | 0.0021                   | 0.9985                     | 0.9982                    | 0.9982                   | 0.0699          | 0.0                      | 0.9714          | 1.0000          | 0.0                      | 0.0                       | 0.0                      | 1.0                       | 0.0                                 | 0.0018                                    | 0.0026              | 0.0006               | 0.0006                         | 146.0                    |
+
+
+### Framework versions
+
+- Transformers 5.13.1
+- Pytorch 2.13.0+cu132
+- Datasets 5.0.0
+- Tokenizers 0.22.2

config.json

@@ -60,6 +60,13 @@
   "nitp_loss_weight": 1.0,
   "nitp_projector_intermediate_size": 2048,
   "nitp_target_layer": 2,
+  "nitp_temporal_apply_loss": false,
+  "nitp_temporal_dynamics_weight": 1.0,
+  "nitp_temporal_horizon": 4,
+  "nitp_temporal_identification_weight": 0.1,
+  "nitp_temporal_intermediate_size": 1024,
+  "nitp_temporal_similarity_temperature": 0.1,
+  "nitp_temporal_target_temperature": 0.5,
   "ntp_loss_backend": "cce",
   "num_attention_heads": 8,
   "num_hidden_layers": 12,
@@ -86,7 +93,7 @@
   "stack_memory_cache_size": 2048,
   "stack_slots": 16,
   "tie_word_embeddings": false,
-  "transformers_version": "5.13.0",
+  "transformers_version": "5.13.1",
   "tweo_eps": 1e-06,
   "tweo_loss_weight": 0.01,
   "tweo_power": 4.0,
@@ -114,6 +121,7 @@
   "use_momentum_attention": true,
   "use_nextlat": false,
   "use_nitp": true,
+  "use_nitp_temporal": true,
   "use_repo": true,
   "use_repo_goat_prior": false,
   "use_repo_grape": true,

generation_config.json

@@ -7,5 +7,5 @@
   "output_attentions": false,
   "output_hidden_states": false,
   "pad_token_id": 0,
-  "transformers_version": "5.13.0"
+  "transformers_version": "5.13.1"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0ba8908970e4f45254f6a9810f92ffb23f1dd2103f82d24e384907002895bafa
-size 240855132
+oid sha256:d36dc7b27ba5811b691ab7b110b2fe30e1bf5ce5445fa85d756d8a784957cbab
+size 245969360

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4d449fb352570446b7cc3ff35d1e2464749cf4a5f3f64c052689f59b3792b985
+oid sha256:1ff7c3d201b035f6c6d0af4363b06d409b623147021216ed251969823f72239c
 size 5329