Model save

README.md

@@ -14,7 +14,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 3.1389
+- Loss: 3.3061
 
 ## Model description
 
@@ -44,16 +44,22 @@ The following hyperparameters were used during training:
 
 ### Training results
 
-| Training Loss | Epoch | Step  | Validation Loss |
-|:-------------:|:-----:|:-----:|:---------------:|
-| 3.8136        | 0.32  | 5000  | 3.7511          |
-| 3.4787        | 0.64  | 10000 | 3.3824          |
-| 3.2781        | 0.96  | 15000 | 3.1389          |
+| Training Loss | Epoch  | Step  | Validation Loss |
+|:-------------:|:------:|:-----:|:---------------:|
+| 3.8671        | 0.1067 | 5000  | 3.7862          |
+| 3.6710        | 0.2133 | 10000 | 3.5860          |
+| 3.5866        | 0.32   | 15000 | 3.5085          |
+| 3.5398        | 0.4267 | 20000 | 3.4550          |
+| 3.5042        | 0.5333 | 25000 | 3.4161          |
+| 3.4765        | 0.64   | 30000 | 3.3908          |
+| 3.4618        | 0.7467 | 35000 | 3.3693          |
+| 3.4085        | 0.8533 | 40000 | 3.3357          |
+| 3.3873        | 0.96   | 45000 | 3.3061          |
 
 
 ### Framework versions
 
-- Transformers 5.0.0
+- Transformers 5.3.0
 - Pytorch 2.10.0+cu130
-- Datasets 4.5.0
+- Datasets 4.7.0
 - Tokenizers 0.22.2

config.json

@@ -19,12 +19,15 @@
   "hidden_size": 512,
   "initializer_range": 0.02,
   "intermediate_size": 1536,
+  "lucid_attention_eps": 1e-06,
   "max_position_embeddings": 512,
+  "mea_component_key_value_heads": 4,
+  "mea_groupnorm_eps": 1e-06,
   "model_type": "neollm",
+  "momentum_gamma": 0.1,
   "num_attention_heads": 8,
   "num_hidden_layers": 12,
-  "num_key_value_heads": 2,
-  "num_stack_heads": 4,
+  "num_key_value_heads": 4,
   "pad_token_id": 151643,
   "partial_rotary_factor": 0.25,
   "rms_norm_eps": 1e-06,
@@ -34,11 +37,11 @@
     "rope_type": "default"
   },
   "rope_theta": 10000.0,
-  "stack_d_model": 16,
-  "stack_slots": 24,
   "tie_word_embeddings": true,
-  "transformers_version": "5.0.0",
+  "transformers_version": "5.3.0",
   "use_cache": false,
-  "use_stack": true,
+  "use_lucid_attention": true,
+  "use_mea_attention": true,
+  "use_momentum_attention": true,
   "vocab_size": 151665
 }

generation_config.json

@@ -6,5 +6,5 @@
   "output_attentions": false,
   "output_hidden_states": false,
   "pad_token_id": 151643,
-  "transformers_version": "5.0.0"
+  "transformers_version": "5.3.0"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:025217f145a5f79a84857573d027db999d04cae8d6b671a3ef43dfe535c156c9
-size 252829736
+oid sha256:b86e70464eac60bf2e31c9bb80ecf49143001756ffde78208af382ee926c3960
+size 254583392

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e2ada6d4b4e8969f240c8d8be2b4784e3b53bf2cf9829ec94786bd9ae5595b09
+oid sha256:e21b9bd176ce940ed56dc325927aefec7471cb0899dffccbbcfcc9774a5ea9a2
 size 5329