Instructions to use MetaCore-LLM/MetaCore-1-Test-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MetaCore-LLM/MetaCore-1-Test-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MetaCore-LLM/MetaCore-1-Test-Base")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MetaCore-LLM/MetaCore-1-Test-Base")
model = AutoModelForCausalLM.from_pretrained("MetaCore-LLM/MetaCore-1-Test-Base", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MetaCore-LLM/MetaCore-1-Test-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MetaCore-LLM/MetaCore-1-Test-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaCore-LLM/MetaCore-1-Test-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/MetaCore-LLM/MetaCore-1-Test-Base

SGLang

How to use MetaCore-LLM/MetaCore-1-Test-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MetaCore-LLM/MetaCore-1-Test-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaCore-LLM/MetaCore-1-Test-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MetaCore-LLM/MetaCore-1-Test-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaCore-LLM/MetaCore-1-Test-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use MetaCore-LLM/MetaCore-1-Test-Base with Docker Model Runner:
```
docker model run hf.co/MetaCore-LLM/MetaCore-1-Test-Base
```

MetaCore-LLM commited on 6 days ago

Commit

17df6bf

verified ·

1 Parent(s): 4bf56db

Upload folder using huggingface_hub

Browse files

Files changed (30) hide show

checkpoint-500/config.json +32 -0
checkpoint-500/generation_config.json +7 -0
checkpoint-500/model.safetensors +3 -0
checkpoint-500/optimizer.pt +3 -0
checkpoint-500/rng_state_0.pth +3 -0
checkpoint-500/rng_state_1.pth +3 -0
checkpoint-500/scaler.pt +3 -0
checkpoint-500/scheduler.pt +3 -0
checkpoint-500/tokenizer.json +0 -0
checkpoint-500/tokenizer_config.json +14 -0
checkpoint-500/trainer_state.json +209 -0
checkpoint-500/training_args.bin +3 -0
checkpoint-822/config.json +32 -0
checkpoint-822/generation_config.json +7 -0
checkpoint-822/model.safetensors +3 -0
checkpoint-822/optimizer.pt +3 -0
checkpoint-822/rng_state_0.pth +3 -0
checkpoint-822/rng_state_1.pth +3 -0
checkpoint-822/scaler.pt +3 -0
checkpoint-822/scheduler.pt +3 -0
checkpoint-822/tokenizer.json +0 -0
checkpoint-822/tokenizer_config.json +14 -0
checkpoint-822/trainer_state.json +321 -0
checkpoint-822/training_args.bin +3 -0
config.json +32 -0
generation_config.json +7 -0
model.safetensors +3 -0
tokenizer.json +0 -0
tokenizer_config.json +14 -0
training_args.bin +3 -0

checkpoint-500/config.json ADDED Viewed

	@@ -0,0 +1,32 @@

+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "head_dim": 32,
+  "hidden_act": "silu",
+  "hidden_size": 128,
+  "initializer_range": 0.02,
+  "intermediate_size": 384,
+  "max_position_embeddings": 256,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 4,
+  "num_key_value_heads": 4,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "vocab_size": 119547
+}

checkpoint-500/generation_config.json ADDED Viewed

	@@ -0,0 +1,7 @@

+{
+  "_from_model_config": true,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.0.0",
+  "use_cache": true
+}

checkpoint-500/model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5c6faa58f5e91e30fec89e8c983207cb3e1534263a73c3fa378885a8c3ee1379
+size 64624680

checkpoint-500/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:aabdb2e40ee8872c7876f043ecd44cc486261a9035d47abf01c0602b176c7851
+size 129274187

checkpoint-500/rng_state_0.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:95e5fc2074c0df31522a514f862c86cb00d71c946a7f15cc9ec0e53a69fb28a7
+size 14917

checkpoint-500/rng_state_1.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0e7153eae67b6c9232a41bc996a2bf5b83229b8c7230d61911ac0fd40e64154e
+size 14917

checkpoint-500/scaler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f77569c2e850b04af982cc8c1389f1430851448915c593b69e5da36ce05b71d7
+size 1383

checkpoint-500/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:23c5de61153693dd94c85e5b79d860b342aedccdcebe359757c59934cc32f12f
+size 1465

checkpoint-500/tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-500/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,14 @@

+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "is_local": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

checkpoint-500/trainer_state.json ADDED Viewed

	@@ -0,0 +1,209 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.8265142857142858,
+  "eval_steps": 500,
+  "global_step": 500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.07314285714285715,
+      "grad_norm": 1.250452995300293,
+      "learning_rate": 2.8499999999999998e-05,
+      "loss": 11.532991027832031,
+      "step": 20
+    },
+    {
+      "epoch": 0.1462857142857143,
+      "grad_norm": 0.9911406636238098,
+      "learning_rate": 5.85e-05,
+      "loss": 11.35167236328125,
+      "step": 40
+    },
+    {
+      "epoch": 0.21942857142857142,
+      "grad_norm": 0.9821921586990356,
+      "learning_rate": 8.849999999999998e-05,
+      "loss": 11.14882583618164,
+      "step": 60
+    },
+    {
+      "epoch": 0.2925714285714286,
+      "grad_norm": 0.9792555570602417,
+      "learning_rate": 0.0001185,
+      "loss": 10.891434478759766,
+      "step": 80
+    },
+    {
+      "epoch": 0.3657142857142857,
+      "grad_norm": 0.9883681535720825,
+      "learning_rate": 0.00014849999999999998,
+      "loss": 10.563937377929687,
+      "step": 100
+    },
+    {
+      "epoch": 0.43885714285714283,
+      "grad_norm": 0.9853702783584595,
+      "learning_rate": 0.00017849999999999997,
+      "loss": 10.18422622680664,
+      "step": 120
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 1.0101815462112427,
+      "learning_rate": 0.00020849999999999997,
+      "loss": 9.786434936523438,
+      "step": 140
+    },
+    {
+      "epoch": 0.5851428571428572,
+      "grad_norm": 0.9789732098579407,
+      "learning_rate": 0.0002385,
+      "loss": 9.40113525390625,
+      "step": 160
+    },
+    {
+      "epoch": 0.6582857142857143,
+      "grad_norm": 0.9101362824440002,
+      "learning_rate": 0.00026849999999999997,
+      "loss": 9.05419464111328,
+      "step": 180
+    },
+    {
+      "epoch": 0.7314285714285714,
+      "grad_norm": 0.8724201917648315,
+      "learning_rate": 0.0002985,
+      "loss": 8.751616668701171,
+      "step": 200
+    },
+    {
+      "epoch": 0.8045714285714286,
+      "grad_norm": 0.7059838175773621,
+      "learning_rate": 0.0002908360128617363,
+      "loss": 8.522535705566407,
+      "step": 220
+    },
+    {
+      "epoch": 0.8777142857142857,
+      "grad_norm": 0.5502440929412842,
+      "learning_rate": 0.00028118971061093245,
+      "loss": 8.364321899414062,
+      "step": 240
+    },
+    {
+      "epoch": 0.9508571428571428,
+      "grad_norm": 0.46737125515937805,
+      "learning_rate": 0.0002715434083601286,
+      "loss": 8.275575256347656,
+      "step": 260
+    },
+    {
+      "epoch": 1.021942857142857,
+      "grad_norm": 0.387386679649353,
+      "learning_rate": 0.00026189710610932475,
+      "loss": 8.194184875488281,
+      "step": 280
+    },
+    {
+      "epoch": 1.0950857142857142,
+      "grad_norm": 0.5874072909355164,
+      "learning_rate": 0.0002522508038585209,
+      "loss": 8.096188354492188,
+      "step": 300
+    },
+    {
+      "epoch": 1.1682285714285714,
+      "grad_norm": 0.6955321431159973,
+      "learning_rate": 0.00024260450160771703,
+      "loss": 8.026647186279297,
+      "step": 320
+    },
+    {
+      "epoch": 1.2413714285714286,
+      "grad_norm": 0.49530029296875,
+      "learning_rate": 0.00023295819935691315,
+      "loss": 7.971723937988282,
+      "step": 340
+    },
+    {
+      "epoch": 1.3145142857142857,
+      "grad_norm": 0.9392899870872498,
+      "learning_rate": 0.0002233118971061093,
+      "loss": 7.904925537109375,
+      "step": 360
+    },
+    {
+      "epoch": 1.387657142857143,
+      "grad_norm": 0.6055660843849182,
+      "learning_rate": 0.00021366559485530546,
+      "loss": 7.842314147949219,
+      "step": 380
+    },
+    {
+      "epoch": 1.4607999999999999,
+      "grad_norm": 0.4336493909358978,
+      "learning_rate": 0.00020401929260450158,
+      "loss": 7.779663848876953,
+      "step": 400
+    },
+    {
+      "epoch": 1.5339428571428573,
+      "grad_norm": 0.532171368598938,
+      "learning_rate": 0.00019437299035369773,
+      "loss": 7.732464599609375,
+      "step": 420
+    },
+    {
+      "epoch": 1.6070857142857142,
+      "grad_norm": 0.5588330626487732,
+      "learning_rate": 0.00018472668810289386,
+      "loss": 7.665067291259765,
+      "step": 440
+    },
+    {
+      "epoch": 1.6802285714285714,
+      "grad_norm": 0.5427436828613281,
+      "learning_rate": 0.00017508038585209004,
+      "loss": 7.601129150390625,
+      "step": 460
+    },
+    {
+      "epoch": 1.7533714285714286,
+      "grad_norm": 0.8764950633049011,
+      "learning_rate": 0.00016543408360128616,
+      "loss": 7.574497985839844,
+      "step": 480
+    },
+    {
+      "epoch": 1.8265142857142858,
+      "grad_norm": 0.48415958881378174,
+      "learning_rate": 0.0001557877813504823,
+      "loss": 7.537950134277343,
+      "step": 500
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 822,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 118333073522688.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-500/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dd9494893863dcfcde3e9f41f481df996dad8a854ef54e48e82f01bb1d81d2a9
+size 5137

checkpoint-822/config.json ADDED Viewed

	@@ -0,0 +1,32 @@

+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "head_dim": 32,
+  "hidden_act": "silu",
+  "hidden_size": 128,
+  "initializer_range": 0.02,
+  "intermediate_size": 384,
+  "max_position_embeddings": 256,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 4,
+  "num_key_value_heads": 4,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "vocab_size": 119547
+}

checkpoint-822/generation_config.json ADDED Viewed

	@@ -0,0 +1,7 @@

+{
+  "_from_model_config": true,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.0.0",
+  "use_cache": true
+}

checkpoint-822/model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1242ec5613e5bd72d81f17c0c884f362678882f23d0a185eb83957c6e44d7766
+size 64624680

checkpoint-822/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a2e8db8e0bc3a7fb1fa1b161a742bd3e6d3b89c8a36160e005b9842e15e18274
+size 129274187

checkpoint-822/rng_state_0.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:61e957b4cd785256be4cb26eb03060ef689e1d58f1766d7f26ca36a62bec4994
+size 14917

checkpoint-822/rng_state_1.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:550c54d430b44b77b0abe44c6e3ceba90a155305315c081b7616b35e2c18d1ce
+size 14917

checkpoint-822/scaler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2ba36415f41e17492ea44f104172147b52f3f43aaa4ece6c4c1b0ab72e7a684f
+size 1383

checkpoint-822/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5ff73d915a1966dc849ecea8a71990b9f6619e9d3135e1ad2aab9d46c778b0de
+size 1465

checkpoint-822/tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-822/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,14 @@

+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "is_local": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

checkpoint-822/trainer_state.json ADDED Viewed

	@@ -0,0 +1,321 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.0,
+  "eval_steps": 500,
+  "global_step": 822,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.07314285714285715,
+      "grad_norm": 1.250452995300293,
+      "learning_rate": 2.8499999999999998e-05,
+      "loss": 11.532991027832031,
+      "step": 20
+    },
+    {
+      "epoch": 0.1462857142857143,
+      "grad_norm": 0.9911406636238098,
+      "learning_rate": 5.85e-05,
+      "loss": 11.35167236328125,
+      "step": 40
+    },
+    {
+      "epoch": 0.21942857142857142,
+      "grad_norm": 0.9821921586990356,
+      "learning_rate": 8.849999999999998e-05,
+      "loss": 11.14882583618164,
+      "step": 60
+    },
+    {
+      "epoch": 0.2925714285714286,
+      "grad_norm": 0.9792555570602417,
+      "learning_rate": 0.0001185,
+      "loss": 10.891434478759766,
+      "step": 80
+    },
+    {
+      "epoch": 0.3657142857142857,
+      "grad_norm": 0.9883681535720825,
+      "learning_rate": 0.00014849999999999998,
+      "loss": 10.563937377929687,
+      "step": 100
+    },
+    {
+      "epoch": 0.43885714285714283,
+      "grad_norm": 0.9853702783584595,
+      "learning_rate": 0.00017849999999999997,
+      "loss": 10.18422622680664,
+      "step": 120
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 1.0101815462112427,
+      "learning_rate": 0.00020849999999999997,
+      "loss": 9.786434936523438,
+      "step": 140
+    },
+    {
+      "epoch": 0.5851428571428572,
+      "grad_norm": 0.9789732098579407,
+      "learning_rate": 0.0002385,
+      "loss": 9.40113525390625,
+      "step": 160
+    },
+    {
+      "epoch": 0.6582857142857143,
+      "grad_norm": 0.9101362824440002,
+      "learning_rate": 0.00026849999999999997,
+      "loss": 9.05419464111328,
+      "step": 180
+    },
+    {
+      "epoch": 0.7314285714285714,
+      "grad_norm": 0.8724201917648315,
+      "learning_rate": 0.0002985,
+      "loss": 8.751616668701171,
+      "step": 200
+    },
+    {
+      "epoch": 0.8045714285714286,
+      "grad_norm": 0.7059838175773621,
+      "learning_rate": 0.0002908360128617363,
+      "loss": 8.522535705566407,
+      "step": 220
+    },
+    {
+      "epoch": 0.8777142857142857,
+      "grad_norm": 0.5502440929412842,
+      "learning_rate": 0.00028118971061093245,
+      "loss": 8.364321899414062,
+      "step": 240
+    },
+    {
+      "epoch": 0.9508571428571428,
+      "grad_norm": 0.46737125515937805,
+      "learning_rate": 0.0002715434083601286,
+      "loss": 8.275575256347656,
+      "step": 260
+    },
+    {
+      "epoch": 1.021942857142857,
+      "grad_norm": 0.387386679649353,
+      "learning_rate": 0.00026189710610932475,
+      "loss": 8.194184875488281,
+      "step": 280
+    },
+    {
+      "epoch": 1.0950857142857142,
+      "grad_norm": 0.5874072909355164,
+      "learning_rate": 0.0002522508038585209,
+      "loss": 8.096188354492188,
+      "step": 300
+    },
+    {
+      "epoch": 1.1682285714285714,
+      "grad_norm": 0.6955321431159973,
+      "learning_rate": 0.00024260450160771703,
+      "loss": 8.026647186279297,
+      "step": 320
+    },
+    {
+      "epoch": 1.2413714285714286,
+      "grad_norm": 0.49530029296875,
+      "learning_rate": 0.00023295819935691315,
+      "loss": 7.971723937988282,
+      "step": 340
+    },
+    {
+      "epoch": 1.3145142857142857,
+      "grad_norm": 0.9392899870872498,
+      "learning_rate": 0.0002233118971061093,
+      "loss": 7.904925537109375,
+      "step": 360
+    },
+    {
+      "epoch": 1.387657142857143,
+      "grad_norm": 0.6055660843849182,
+      "learning_rate": 0.00021366559485530546,
+      "loss": 7.842314147949219,
+      "step": 380
+    },
+    {
+      "epoch": 1.4607999999999999,
+      "grad_norm": 0.4336493909358978,
+      "learning_rate": 0.00020401929260450158,
+      "loss": 7.779663848876953,
+      "step": 400
+    },
+    {
+      "epoch": 1.5339428571428573,
+      "grad_norm": 0.532171368598938,
+      "learning_rate": 0.00019437299035369773,
+      "loss": 7.732464599609375,
+      "step": 420
+    },
+    {
+      "epoch": 1.6070857142857142,
+      "grad_norm": 0.5588330626487732,
+      "learning_rate": 0.00018472668810289386,
+      "loss": 7.665067291259765,
+      "step": 440
+    },
+    {
+      "epoch": 1.6802285714285714,
+      "grad_norm": 0.5427436828613281,
+      "learning_rate": 0.00017508038585209004,
+      "loss": 7.601129150390625,
+      "step": 460
+    },
+    {
+      "epoch": 1.7533714285714286,
+      "grad_norm": 0.8764950633049011,
+      "learning_rate": 0.00016543408360128616,
+      "loss": 7.574497985839844,
+      "step": 480
+    },
+    {
+      "epoch": 1.8265142857142858,
+      "grad_norm": 0.48415958881378174,
+      "learning_rate": 0.0001557877813504823,
+      "loss": 7.537950134277343,
+      "step": 500
+    },
+    {
+      "epoch": 1.899657142857143,
+      "grad_norm": 0.5703415274620056,
+      "learning_rate": 0.00014614147909967844,
+      "loss": 7.497711181640625,
+      "step": 520
+    },
+    {
+      "epoch": 1.9727999999999999,
+      "grad_norm": 0.692701518535614,
+      "learning_rate": 0.0001364951768488746,
+      "loss": 7.422080993652344,
+      "step": 540
+    },
+    {
+      "epoch": 2.043885714285714,
+      "grad_norm": 0.6631471514701843,
+      "learning_rate": 0.00012684887459807074,
+      "loss": 7.393524169921875,
+      "step": 560
+    },
+    {
+      "epoch": 2.1170285714285715,
+      "grad_norm": 0.5688566565513611,
+      "learning_rate": 0.00011720257234726686,
+      "loss": 7.329080200195312,
+      "step": 580
+    },
+    {
+      "epoch": 2.1901714285714284,
+      "grad_norm": 0.6037266850471497,
+      "learning_rate": 0.00010755627009646302,
+      "loss": 7.306385040283203,
+      "step": 600
+    },
+    {
+      "epoch": 2.263314285714286,
+      "grad_norm": 0.5700935125350952,
+      "learning_rate": 9.790996784565915e-05,
+      "loss": 7.3006591796875,
+      "step": 620
+    },
+    {
+      "epoch": 2.336457142857143,
+      "grad_norm": 0.5397514700889587,
+      "learning_rate": 8.826366559485529e-05,
+      "loss": 7.2862548828125,
+      "step": 640
+    },
+    {
+      "epoch": 2.4096,
+      "grad_norm": 0.5593230724334717,
+      "learning_rate": 7.861736334405144e-05,
+      "loss": 7.267812347412109,
+      "step": 660
+    },
+    {
+      "epoch": 2.482742857142857,
+      "grad_norm": 0.720418393611908,
+      "learning_rate": 6.897106109324758e-05,
+      "loss": 7.238247680664062,
+      "step": 680
+    },
+    {
+      "epoch": 2.555885714285714,
+      "grad_norm": 0.6487265229225159,
+      "learning_rate": 5.9324758842443726e-05,
+      "loss": 7.224815368652344,
+      "step": 700
+    },
+    {
+      "epoch": 2.6290285714285715,
+      "grad_norm": 0.7691522836685181,
+      "learning_rate": 4.967845659163987e-05,
+      "loss": 7.226226043701172,
+      "step": 720
+    },
+    {
+      "epoch": 2.7021714285714284,
+      "grad_norm": 0.5983503460884094,
+      "learning_rate": 4.003215434083601e-05,
+      "loss": 7.176812744140625,
+      "step": 740
+    },
+    {
+      "epoch": 2.775314285714286,
+      "grad_norm": 0.5417843461036682,
+      "learning_rate": 3.038585209003215e-05,
+      "loss": 7.211582946777344,
+      "step": 760
+    },
+    {
+      "epoch": 2.848457142857143,
+      "grad_norm": 0.5134989023208618,
+      "learning_rate": 2.0739549839228295e-05,
+      "loss": 7.171044921875,
+      "step": 780
+    },
+    {
+      "epoch": 2.9215999999999998,
+      "grad_norm": 0.4077773988246918,
+      "learning_rate": 1.1093247588424437e-05,
+      "loss": 7.167872619628906,
+      "step": 800
+    },
+    {
+      "epoch": 2.994742857142857,
+      "grad_norm": 0.49174758791923523,
+      "learning_rate": 1.4469453376205788e-06,
+      "loss": 7.183602905273437,
+      "step": 820
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 822,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 194099916963840.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-822/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dd9494893863dcfcde3e9f41f481df996dad8a854ef54e48e82f01bb1d81d2a9
+size 5137

config.json ADDED Viewed

	@@ -0,0 +1,32 @@

+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "head_dim": 32,
+  "hidden_act": "silu",
+  "hidden_size": 128,
+  "initializer_range": 0.02,
+  "intermediate_size": 384,
+  "max_position_embeddings": 256,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 4,
+  "num_key_value_heads": 4,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "vocab_size": 119547
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,7 @@

+{
+  "_from_model_config": true,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.0.0",
+  "use_cache": true
+}

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1242ec5613e5bd72d81f17c0c884f362678882f23d0a185eb83957c6e44d7766
+size 64624680

tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,14 @@

+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "is_local": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dd9494893863dcfcde3e9f41f481df996dad8a854ef54e48e82f01bb1d81d2a9
+size 5137