Fix config hyperparameters for 80k architecture and tie embeddings

README.md

@@ -23,11 +23,11 @@ tags:
 * **Architecture:** Custom Decoder-only Transformer
 * **Total Parameters:** ~80,112
 * **Layers:** 2
-* **Hidden Dimension ($d_{\text{model}}$):** 48
+* **Hidden Dimension:** 48
 * **Attention Heads:** 4
-* **Feed-Forward Dimension ($d_{\text{ff}}$):** 128
+* **Feed-Forward Dimension:** 128
 * **Positional Encoding:** Rotary Position Embeddings (RoPE)
-* **Normalization:** RMSNorm ($\epsilon = 10^{-5}$)
+* **Normalization:** RMSNorm 
 * **Activation:** SwiGLU
 * **Vocabulary:** 512 Byte-Pair Encoding (BPE) tokens
 * **Context Length:** 256 tokens
@@ -42,12 +42,11 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
 
 repo_id = "Quantech/spin-80k"
 
-# Load Tokenizer & Model
+
 tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
 model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
 model.eval()
 
-# ChatML Format
 prompt = "<|im_start|>user\nWrite a short story about a dog.<|im_end|>\n<|im_start|>assistant\n"
 inputs = tokenizer(prompt, return_tensors="pt")