| ---
|
| language:
|
| - en
|
| license: mit
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| tags:
|
| - tiny-models
|
| - custom-architecture
|
| - story-generation
|
| - experimental
|
| ---
|
|
|
| # Spin-80k
|
|
|
| **Spin-80k** is a lightweight, 80k-parameter decoder-only language model built from scratch by **Quantech** to demonstrate custom Transformer architecture
|
|
|
| ---
|
|
|
| ## Model Specifications
|
|
|
| * **Organization:** Quantech
|
| * **Architecture:** Custom Decoder-only Transformer
|
| * **Total Parameters:** ~80,112
|
| * **Layers:** 2
|
| * **Hidden Dimension ($d_{\text{model}}$):** 48
|
| * **Attention Heads:** 4
|
| * **Feed-Forward Dimension ($d_{\text{ff}}$):** 128
|
| * **Positional Encoding:** Rotary Position Embeddings (RoPE)
|
| * **Normalization:** RMSNorm ($\epsilon = 10^{-5}$)
|
| * **Activation:** SwiGLU
|
| * **Vocabulary:** 512 Byte-Pair Encoding (BPE) tokens
|
| * **Context Length:** 256 tokens
|
|
|
| ---
|
|
|
| ## Quickstart
|
|
|
| ```python
|
| import torch
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
| repo_id = "Quantech/spin-80k"
|
|
|
| # Load Tokenizer & Model
|
| tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
|
| model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
|
| model.eval()
|
|
|
| # ChatML Format
|
| prompt = "<|im_start|>user\nWrite a short story about a dog.<|im_end|>\n<|im_start|>assistant\n"
|
| inputs = tokenizer(prompt, return_tensors="pt")
|
|
|
| with torch.no_grad():
|
| outputs = model.generate(
|
| **inputs,
|
| max_new_tokens=50,
|
| temperature=0.7,
|
| do_sample=True,
|
| pad_token_id=tokenizer.eos_token_id
|
| )
|
|
|
| print(tokenizer.decode(outputs[0])) |
