---
language:
- en
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- tiny-models
- custom-architecture
- story-generation
- experimental
---

# Spin-80k

**Spin-80k** is a lightweight, 80k-parameter decoder-only language model built from scratch by **Quantech** to demonstrate custom Transformer architecture 

---

## Model Specifications

* **Organization:** Quantech
* **Architecture:** Custom Decoder-only Transformer
* **Total Parameters:** ~80,112
* **Layers:** 2
* **Hidden Dimension ($d_{\text{model}}$):** 48
* **Attention Heads:** 4
* **Feed-Forward Dimension ($d_{\text{ff}}$):** 128
* **Positional Encoding:** Rotary Position Embeddings (RoPE)
* **Normalization:** RMSNorm ($\epsilon = 10^{-5}$)
* **Activation:** SwiGLU
* **Vocabulary:** 512 Byte-Pair Encoding (BPE) tokens
* **Context Length:** 256 tokens

---

## Quickstart

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Quantech/spin-80k"

# Load Tokenizer & Model
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

# ChatML Format
prompt = "<|im_start|>user\nWrite a short story about a dog.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0]))