Create README.md

README.md

@@ -0,0 +1,188 @@
+---
+license: apache-2.0
+language:
+- en
+tags:
+- prompt-routing
+- complexity-classifier
+- deberta-v3
+- llm-router
+- cost-optimization
+datasets:
+- RowRed/ComplexityRouter
+- OpenAssistant/oasst2
+base_model:
+- microsoft/deberta-v3-base
+---
+
+# ComplexityRouter: A Complexity based LLM Router
+
+Introducing ComplexityRouter, A lightweight prompt complexity classifier finetuned from **microsoft/deberta-v3-base**. Using prompts from [Open Assistant Conversations Dataset Release 2 (OASST2)](https://huggingface.co/datasets/OpenAssistant/oasst2), some from myself, and some made more complex by Qwen3.5-4B (non-thinking mode), I created a synthetic dataset classifying 4,400 of the prompts using [Qwen3.5-4B Non Thinking Mode](https://huggingface.co/Qwen/Qwen3.5-4B). 
+
+It assigns prompts to one of 4 complexity levels, making it useful for routing queries to the appropriate LLM tier.
+
+## Model Details
+
+### Model Description
+
+- **Model type:** Text Classification (multi‑class)
+- **Language:** English
+- **Backbone:** microsoft/deberta-v3-base
+- **License:** Apache‑2.0
+- **Finetuned from model:** microsoft/deberta-v3-base
+- **Training data:** OASST2 + synthetic augmentations + manually created prompts  
+  Labels generated by **Qwen3.5‑4B** (non‑thinking mode).
+
+### Model Sources
+
+- **Dataset repository:** https://huggingface.co/datasets/RowRed/ComplexityRouter
+
+## Uses
+
+### Direct Use
+
+Route prompts to appropriate LLM tiers based on predicted complexity:
+
+| Level | Meaning | Suggested LLM Tier |
+|-------|---------|-------------------|
+| 0 (Trivial) | Simple lookups, basic Q&A (e.g., “What is 2+2?”) | Fast/cheap local model |
+| 1 (Simple) | Moderate reasoning, basic domain knowledge | Mid‑tier model |
+| 2 (Moderate) | Complex reasoning, deep knowledge required | Strong model |
+| 3 (Complex) | Very complex reasoning, niche expertise | Frontier API model |
+
+**Recommended routing strategy:** Group levels **0 and 1** together (fast/cheap tier), level 2 as standard, level 3 as premium. The model achieves **92.5% adjacent accuracy** on my test, meaning it rarely misroutes by more than one tier.
+
+### Out‑of‑Scope Use
+
+- Multi‑turn conversation routing (single prompts only).
+- Non‑English prompts (training data was English‑only).
+- Prompts requiring image or multimodal understanding.
+
+## Bias, Risks, and Limitations
+
+- Training data is synthetic and may not represent all real‑world prompt distributions.
+- Level 1 (Simple) and Level 2 (Moderate) have lower per‑class F1 scores – boundary cases are inherently ambiguous.
+- The model may struggle with very domain‑specific technical jargon.
+- Performance may degrade on prompts that are very different from the training distribution.
+
+## Notice
+This is my first attempt making a widespread finetune. There are probably lots of issues, but thought the idea was sound. I might make a second (hopefully better) version eventually, but am not sure where to get lots of high-quality open source data.
+
+## Training Details
+### Training Data
+
+| Split       | Samples | Source File          | Notes |
+|-------------|---------|----------------------|-------|
+| Training    | 2,800   | TRAINING.jsonl       | Used for model training |
+| Validation  | 600     | TRAINING.jsonl       | Used for early stopping / hyperparameter tuning |
+| Test (internal) | 600 | TRAINING.jsonl      | Used for in‑distribution evaluation |
+| Test (held‑out) | 400 | TEST.jsonl   | Fully independent test set (reported results) |
+
+**Total unique prompts:** 4,400
+
+Class distribution (training):
+Level 0: 762 (27.2%) • Level 1: 674 (24.1%) • Level 2: 795 (28.4%) • Level 3: 569 (20.3%)
+
+## Training Procedure
+- Hardware: NVIDIA T4 (16 GB VRAM, Google Colab)
+- Framework: PyTorch 2.11 + Hugging Face Transformers
+- Optimizer: AdamW (lr=2e-5, weight_decay=0.01)
+- Scheduler: Linear warmup (10% of steps) → linear decay
+- Loss: Weighted Cross‑Entropy (classification) + MSE (regression)
+- Batch size: 16 (effective 32 with gradient accumulation)
+- Epochs: 7 (early stopping patience = 3)
+- Training time: ~18 minutes
+- Class balancing: sqrt‑scaled class weights + weighted random sampler
+
+## Evaluation Results
+Test Set (400 held‑out samples)
+
+| Metric                   | Value |
+|--------------------------|-------|
+| Exact Match Accuracy     | 65.0% |
+| Adjacent (±1) Accuracy   | 92.5% |
+| Macro F1                 | 0.663 |
+| Weighted F1              | 0.653 |
+
+## How to Get Started with the Model
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+import torch.nn as nn
+
+class PromptComplexityRouter(nn.Module):
+    def __init__(self, backbone="microsoft/deberta-v3-base", num_labels=4):
+        super().__init__()
+        self.backbone = AutoModel.from_pretrained(backbone)
+        hidden_size = self.backbone.config.hidden_size
+        self.classifier = nn.Sequential(
+            nn.Dropout(0.1),
+            nn.Linear(hidden_size, 256),
+            nn.GELU(),
+            nn.Dropout(0.1),
+            nn.Linear(256, num_labels),
+        )
+    
+    def forward(self, input_ids, attention_mask):
+        outputs = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
+        cls_output = outputs.last_hidden_state[:, 0, :]
+        return self.classifier(cls_output)
+
+# Load
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+tokenizer = AutoTokenizer.from_pretrained("RowRed/ComplexityRouter")
+model = PromptComplexityRouter()
+model.load_state_dict(
+    torch.load("pytorch_model.bin", map_location=device),
+    strict=False
+)
+model.to(device)
+model.eval()
+
+# Predict
+prompts = ["What is 2+2?", "Explain quantum entanglement in detail."]
+encoded = tokenizer(prompts, padding=True, truncation=True, return_tensors="pt").to(device)
+with torch.no_grad():
+    logits = model(encoded["input_ids"], encoded["attention_mask"])
+    probs = torch.softmax(logits, dim=-1)
+    predictions = torch.argmax(probs, dim=-1)
+
+for prompt, level in zip(prompts, predictions):
+    print(f"Level {level.item()}: {prompt}")
+```
+
+## Citation
+If you use this model, please cite:
+
+```bibtex
+@software{ComplexityRouter,
+  author = {RowRed},
+  title = {ComplexityRouter},
+  year = {2026},
+  url = {https://huggingface.co/RowRed/ComplexityRouter}
+}
+```
+
+Additionally, acknowledge the base dataset and labeling model:
+
+```bibtex
+@dataset{oasst2,
+  author = {OpenAssistant Contributors},
+  title = {Open Assistant Conversations Dataset Release 2},
+  year = {2023},
+  url = {https://huggingface.co/datasets/OpenAssistant/oasst2}
+}
+
+@software{qwen3.5-4b,
+  author = {Qwen Team},
+  title = {Qwen3.5-4B},
+  year = {2026},
+  url = {https://huggingface.co/Qwen/Qwen3.5-4B}
+}
+```
+
+## License
+This model is released under Apache‑2.0.
+The backbone (microsoft/deberta-v3-base) is MIT‑licensed.
+The training dataset is derived from OASST2 (Apache‑2.0) and Qwen3.5‑4B outputs (Apache‑2.0).