| --- |
| language: |
| - en |
| license: apache-2.0 |
| tags: |
| - ssm |
| - state-space-model |
| - causal-lm |
| - rabbit |
| - rtaforge |
| - india |
| - sovereign-ai |
| pipeline_tag: text-generation |
| --- |
| |
| # Anvaya-Rabbit 2.7B |
|
|
| **India's first sovereign SSM-based language model.** |
|
|
| Non-transformer architecture. No attention mechanism. Constitutional training via Gurukul. 7 patents filed at IP India. |
|
|
| --- |
|
|
| ## ⚠️ Checkpoint Deprecation Notice |
|
|
| | Checkpoint | Status | Notes | |
| |---|---|---| |
| | `Anvaya-Rabbit-2.7B-0.55-base.pt` | ✅ **CURRENT** | Wikipedia warmup complete, CE 0.993x | |
| | Any prior checkpoint | ⚠️ **DEPRECATED** | Do not use for inference | |
|
|
| Prior checkpoints are retained for research transparency. |
| The current checkpoint reflects iterative refinement of the |
| ANVAYA RtaSSM architecture and training pipeline. |
|
|
| **Always use the latest `-base.pt` for any downstream work.** |
|
|
| --- |
|
|
| ## What's in this repo |
|
|
| | Tier | File | Use this when… | |
| |---|---|---| |
| | **Base** | `base/Anvaya-Rabbit-2.7B-0.55-base.pt` | You want raw pretrained weights for your own fine-tuning | |
|
|
| Instruct and Imprint tiers are in preparation (epoch 2 → SFT → imprint pipeline). |
|
|
| --- |
|
|
| ## Quickstart |
|
|
| ```bash |
| pip install rtaforge transformers |
| ``` |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b") |
| tokenizer.add_special_tokens({"additional_special_tokens": ["<|im_start|>", "<|im_end|>"]}) |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "RtaForge/Anvaya-Rabbit-2.7B", |
| trust_remote_code=True, |
| torch_dtype="bfloat16", |
| device_map="auto", |
| ) |
| |
| prompt = "Rabbit is a helpful and honest assistant.\n\nUser: Who are you?\nRabbit:" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| outputs = model.generate(**inputs, max_new_tokens=60, repetition_penalty=1.3) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| > The `rtaforge` runtime package provides the compiled architecture. Source is not distributed. |
|
|
| --- |
|
|
| ## Why SSM? |
|
|
| > Transformers scale quadratically with context length because every token attends to every other token. SSMs replace attention with a fixed-size recurrent state: inference cost stays **constant per token** regardless of context length, VRAM footprint shrinks dramatically, and long-document throughput improves by orders of magnitude — all at the same parameter count. |
|
|
| --- |
|
|
| ## Architecture |
|
|
| Rabbit is built on **RtaSSM v7.2.2-FU "Fortress Unbroken"**, a custom state-space model developed at RtaForge: |
|
|
| - **No attention mechanism** — purely recurrent SSM layers with learned state dynamics |
| - **64 layers, 2560 hidden dimensions**, 2.7B parameters, bfloat16 |
| - **Constitutional training** — Gurukul curriculum with wiki pretraining → instruct SFT → persona imprint |
| - **Vocabulary** 50,280 tokens (GPT-NeoX tokenizer) |
|
|
| --- |
|
|
| ## Training |
|
|
| | Stage | Data | Notes | |
| |---|---|---| |
| | Wiki warmup (v0.55) | Wikipedia (en) | 700 constitutional proposals via Gurukul — **complete** | |
| | Epoch 2 (planned) | RedPajama | Gate-only, ~3,350 proposals | |
| | Instruct SFT (planned) | ChatML instruction pairs | `gate_only` trainable strategy | |
| | Persona imprint (planned) | Rabbit constitutional corpus | Identity and value alignment | |
|
|
| --- |
|
|
| ## Evaluation Access |
|
|
| Weights are publicly available. Runtime package is live: |
|
|
| ```bash |
| pip install rtaforge |
| ``` |
|
|
| To evaluate Rabbit or discuss deployment: |
| 📧 guha@rtaforge.in |
| 🌐 rtaforge.in |
|
|
| Runtime documentation coming soon. |
|
|
| --- |
|
|
| ## Maturity and Roadmap |
|
|
| **v0.55 is a base pretrained checkpoint** — Wikipedia warmup complete, CE ratio 0.993×. |
| Usable conversational behaviour is targeted at **v0.8–v0.9**, currently in training. |
|
|
| - Evaluating for deployment? Wait for v0.9. |
| - Evaluating the architecture or training methodology? v0.55-base is exactly what you need. |
|
|
| ## Limitations |
|
|
| v0.55 has not been evaluated on standard benchmarks. She is small, she is new, and she is learning. Feedback welcome at guha@rtaforge.in. |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{anvaya-rabbit-2026, |
| title = {Anvaya-Rabbit: A Sovereign SSM Language Model}, |
| author = {RtaForge}, |
| year = {2026}, |
| url = {https://huggingface.co/RtaForge/Anvaya-Rabbit-2.7B} |
| } |
| ``` |
|
|
| --- |
|
|
| *Anvaya (अन्वय) — logical connection, coherence. Rabbit — the fast runner.* |
|
|
