docs: v0.55 — wiki warmup complete, checkpoint deprecation notice
Browse files
README.md
CHANGED
|
@@ -21,17 +21,28 @@ Non-transformer architecture. No attention mechanism. Constitutional training vi
|
|
| 21 |
|
| 22 |
---
|
| 23 |
|
| 24 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
-
|
|
|
|
|
|
|
| 27 |
|
| 28 |
| Tier | File | Use this when… |
|
| 29 |
|---|---|---|
|
| 30 |
-
| **Base** | `base/Anvaya-Rabbit-2.7B-0.
|
| 31 |
-
| **Instruct** | `instruct/Anvaya-Rabbit-2.7B-0.5-alpha-instruct.pt` | You want a general-purpose assistant that follows instructions |
|
| 32 |
-
| **Imprint** | `imprint/Anvaya-Rabbit-2.7B-0.5-alpha-imprint.pt` | You want the full Rabbit persona — opinionated, constitutional, identity-aware |
|
| 33 |
|
| 34 |
-
|
| 35 |
|
| 36 |
---
|
| 37 |
|
|
@@ -54,15 +65,12 @@ model = AutoModelForCausalLM.from_pretrained(
|
|
| 54 |
device_map="auto",
|
| 55 |
)
|
| 56 |
|
| 57 |
-
# v0.5-alpha uses raw completion format
|
| 58 |
prompt = "Rabbit is a helpful and honest assistant.\n\nUser: Who are you?\nRabbit:"
|
| 59 |
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
| 60 |
outputs = model.generate(**inputs, max_new_tokens=60, repetition_penalty=1.3)
|
| 61 |
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
| 62 |
```
|
| 63 |
|
| 64 |
-
> *v0.5-alpha uses raw completion format. Chat template support (ChatML) coming in v0.9.*
|
| 65 |
-
|
| 66 |
> The `rtaforge` runtime package provides the compiled architecture. Source is not distributed.
|
| 67 |
|
| 68 |
---
|
|
@@ -88,9 +96,10 @@ Rabbit is built on **RtaSSM v7.2.2-FU "Fortress Unbroken"**, a custom state-spac
|
|
| 88 |
|
| 89 |
| Stage | Data | Notes |
|
| 90 |
|---|---|---|
|
| 91 |
-
| Wiki
|
| 92 |
-
|
|
| 93 |
-
|
|
|
|
|
| 94 |
|
| 95 |
---
|
| 96 |
|
|
@@ -112,16 +121,15 @@ Runtime documentation coming soon.
|
|
| 112 |
|
| 113 |
## Maturity and Roadmap
|
| 114 |
|
| 115 |
-
**v0.
|
| 116 |
-
|
| 117 |
Usable conversational behaviour is targeted at **v0.8–v0.9**, currently in training.
|
| 118 |
|
| 119 |
- Evaluating for deployment? Wait for v0.9.
|
| 120 |
-
- Evaluating the architecture or training methodology? v0.
|
| 121 |
|
| 122 |
## Limitations
|
| 123 |
|
| 124 |
-
v0.
|
| 125 |
|
| 126 |
---
|
| 127 |
|
|
|
|
| 21 |
|
| 22 |
---
|
| 23 |
|
| 24 |
+
## ⚠️ Checkpoint Deprecation Notice
|
| 25 |
+
|
| 26 |
+
| Checkpoint | Status | Notes |
|
| 27 |
+
|---|---|---|
|
| 28 |
+
| `Anvaya-Rabbit-2.7B-0.55-base.pt` | ✅ **CURRENT** | Wikipedia warmup complete, CE 0.993x |
|
| 29 |
+
| Any prior checkpoint | ⚠️ **DEPRECATED** | Do not use for inference |
|
| 30 |
+
|
| 31 |
+
Prior checkpoints are retained for research transparency.
|
| 32 |
+
The current checkpoint reflects iterative refinement of the
|
| 33 |
+
ANVAYA RtaSSM architecture and training pipeline.
|
| 34 |
+
|
| 35 |
+
**Always use the latest `-base.pt` for any downstream work.**
|
| 36 |
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
## What's in this repo
|
| 40 |
|
| 41 |
| Tier | File | Use this when… |
|
| 42 |
|---|---|---|
|
| 43 |
+
| **Base** | `base/Anvaya-Rabbit-2.7B-0.55-base.pt` | You want raw pretrained weights for your own fine-tuning |
|
|
|
|
|
|
|
| 44 |
|
| 45 |
+
Instruct and Imprint tiers are in preparation (epoch 2 → SFT → imprint pipeline).
|
| 46 |
|
| 47 |
---
|
| 48 |
|
|
|
|
| 65 |
device_map="auto",
|
| 66 |
)
|
| 67 |
|
|
|
|
| 68 |
prompt = "Rabbit is a helpful and honest assistant.\n\nUser: Who are you?\nRabbit:"
|
| 69 |
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
| 70 |
outputs = model.generate(**inputs, max_new_tokens=60, repetition_penalty=1.3)
|
| 71 |
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
| 72 |
```
|
| 73 |
|
|
|
|
|
|
|
| 74 |
> The `rtaforge` runtime package provides the compiled architecture. Source is not distributed.
|
| 75 |
|
| 76 |
---
|
|
|
|
| 96 |
|
| 97 |
| Stage | Data | Notes |
|
| 98 |
|---|---|---|
|
| 99 |
+
| Wiki warmup (v0.55) | Wikipedia (en) | 700 constitutional proposals via Gurukul — **complete** |
|
| 100 |
+
| Epoch 2 (planned) | RedPajama | Gate-only, ~3,350 proposals |
|
| 101 |
+
| Instruct SFT (planned) | ChatML instruction pairs | `gate_only` trainable strategy |
|
| 102 |
+
| Persona imprint (planned) | Rabbit constitutional corpus | Identity and value alignment |
|
| 103 |
|
| 104 |
---
|
| 105 |
|
|
|
|
| 121 |
|
| 122 |
## Maturity and Roadmap
|
| 123 |
|
| 124 |
+
**v0.55 is a base pretrained checkpoint** — Wikipedia warmup complete, CE ratio 0.993×.
|
|
|
|
| 125 |
Usable conversational behaviour is targeted at **v0.8–v0.9**, currently in training.
|
| 126 |
|
| 127 |
- Evaluating for deployment? Wait for v0.9.
|
| 128 |
+
- Evaluating the architecture or training methodology? v0.55-base is exactly what you need.
|
| 129 |
|
| 130 |
## Limitations
|
| 131 |
|
| 132 |
+
v0.55 has not been evaluated on standard benchmarks. She is small, she is new, and she is learning. Feedback welcome at guha@rtaforge.in.
|
| 133 |
|
| 134 |
---
|
| 135 |
|