flashvenom commited on
Commit
fd5f34e
·
verified ·
1 Parent(s): 1ea6dac

Lead with the adaptation loop; add 'when to use this'

Browse files
Files changed (1) hide show
  1. README.md +59 -22
README.md CHANGED
@@ -71,25 +71,63 @@ model-index:
71
  ---
72
  # 🧵 Thimble
73
 
74
- **Tool calling in 48M parameters.** 86.3% ordered strict exact match on a real
75
- app-intent catalog, 100% well-formed JSON by construction.
76
 
77
- [**GitHub (code, evals, full experimental record)**](https://github.com/nikshepsvn/thimble) · MIT · 48.12M params · 768-token context · $260 total build cost
 
 
 
 
 
78
 
79
  ![Results](results.png)
80
 
81
- Calling tools against a *known* catalog is not an emergent capability of large
82
- models — it is a structured extraction problem, and it fits in 48M parameters.
83
- This card covers what the model does, how it was built, and exactly where it
84
- stops working.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
 
86
  ## What it does
87
 
88
  Ordered strict exact match: a row passes only if the function names, the call
89
- order, and *every* argument value match. The right-hand column is a yardstick, not a
90
- rival: Needle 2 (Cactus Compute, 45M params, 153B training tokens), their
91
- published numbers on their metric. It is there so the left column has a scale
92
- 86.3 means little until you know what else scores on that suite.
93
 
94
  **Known catalog** — represented in training, eval rows firewalled out:
95
 
@@ -107,19 +145,18 @@ published numbers on their metric. It is there so the left column has a scale
107
  | Seal-Tools out-of-domain (654) | 28.1 | 28.7 |
108
  | BFCL v4 single-turn (3,641) | 23.5 | 42.6 |
109
 
110
- Those two tables are the whole finding. Familiar catalog, it works; unfamiliar
111
- catalog, it degrades — and the degradation shows up *inside a single suite*:
112
- Seal-Tools in-domain 33.1 vs out-of-domain 28.1 is the same model on the same
113
- metric with only the catalogs changed. Name-sequence accuracy tracks it exactly,
114
- 88% in-domain against 79% out.
115
 
116
  **Before quoting the table.** Mobile Actions' public train split (8,693 rows,
117
- disjoint from eval) is in the training mix — that is what "known catalog" means,
118
- and it is the intended operating condition. The Seal-in margin over the
119
- calibration column is +0.5 on 700 rows, within sampling noise. The pre-registered
120
- model selector picked a sibling checkpoint that scored worse; the failure is
121
- diagnosed and both models' results are published in
122
- [RESULTS.md](https://github.com/nikshepsvn/thimble/blob/master/RESULTS.md).
123
 
124
  ## How it was built
125
 
 
71
  ---
72
  # 🧵 Thimble
73
 
74
+ **Tool calling in 48M parameters small enough to specialize per domain.**
 
75
 
76
+ Most tool-calling models are trained once and prompted everywhere. This one is
77
+ built to be *re-specialized*: at 48M parameters, adapting it to your own tool
78
+ catalog is a few hours on one GPU and roughly $60 of synthesis, so you can have
79
+ a model **per** domain instead of a prompt per domain.
80
+
81
+ [**GitHub — code, adaptation loop, full experimental record**](https://github.com/nikshepsvn/thimble) · MIT · 48.12M params · 768-token context
82
 
83
  ![Results](results.png)
84
 
85
+ ## Make it yours
86
+
87
+ ```bash
88
+ git clone https://github.com/nikshepsvn/thimble && cd thimble
89
+ uv venv && uv pip install -e .
90
+ # put thimble-v6.pt in checkpoints/, tokenizer.json in data/
91
+
92
+ # where do you stand on your own tools?
93
+ python scripts/eval_catalog.py --ckpt thimble-v6 \
94
+ --catalog my_tools.json --gold my_eval.jsonl
95
+
96
+ # adapt: synthesize against your schemas, anneal into the decay phase
97
+ python scripts/adapt.py --catalog my_tools.json --name mydomain
98
+
99
+ # did it help?
100
+ python scripts/eval_catalog.py --ckpt mydomain --baseline thimble-v6 \
101
+ --catalog my_tools.json --gold my_eval.jsonl
102
+ ```
103
+
104
+ Two things about that recipe are load-bearing, and both were measured rather
105
+ than assumed. **Anneal, don't retrain** — the same corrective corpus scored 28.4
106
+ fed from scratch and 33.1 annealed into the LR-decay phase. **Keep the guard
107
+ data** — annealing purely on your catalog trades away the general competence you
108
+ are building on.
109
+
110
+ `adapt.py` wires together exactly the machinery that produced the v6 result, but
111
+ no third-party catalog has been adapted and published yet. The recipe is
112
+ measured; the ergonomics are new.
113
+
114
+ ## When to use this
115
+
116
+ Reach for it when you have a **fixed catalog you control**, English queries,
117
+ argument values that appear in the query, and a reason to care about 48M
118
+ parameters — a memory ceiling, a latency floor, or wanting a separate model per
119
+ customer rather than one prompted model for all of them.
120
+
121
+ Do not reach for it for open-world catalogs, Java or JavaScript schema dialects,
122
+ or parallel calls. And if you can afford 600M parameters, fine-tune Qwen
123
+ instead — it will probably score higher. This is for when you cannot.
124
 
125
  ## What it does
126
 
127
  Ordered strict exact match: a row passes only if the function names, the call
128
+ order, and *every* argument value match. The right-hand column is a yardstick,
129
+ not a rival: Needle 2 (Cactus Compute, 45M params, 153B training tokens), their
130
+ published numbers on their metric. It is there so the left column has a scale.
 
131
 
132
  **Known catalog** — represented in training, eval rows firewalled out:
133
 
 
145
  | Seal-Tools out-of-domain (654) | 28.1 | 28.7 |
146
  | BFCL v4 single-turn (3,641) | 23.5 | 42.6 |
147
 
148
+ Those two tables are why the adaptation loop exists. Familiar catalog, it works;
149
+ unfamiliar catalog, it degrades — measurably, *inside a single suite*: 33.1
150
+ in-domain versus 28.1 out, same model, same metric, only the catalogs changed.
151
+ Name-sequence accuracy tracks it exactly, 88% in-domain against 79% out. Getting
152
+ your catalog into the first column is the whole job.
153
 
154
  **Before quoting the table.** Mobile Actions' public train split (8,693 rows,
155
+ disjoint from eval) is in the training mix — that is what "known catalog" means.
156
+ The Seal-in margin over the yardstick is +0.5 on 700 rows, within sampling
157
+ noise. The pre-registered selector picked a sibling checkpoint that scored
158
+ worse; the failure is diagnosed and both models' results are published in
159
+ [FINDINGS.md](https://github.com/nikshepsvn/thimble/blob/master/FINDINGS.md).
 
160
 
161
  ## How it was built
162