Lead with the adaptation loop; add 'when to use this'
Browse files
README.md
CHANGED
|
@@ -71,25 +71,63 @@ model-index:
|
|
| 71 |
---
|
| 72 |
# 🧵 Thimble
|
| 73 |
|
| 74 |
-
**Tool calling in 48M parameters
|
| 75 |
-
app-intent catalog, 100% well-formed JSON by construction.
|
| 76 |
|
| 77 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |

|
| 80 |
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
## What it does
|
| 87 |
|
| 88 |
Ordered strict exact match: a row passes only if the function names, the call
|
| 89 |
-
order, and *every* argument value match. The right-hand column is a yardstick,
|
| 90 |
-
rival: Needle 2 (Cactus Compute, 45M params, 153B training tokens), their
|
| 91 |
-
published numbers on their metric. It is there so the left column has a scale
|
| 92 |
-
86.3 means little until you know what else scores on that suite.
|
| 93 |
|
| 94 |
**Known catalog** — represented in training, eval rows firewalled out:
|
| 95 |
|
|
@@ -107,19 +145,18 @@ published numbers on their metric. It is there so the left column has a scale
|
|
| 107 |
| Seal-Tools out-of-domain (654) | 28.1 | 28.7 |
|
| 108 |
| BFCL v4 single-turn (3,641) | 23.5 | 42.6 |
|
| 109 |
|
| 110 |
-
Those two tables are the
|
| 111 |
-
catalog, it degrades —
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
|
| 116 |
**Before quoting the table.** Mobile Actions' public train split (8,693 rows,
|
| 117 |
-
disjoint from eval) is in the training mix — that is what "known catalog" means
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
[RESULTS.md](https://github.com/nikshepsvn/thimble/blob/master/RESULTS.md).
|
| 123 |
|
| 124 |
## How it was built
|
| 125 |
|
|
|
|
| 71 |
---
|
| 72 |
# 🧵 Thimble
|
| 73 |
|
| 74 |
+
**Tool calling in 48M parameters — small enough to specialize per domain.**
|
|
|
|
| 75 |
|
| 76 |
+
Most tool-calling models are trained once and prompted everywhere. This one is
|
| 77 |
+
built to be *re-specialized*: at 48M parameters, adapting it to your own tool
|
| 78 |
+
catalog is a few hours on one GPU and roughly $60 of synthesis, so you can have
|
| 79 |
+
a model **per** domain instead of a prompt per domain.
|
| 80 |
+
|
| 81 |
+
[**GitHub — code, adaptation loop, full experimental record**](https://github.com/nikshepsvn/thimble) · MIT · 48.12M params · 768-token context
|
| 82 |
|
| 83 |

|
| 84 |
|
| 85 |
+
## Make it yours
|
| 86 |
+
|
| 87 |
+
```bash
|
| 88 |
+
git clone https://github.com/nikshepsvn/thimble && cd thimble
|
| 89 |
+
uv venv && uv pip install -e .
|
| 90 |
+
# put thimble-v6.pt in checkpoints/, tokenizer.json in data/
|
| 91 |
+
|
| 92 |
+
# where do you stand on your own tools?
|
| 93 |
+
python scripts/eval_catalog.py --ckpt thimble-v6 \
|
| 94 |
+
--catalog my_tools.json --gold my_eval.jsonl
|
| 95 |
+
|
| 96 |
+
# adapt: synthesize against your schemas, anneal into the decay phase
|
| 97 |
+
python scripts/adapt.py --catalog my_tools.json --name mydomain
|
| 98 |
+
|
| 99 |
+
# did it help?
|
| 100 |
+
python scripts/eval_catalog.py --ckpt mydomain --baseline thimble-v6 \
|
| 101 |
+
--catalog my_tools.json --gold my_eval.jsonl
|
| 102 |
+
```
|
| 103 |
+
|
| 104 |
+
Two things about that recipe are load-bearing, and both were measured rather
|
| 105 |
+
than assumed. **Anneal, don't retrain** — the same corrective corpus scored 28.4
|
| 106 |
+
fed from scratch and 33.1 annealed into the LR-decay phase. **Keep the guard
|
| 107 |
+
data** — annealing purely on your catalog trades away the general competence you
|
| 108 |
+
are building on.
|
| 109 |
+
|
| 110 |
+
`adapt.py` wires together exactly the machinery that produced the v6 result, but
|
| 111 |
+
no third-party catalog has been adapted and published yet. The recipe is
|
| 112 |
+
measured; the ergonomics are new.
|
| 113 |
+
|
| 114 |
+
## When to use this
|
| 115 |
+
|
| 116 |
+
Reach for it when you have a **fixed catalog you control**, English queries,
|
| 117 |
+
argument values that appear in the query, and a reason to care about 48M
|
| 118 |
+
parameters — a memory ceiling, a latency floor, or wanting a separate model per
|
| 119 |
+
customer rather than one prompted model for all of them.
|
| 120 |
+
|
| 121 |
+
Do not reach for it for open-world catalogs, Java or JavaScript schema dialects,
|
| 122 |
+
or parallel calls. And if you can afford 600M parameters, fine-tune Qwen
|
| 123 |
+
instead — it will probably score higher. This is for when you cannot.
|
| 124 |
|
| 125 |
## What it does
|
| 126 |
|
| 127 |
Ordered strict exact match: a row passes only if the function names, the call
|
| 128 |
+
order, and *every* argument value match. The right-hand column is a yardstick,
|
| 129 |
+
not a rival: Needle 2 (Cactus Compute, 45M params, 153B training tokens), their
|
| 130 |
+
published numbers on their metric. It is there so the left column has a scale.
|
|
|
|
| 131 |
|
| 132 |
**Known catalog** — represented in training, eval rows firewalled out:
|
| 133 |
|
|
|
|
| 145 |
| Seal-Tools out-of-domain (654) | 28.1 | 28.7 |
|
| 146 |
| BFCL v4 single-turn (3,641) | 23.5 | 42.6 |
|
| 147 |
|
| 148 |
+
Those two tables are why the adaptation loop exists. Familiar catalog, it works;
|
| 149 |
+
unfamiliar catalog, it degrades — measurably, *inside a single suite*: 33.1
|
| 150 |
+
in-domain versus 28.1 out, same model, same metric, only the catalogs changed.
|
| 151 |
+
Name-sequence accuracy tracks it exactly, 88% in-domain against 79% out. Getting
|
| 152 |
+
your catalog into the first column is the whole job.
|
| 153 |
|
| 154 |
**Before quoting the table.** Mobile Actions' public train split (8,693 rows,
|
| 155 |
+
disjoint from eval) is in the training mix — that is what "known catalog" means.
|
| 156 |
+
The Seal-in margin over the yardstick is +0.5 on 700 rows, within sampling
|
| 157 |
+
noise. The pre-registered selector picked a sibling checkpoint that scored
|
| 158 |
+
worse; the failure is diagnosed and both models' results are published in
|
| 159 |
+
[FINDINGS.md](https://github.com/nikshepsvn/thimble/blob/master/FINDINGS.md).
|
|
|
|
| 160 |
|
| 161 |
## How it was built
|
| 162 |
|