Add highlights and model details, refresh comparison tables

#2
by anurag051194 - opened
Files changed (1) hide show
  1. README.md +55 -8
README.md CHANGED
@@ -33,6 +33,54 @@ why it is the recommended release of the pair.
33
 
34
  ![TwIL-LM3 formal and general reasoning benchmarks against gpt-oss-120b, Qwen3-8B, LFM2-2.6B and Llama-3.2-3B](benchmarks.jpg)
35
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  ## Results
37
 
38
  ### Track A — in-domain formal logic
@@ -167,9 +215,7 @@ truncation artefact rather than a measured score; excluding the row, its 13-data
167
 
168
  Lengths marked ≈ are derived from stored generations using each model's characters-per-token
169
  ratio rather than re-tokenized directly; the method reproduces the three directly measured
170
- lengths to within 3.5%. All the models are evaluated on our evaluation harness with 300 randomly shuffled
171
- samples (samples are consistent and same for all the models) from each of the datasets for quick compute. The results might
172
- vary on different test set sizes but the comparitive accuracies are statistically significant across models.
173
 
174
  The honest summary of this table is that TwIL-LM3 does not lead it. Larger models score higher,
175
  in order of size, and the 120B leads nine of fourteen rows. Two things are worth extracting
@@ -302,11 +348,12 @@ identity so a mismatched runner fails loudly instead of quietly producing a diff
302
 
303
  ## Relationship to TwIL-LM
304
 
305
- [`webAI-Official/TwIL-LM`](https://huggingface.co/webAI-Official/TwIL-LM) is the 1.7B member of
306
- this family, built from SmolLM2 by the same pipeline. It reaches a higher in-domain score
307
- relative to its own base but gives back held-out capability; this model is the one that improves
308
- both. Unlike TwIL-LM's `main` branch, which ships a PEFT LoRA adapter, this repository ships a
309
- full merged model loaded directly with `AutoModelForCausalLM`.
 
310
 
311
  ## License and attribution
312
 
 
33
 
34
  ![TwIL-LM3 formal and general reasoning benchmarks against gpt-oss-120b, Qwen3-8B, LFM2-2.6B and Llama-3.2-3B](benchmarks.jpg)
35
 
36
+ ## Highlights
37
+
38
+ * **Gains on both tracks at once** — the only arm in this project that does. In-domain macro gate
39
+ 0.336 → 0.422, and the held-out 10-dataset macro 0.7193 → 0.7339 rather than the usual collapse
40
+ that follows task-specific fine-tuning.
41
+ * **Beats every arm up to and including LFM2.5-8B-A1B** — roughly three times its parameter count
42
+ — on all six Track A objective lanes and all four summary rows, not on average alone.
43
+ * **Competitive with 8B on strict scoring.** On strict-7, which gives no loose-match credit
44
+ anywhere, it sits 0.012 behind Qwen3-8B (0.1971 against 0.2093) at 2.6x fewer parameters, and
45
+ ahead of it on Lean formalisation (token-F1 0.5869 against 0.4022) and semantic parsing (0.4416
46
+ against 0.4257).
47
+ * **Structured formal output.** Tuned for the objects rather than the prose: FOL translation,
48
+ entailment labels, semantic parses, Lean statements and Lean proof critique.
49
+ * **The most efficient arm measured, at any scale.** 482-token Track B generations and 32.9
50
+ completed answers per second — about eight times gpt-oss-120b's rate — because it answers
51
+ short rather than because it decodes unusually fast.
52
+ * **Lowest maths-corpus perplexity of any released arm in the table** (3.8229), including
53
+ Qwen3-8B at 4.0083.
54
+ * **Runs anywhere.** 3.08B parameters in bf16, with Q4\_K\_M GGUF at 1.78 GiB for CPU or 4 GB of
55
+ VRAM.
56
+
57
+ It is not a general assistant: there is no safety or preference tuning here beyond what SmolLM3
58
+ carries, and instruction following regressed slightly. See
59
+ [Limitations](#limitations-and-caveats).
60
+
61
+ ## Model Details
62
+
63
+ | Property | Value |
64
+ | ------------------------- | ------------------------------------------------------------------------------------------- |
65
+ | Model ID | `webAI-Official/TwIL-LM3` |
66
+ | Base model | [`HuggingFaceTB/SmolLM3-3B`](https://huggingface.co/HuggingFaceTB/SmolLM3-3B) |
67
+ | Total parameters | 3.08B |
68
+ | Architecture | SmolLM3 decoder-only transformer; 36 layers, hidden size 2048 |
69
+ | Input / output | Text / text |
70
+ | Language | English |
71
+ | Tokenizer vocabulary size | 128,256 |
72
+ | Context window | 65,536 tokens |
73
+ | Checkpoint precision | bfloat16 (5.73 GiB), plus Q4\_K\_M / Q5\_K\_M / Q6\_K / Q8\_0 / F16 GGUF builds |
74
+ | Post-training | LoRA SFT → checkpoint fusion → WiSE-FT (λ = 0.25) → MGPO reinforcement learning (step 2071) |
75
+ | Reasoning format | Emits a `<think>…</think>` block before the answer |
76
+ | Evaluated decoding | Greedy, 2048 new tokens, `max_seq_len` 8192 |
77
+ | Specialisation | Formal logic: FOL translation, entailment, semantic parsing, Lean formalisation and critique |
78
+ | License | webAI Non-Commercial License ver. 1.0 |
79
+
80
+ The base model's 65,536-token context is carried through unchanged, but every score on this card
81
+ was measured inside an 8,192-token window; longer contexts are inherited rather than validated
82
+ here.
83
+
84
  ## Results
85
 
86
  ### Track A — in-domain formal logic
 
215
 
216
  Lengths marked ≈ are derived from stored generations using each model's characters-per-token
217
  ratio rather than re-tokenized directly; the method reproduces the three directly measured
218
+ lengths to within 3.5%.
 
 
219
 
220
  The honest summary of this table is that TwIL-LM3 does not lead it. Larger models score higher,
221
  in order of size, and the 120B leads nine of fourteen rows. Two things are worth extracting
 
348
 
349
  ## Relationship to TwIL-LM
350
 
351
+ [**TwIL-LM2**](https://huggingface.co/webAI-Official/TwIL-LM) is the 1.7B member of this family, built
352
+ from SmolLM2 by the same pipeline. It reaches a higher in-domain score relative to its own base —
353
+ and leads every arm we have measured on Track A strict-7, at any size but it gives back
354
+ held-out capability; this model is the one that improves both. Both repositories now ship full
355
+ merged models on `main`, loaded directly with `AutoModelForCausalLM`; the original LoRA-adapter
356
+ release is archived on that repository's `TwIL-LM1` branch.
357
 
358
  ## License and attribution
359