Model card: 72-28 result, what produced the 12-point gain, limitations

README.md

@@ -0,0 +1,146 @@
+---
+license: llama3.3
+base_model: meta-llama/Llama-3.3-70B-Instruct
+datasets:
+- flamiinngo/math-code-qa-v2
+tags:
+- math
+- code
+- lora
+- peft
+- adapter
+- autoscientist
+- adaption
+language:
+- en
+pipeline_tag: text-generation
+library_name: peft
+---
+
+# Math & Code — Llama-3.3-70B LoRA
+
+A LoRA adapter for **Llama-3.3-70B-Instruct**, fine-tuned to solve mathematical
+problems — arithmetic word problems through algebra, geometry and combinatorics —
+and answer short coding questions.
+
+Trained with **Adaption Labs' AutoScientist** for the AutoScientist Challenge
+(Math & Code category).
+
+## Result
+
+| Evaluation | Base | Adapted |
+|---|---|---|
+| Math category | 28 | **72** |
+| In-distribution test set | 52 | 48 |
+
+Wins in a paired comparison, not accuracy percentages.
+
+The rows disagree, which is worth explaining. On the narrow in-distribution set a
+judge slightly prefers the base model's phrasing. Across the wider category —
+including problems well outside the training distribution — the adapted model wins
+decisively. The mathematical substance generalised further than the answer style.
+
+## What produced the 12-point gain
+
+An earlier version of this model scored **60–28** on the same category evaluation.
+The difference was a single filter in the training data.
+
+v1 capped every solution at 18–75 words. In the upstream corpus, MATH-level
+solutions have a **median length of 121–156 words**, while grade-school word
+problems sit at 89–101. The cap therefore kept only the shortest, easiest examples
+from the hard sources — the model trained almost entirely on arithmetic and was
+then evaluated across the full difficulty range.
+
+Setting the word budget per source (30–150 for algebra and geometry, 18–80 for
+word problems) raised solution p90 from 77 words to 133, and the category win rate
+from 60 to 72.
+
+## Usage
+
+The adapter is stored unpacked and loads directly.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+
+BASE = "meta-llama/Llama-3.3-70B-Instruct"
+ADAPTER = "flamiinngo/adaption_math_word_problems_solutions"
+
+tokenizer = AutoTokenizer.from_pretrained(BASE)
+model = AutoModelForCausalLM.from_pretrained(
+    BASE, torch_dtype=torch.bfloat16, device_map="auto"
+)
+model = PeftModel.from_pretrained(model, ADAPTER)
+model.eval()
+
+messages = [{"role": "user", "content":
+    "Mrs Thompson has 7 Harry Potter books, 6 Twilight books and 5 Hunger Games "
+    "books. Each series must stay together on the shelf. How many orderings are "
+    "there?"}]
+inputs = tokenizer.apply_chat_template(
+    messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
+
+out = model.generate(inputs, max_new_tokens=400, do_sample=False)
+print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
+```
+
+**Hardware:** the 70B base needs roughly 140 GB in bf16, or about 40 GB with 4-bit
+quantisation. The adapter is 3.3 GB.
+
+**Output style:** brief worked steps, then the result stated explicitly as
+"The answer is X."
+
+**Note on the base model name.** `adapter_config.json` records
+`togethercomputer/Meta-Llama-3.3-70B-Instruct-Reference`, the base as served during
+training. Same architecture — load against `meta-llama/Llama-3.3-70B-Instruct`.
+
+## Training
+
+| Parameter | Value |
+|---|---|
+| Base | `meta-llama/Llama-3.3-70B-Instruct` |
+| Rank (`r`) | 64 |
+| `lora_alpha` | 128 |
+| Target modules | all-linear |
+| Epochs | 3 |
+| Peak learning rate | 1e-4, cosine |
+
+## Dataset
+
+[**flamiinngo/math-code-qa-v2**](https://huggingface.co/datasets/flamiinngo/math-code-qa-v2)
+— 5,297 rows (4,197 math, 1,100 code), every math answer ending in a result
+verified against the upstream `expected_answer` column. Derived from
+[nvidia/OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2)
+and [sahil2801/CodeAlpaca-20k](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k),
+both CC-BY-4.0.
+
+Also on Kaggle:
+[model](https://www.kaggle.com/models/flamiinngo/adaption_math-41946c32-256d-4d24-a1ee-6effb91b690c) ·
+[dataset](https://www.kaggle.com/datasets/flamiinngo/math-code-qa-v2)
+
+## Limitations
+
+- **It can produce confident wrong reasoning.** The training solutions are
+  model-generated upstream; only their final answers were verified. Errors in
+  algebraic reasoning exist in the data and this model reproduces that style of
+  mistake. Check any result that matters.
+- **Not a calculator.** Fine-tuning improved the working, not arithmetic guarantees.
+- **Scope is school through early-undergraduate.** Not olympiad or research
+  mathematics.
+- **Code output is untested.** The code training data was filtered for length,
+  not executed. Treat generated code as a draft.
+- **Win rate is not accuracy.** It measures preference against one base model on
+  one evaluation.
+- **English only.**
+
+## License
+
+The adapter is a derivative of Llama-3.3-70B-Instruct and is subject to the
+**Llama 3.3 Community License**. The training data is CC-BY-4.0.
+
+## Acknowledgements
+
+- **Adaption Labs** — AutoScientist platform and the challenge
+- **NVIDIA** and **sahil2801** — upstream open datasets
+- **Meta** — Llama 3.3 base model