docs: fix in-distribution results table, correct comparison claim, add training details

README.md

@@ -28,12 +28,13 @@ LoRA adapter fine-tuned from [deepseek-ai/deepseek-coder-7b-instruct-v1.5](https
 **Limitations**:
 - Not evaluated on production/adversarial inputs, non-English questions, or dialects outside SQLite-compatible syntax.
 - Execution accuracy on Spider (60.8%) means roughly 2 in 5 generated queries on unseen schemas are still wrong — **always validate generated SQL before running it against a real database**, especially for destructive statements (this adapter was only trained/evaluated on read (`SELECT`) queries).
+- **JOIN over-generation**: the SQL training data is almost entirely single-table, so on multi-table schemas the model tends to join every available table indiscriminately. This is the main source of exact-match errors on Spider.
 - Code capability (HumanEval+) is retained better than the pure-SQL version but still ~4pp below the un-finetuned base model — for general-purpose coding tasks unrelated to SQL, the base model remains the stronger choice. See Results below.
 
 ## What's on `main` vs `pure-sql`
 
 - **`main` (this version)** — trained on a 50/50 mix of SQL and general code-instruction data. Better SQL generalization to unseen schemas *and* better retention of general code ability than the pure-SQL version.
-- **`pure-sql`** branch — the original version trained on 100% SQL data. Higher in-distribution SQL accuracy on the training-like distribution, but noticeably worse code capability retention and worse generalization to unseen database schemas.
+- **`pure-sql`** branch — the original version trained on 100% SQL data. Comparable in-distribution SQL accuracy, but noticeably worse code capability retention and worse generalization to unseen database schemas.
 
 ```python
 # to load the pure-SQL version instead:
@@ -49,7 +50,9 @@ The pure-SQL version showed catastrophic forgetting of general code generation a
 | HumanEval pass@1 | 52.0% | 40.0% | -12.0pp |
 | HumanEval+ (999 edge cases) | 46.0% | 34.0% | -12.0pp |
 
-Training data was rebalanced to 50% [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) + 50% [ise-uiuc/Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K) (interleaved batch-wise via `datasets.interleave_datasets`), 1 epoch.
+Training data was rebalanced to 50% [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) + 50% [ise-uiuc/Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K) (interleaved batch-wise via `datasets.interleave_datasets`, `stopping_strategy="first_exhausted"`), 1 epoch.
+
+The two datasets are nearly the same size (~70.7k vs ~75.2k), so a 50/50 ratio consumes essentially all of both — 141,771 training examples, 8,861 optimizer steps.
 
 ## Results
 
@@ -67,15 +70,21 @@ Training data was rebalanced to 50% [b-mc2/sql-create-context](https://huggingfa
 | Official Execution Accuracy | 39.9% | 50.4% | **60.8%** |
 | Official Exact Match (structural) | 32.1% | 37.4% | **47.3%** |
 
-This version beats the pure-SQL version on *every* axis measured — code capability retention, SQL generalization to unseen schemas, and in-distribution SQL accuracy (see below). Mixing in code data appears to act as a regularizer against overfitting to the narrow single-domain SQL distribution.
+Compared with the pure-SQL version, this version generalizes substantially better to unseen schemas (+10.4pp execution accuracy) and retains more code capability, at no cost to in-distribution SQL accuracy (see below). Mixing in code data appears to act as a regularizer against overfitting to the narrow single-domain SQL distribution.
 
 ### In-distribution SQL accuracy (b-mc2/sql-create-context, same distribution as training)
 
-| Model | Exact Match |
+Measured on the same 20-sample held-out set for all three models, in the same evaluation run:
+
+| Model | Exact Match (n=20) |
 |------|-------------|
-| Base | 4% (2/50) |
-| Pure-SQL SFT | 78% (39/50) |
-| This version (mixed) | 70% (14/20, different eval run — see repo `text-to-sql/docs/experiment_log.md` for full methodology) |
+| Base | 0.0% |
+| Pure-SQL SFT | 70.0% |
+| This version (mixed) | 70.0% |
+
+The two fine-tuned versions are **tied** here — mixing in code data cost nothing on the training distribution. Note that n=20 gives a wide confidence interval; the Spider result above (n=1034, real databases, official scripts) is the more reliable measurement.
+
+Earlier numbers on this dataset used different splits and are superseded by the table above. Note that `sql-create-context` ships schemas with no data rows, so execution-based metrics on it mostly measure syntax/schema alignment rather than semantic correctness.
 
 ## LoRA configuration
 
@@ -90,13 +99,16 @@ quantization: 4-bit NF4 (QLoRA), bf16 compute
 ## Quick start
 
 ```python
+import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
 from peft import PeftModel
 
 base_model_id = "deepseek-ai/deepseek-coder-7b-instruct-v1.5"
 adapter_id = "albertkingdom/deepseek-coder-7b-text2sql-magicoder-lora"
 
-model = AutoModelForCausalLM.from_pretrained(base_model_id, device_map="auto")
+model = AutoModelForCausalLM.from_pretrained(
+    base_model_id, torch_dtype=torch.bfloat16, device_map="auto"
+)
 tokenizer = AutoTokenizer.from_pretrained(base_model_id)
 model = PeftModel.from_pretrained(model, adapter_id)  # main = mixed training version
 
@@ -113,20 +125,23 @@ Find all users with gmail addresses"""
 }]
 
 inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
-outputs = model.generate(inputs, max_new_tokens=200)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+outputs = model.generate(inputs, max_new_tokens=200, do_sample=False)
+print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
 ```
 
+The adapter was trained with exactly this instruction + `### Schema` / `### Question` layout wrapped in the DeepSeek chat template. Free-form prompts still work, but accuracy drops.
+
 ## Training procedure
 
-Trained with SFT (TRL `SFTTrainer`) on an interleaved SQL + code instruction dataset, 1 epoch, effective batch size 16, bf16, 4-bit QLoRA, on a rented RTX 5090.
+Trained with SFT (TRL `SFTTrainer`) on an interleaved SQL + code instruction dataset, 1 epoch over 141,771 examples (8,861 steps), learning rate 1e-4 with linear decay and 3% warmup, effective batch size 16 (per-device 4 × gradient accumulation 4), max sequence length 1024, `paged_adamw_8bit`, bf16, 4-bit QLoRA, on a rented RTX 5090 (~12h). Final train loss 0.3986, eval loss 0.3437.
 
 ### Framework versions
 
-- PEFT 0.18.0
+- PEFT: 0.18.0
 - TRL: 0.26.2
 - Transformers: 4.57.3
 - Datasets: 4.4.2
+- PyTorch: 2.9.1+cu128
 
 ## License
 
@@ -160,3 +175,21 @@ Commercial use is otherwise permitted, consistent with the base model's license.
 	howpublished = {\url{https://github.com/huggingface/trl}}
 }
 ```
+
+```bibtex
+@article{yu2018spider,
+	title   = {Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task},
+	author  = {Yu, Tao and Zhang, Rui and Yang, Kai and Yasunaga, Michihiro and Wang, Dongxu and Li, Zifan and Ma, James and Li, Irene and Yao, Qingning and Roman, Shanelle and Zhang, Zilin and Radev, Dragomir},
+	journal = {arXiv preprint arXiv:1809.08887},
+	year    = 2018
+}
+```
+
+```bibtex
+@article{wei2023magicoder,
+	title   = {Magicoder: Empowering Code Generation with OSS-Instruct},
+	author  = {Wei, Yuxiang and Wang, Zhe and Liu, Jiawei and Ding, Yifeng and Zhang, Lingming},
+	journal = {arXiv preprint arXiv:2312.02120},
+	year    = 2023
+}
+```