Rewrite model card: correct repo references, mixed-training results, intended use & limitations

README.md

@@ -1,51 +1,133 @@
 ---
 base_model: deepseek-ai/deepseek-coder-7b-instruct-v1.5
-library_name: transformers
-model_name: deepseek-sql-magicoder-adapter
+library_name: peft
+model_name: deepseek-coder-7b-instruct-sql-create-context-lora
 tags:
-- generated_from_trainer
-- trl
+- base_model:adapter:deepseek-ai/deepseek-coder-7b-instruct-v1.5
+- lora
 - sft
-licence: license
+- text-to-sql
+- trl
+license: apache-2.0
+pipeline_tag: text-generation
+datasets:
+- b-mc2/sql-create-context
+- ise-uiuc/Magicoder-OSS-Instruct-75K
 ---
 
-# Model Card for deepseek-sql-magicoder-adapter
+# Model Card for deepseek-coder-7b-instruct-sql-create-context-lora
 
-This model is a fine-tuned version of [deepseek-ai/deepseek-coder-7b-instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+LoRA adapter fine-tuned from [deepseek-ai/deepseek-coder-7b-instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5) for text-to-SQL generation, trained with [TRL](https://github.com/huggingface/trl) SFTTrainer.
 
-## Quick start
+## Intended Use & Limitations
+
+**Intended use**: generating a single SQL query from a natural-language question given a `CREATE TABLE` schema, in English, for single or lightly-joined relational databases similar in style to Spider / sql-create-context schemas.
+
+**Limitations**:
+- Not evaluated on production/adversarial inputs, non-English questions, or dialects outside SQLite-compatible syntax.
+- Execution accuracy on Spider (60.8%) means roughly 2 in 5 generated queries on unseen schemas are still wrong — **always validate generated SQL before running it against a real database**, especially for destructive statements (this adapter was only trained/evaluated on read (`SELECT`) queries).
+- Code capability (HumanEval+) is retained better than the pure-SQL version but still ~4pp below the un-finetuned base model — for general-purpose coding tasks unrelated to SQL, the base model remains the stronger choice. See Results below.
+
+## What's on `main` vs `pure-sql`
+
+- **`main` (this version)** — trained on a 50/50 mix of SQL and general code-instruction data. Better SQL generalization to unseen schemas *and* better retention of general code ability than the pure-SQL version.
+- **`pure-sql`** branch — the original version trained on 100% SQL data. Higher in-distribution SQL accuracy on the training-like distribution, but noticeably worse code capability retention and worse generalization to unseen database schemas.
 
 ```python
-from transformers import pipeline
+# to load the pure-SQL version instead:
+PeftModel.from_pretrained(model, adapter_id, revision="pure-sql")
+```
+
+## Why mix in code data
+
+The pure-SQL version showed catastrophic forgetting of general code generation ability:
+
+| Metric | Base | Pure-SQL SFT | Δ |
+|---|---|---|---|
+| HumanEval pass@1 | 52.0% | 40.0% | -12.0pp |
+| HumanEval+ (999 edge cases) | 46.0% | 34.0% | -12.0pp |
+
+Training data was rebalanced to 50% [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) + 50% [ise-uiuc/Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K) (interleaved batch-wise via `datasets.interleave_datasets`), 1 epoch.
+
+## Results
+
+### Code capability retention (n=50, HumanEval/HumanEval+)
+
+| Metric | Base | Pure-SQL SFT | **This version (mixed)** |
+|---|---|---|---|
+| HumanEval pass@1 | 52.0% | 40.0% (-12.0pp) | **44.0% (-8.0pp)** |
+| HumanEval+ (plus) | 46.0% | 34.0% (-12.0pp) | **42.0% (-4.0pp)** |
+
+### SQL generalization on unseen schemas (Spider 1.0, n=1034, real databases + official eval)
+
+| Metric | Base | Pure-SQL SFT | **This version (mixed)** |
+|---|---|---|---|
+| Official Execution Accuracy | 39.9% | 50.4% | **60.8%** |
+| Official Exact Match (structural) | 32.1% | 37.4% | **47.3%** |
+
+This version beats the pure-SQL version on *every* axis measured — code capability retention, SQL generalization to unseen schemas, and in-distribution SQL accuracy (see below). Mixing in code data appears to act as a regularizer against overfitting to the narrow single-domain SQL distribution.
+
+### In-distribution SQL accuracy (b-mc2/sql-create-context, same distribution as training)
+
+| Model | Exact Match |
+|------|-------------|
+| Base | 4% (2/50) |
+| Pure-SQL SFT | 78% (39/50) |
+| This version (mixed) | 70% (14/20, different eval run — see repo `text-to-sql/docs/experiment_log.md` for full methodology) |
+
+## LoRA configuration
 
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="albertkingdom/deepseek-sql-magicoder-adapter", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
+```
+r: 16
+lora_alpha: 32
+lora_dropout: 0.05
+target_modules: [q_proj, k_proj, v_proj, o_proj]
+quantization: 4-bit NF4 (QLoRA), bf16 compute
 ```
 
-## Training procedure
+## Quick start
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+
+base_model_id = "deepseek-ai/deepseek-coder-7b-instruct-v1.5"
+adapter_id = "albertkingdom/deepseek-coder-7b-instruct-sql-create-context-lora"
 
- 
+model = AutoModelForCausalLM.from_pretrained(base_model_id, device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained(base_model_id)
+model = PeftModel.from_pretrained(model, adapter_id)  # main = mixed training version
 
+messages = [{
+    "role": "user",
+    "content": """Given the database schema below, write a SQL query that answers the user's question.
+Only output the SQL query. Do not add any explanation.
 
-This model was trained with SFT.
+### Schema
+CREATE TABLE users (id INT, name VARCHAR(100), email VARCHAR(100))
+
+### Question
+Find all users with gmail addresses"""
+}]
+
+inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
+outputs = model.generate(inputs, max_new_tokens=200)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Training procedure
+
+Trained with SFT (TRL `SFTTrainer`) on an interleaved SQL + code instruction dataset, 1 epoch, effective batch size 16, bf16, 4-bit QLoRA, on a rented RTX 5090.
 
 ### Framework versions
 
+- PEFT 0.18.0
 - TRL: 0.26.2
 - Transformers: 4.57.3
-- Pytorch: 2.13.0
 - Datasets: 4.4.2
-- Tokenizers: 0.22.2
 
 ## Citations
 
-
-
-Cite TRL as:
-    
 ```bibtex
 @misc{vonwerra2022trl,
 	title        = {{TRL: Transformer Reinforcement Learning}},
@@ -55,4 +137,4 @@ Cite TRL as:
 	publisher    = {GitHub},
 	howpublished = {\url{https://github.com/huggingface/trl}}
 }
-```
+```