README.md

---
base_model: deepseek-ai/deepseek-coder-7b-instruct-v1.5
library_name: peft
model_name: deepseek-coder-7b-text2sql-magicoder-lora
tags:
- base_model:adapter:deepseek-ai/deepseek-coder-7b-instruct-v1.5
- lora
- sft
- text-to-sql
- trl
license: other
license_name: deepseek
license_link: https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/LICENSE-MODEL
pipeline_tag: text-generation
datasets:
- b-mc2/sql-create-context
- ise-uiuc/Magicoder-OSS-Instruct-75K
---

# Model Card for deepseek-coder-7b-text2sql-magicoder-lora

LoRA adapter fine-tuned from [deepseek-ai/deepseek-coder-7b-instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5) for text-to-SQL generation, trained with [TRL](https://github.com/huggingface/trl) SFTTrainer.

## Intended Use & Limitations

**Intended use**: generating a single SQL query from a natural-language question given a `CREATE TABLE` schema, in English, for single or lightly-joined relational databases similar in style to Spider / sql-create-context schemas.

**Limitations**:
- Not evaluated on production/adversarial inputs, non-English questions, or dialects outside SQLite-compatible syntax.
- Execution accuracy on Spider (60.8%) means roughly 2 in 5 generated queries on unseen schemas are still wrong — **always validate generated SQL before running it against a real database**, especially for destructive statements (this adapter was only trained/evaluated on read (`SELECT`) queries).
- Code capability (HumanEval+) is retained better than the pure-SQL version but still ~4pp below the un-finetuned base model — for general-purpose coding tasks unrelated to SQL, the base model remains the stronger choice. See Results below.

## What's on `main` vs `pure-sql`

- **`main` (this version)** — trained on a 50/50 mix of SQL and general code-instruction data. Better SQL generalization to unseen schemas *and* better retention of general code ability than the pure-SQL version.
- **`pure-sql`** branch — the original version trained on 100% SQL data. Higher in-distribution SQL accuracy on the training-like distribution, but noticeably worse code capability retention and worse generalization to unseen database schemas.

```python
# to load the pure-SQL version instead:
PeftModel.from_pretrained(model, adapter_id, revision="pure-sql")
```

## Why mix in code data

The pure-SQL version showed catastrophic forgetting of general code generation ability:

| Metric | Base | Pure-SQL SFT | Δ |
|---|---|---|---|
| HumanEval pass@1 | 52.0% | 40.0% | -12.0pp |
| HumanEval+ (999 edge cases) | 46.0% | 34.0% | -12.0pp |

Training data was rebalanced to 50% [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) + 50% [ise-uiuc/Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K) (interleaved batch-wise via `datasets.interleave_datasets`), 1 epoch.

## Results

### Code capability retention (n=50, HumanEval/HumanEval+)

| Metric | Base | Pure-SQL SFT | **This version (mixed)** |
|---|---|---|---|
| HumanEval pass@1 | 52.0% | 40.0% (-12.0pp) | **44.0% (-8.0pp)** |
| HumanEval+ (plus) | 46.0% | 34.0% (-12.0pp) | **42.0% (-4.0pp)** |

### SQL generalization on unseen schemas (Spider 1.0, n=1034, real databases + official eval)

| Metric | Base | Pure-SQL SFT | **This version (mixed)** |
|---|---|---|---|
| Official Execution Accuracy | 39.9% | 50.4% | **60.8%** |
| Official Exact Match (structural) | 32.1% | 37.4% | **47.3%** |

This version beats the pure-SQL version on *every* axis measured — code capability retention, SQL generalization to unseen schemas, and in-distribution SQL accuracy (see below). Mixing in code data appears to act as a regularizer against overfitting to the narrow single-domain SQL distribution.

### In-distribution SQL accuracy (b-mc2/sql-create-context, same distribution as training)

| Model | Exact Match |
|------|-------------|
| Base | 4% (2/50) |
| Pure-SQL SFT | 78% (39/50) |
| This version (mixed) | 70% (14/20, different eval run — see repo `text-to-sql/docs/experiment_log.md` for full methodology) |

## LoRA configuration

```
r: 16
lora_alpha: 32
lora_dropout: 0.05
target_modules: [q_proj, k_proj, v_proj, o_proj]
quantization: 4-bit NF4 (QLoRA), bf16 compute
```

## Quick start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "deepseek-ai/deepseek-coder-7b-instruct-v1.5"
adapter_id = "albertkingdom/deepseek-coder-7b-text2sql-magicoder-lora"

model = AutoModelForCausalLM.from_pretrained(base_model_id, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(model, adapter_id)  # main = mixed training version

messages = [{
    "role": "user",
    "content": """Given the database schema below, write a SQL query that answers the user's question.
Only output the SQL query. Do not add any explanation.

### Schema
CREATE TABLE users (id INT, name VARCHAR(100), email VARCHAR(100))

### Question
Find all users with gmail addresses"""
}]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training procedure

Trained with SFT (TRL `SFTTrainer`) on an interleaved SQL + code instruction dataset, 1 epoch, effective batch size 16, bf16, 4-bit QLoRA, on a rented RTX 5090.

### Framework versions

- PEFT 0.18.0
- TRL: 0.26.2
- Transformers: 4.57.3
- Datasets: 4.4.2

## License

This adapter is a derivative of [deepseek-ai/deepseek-coder-7b-instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5), which is released under the [DeepSeek Model License](https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/LICENSE-MODEL) rather than a standard open-source license. Per that license, derivative models must carry forward at least the same use-based restrictions, so this adapter — and any model merged/derived from it — inherits them:

- No use for military purposes.
- No use that harms minors.
- No generation of false information intended to harm others.
- No creation of non-consensual personal identifiable information.
- No fully automated decision-making that adversely affects an individual's legal rights.
- No discrimination based on protected characteristics.
- See the [full license text](https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/LICENSE-MODEL) (Attachment A) for the complete list.

Commercial use is otherwise permitted, consistent with the base model's license.

## Credits & Data Provenance

- **Base model**: [deepseek-ai/deepseek-coder-7b-instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5) (DeepSeek Model License)
- **[b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context)** (CC-BY-4.0) — itself derived from [WikiSQL](https://github.com/salesforce/WikiSQL) and [Spider](https://yale-lily.github.io/spider); credit to both original sources per CC-BY-4.0 attribution terms.
- **[ise-uiuc/Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K)** (MIT) — generated via the OSS-Instruct method using `gpt-3.5-turbo-1106`. Outputs are subject to [OpenAI's usage policies](https://openai.com/policies/usage-policies) in addition to the dataset's own MIT license.

## Citations

```bibtex
@misc{vonwerra2022trl,
	title        = {{TRL: Transformer Reinforcement Learning}},
	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
	year         = 2020,
	journal      = {GitHub repository},
	publisher    = {GitHub},
	howpublished = {\url{https://github.com/huggingface/trl}}
}
```