Update quick-start snippet to reference renamed repo

dc4e2b0 verified about 1 month ago

7.92 kB

base_model: deepseek-ai/deepseek-coder-7b-instruct-v1.5
library_name: peft
model_name: deepseek-coder-7b-text2sql-magicoder-lora
tags:
  - base_model:adapter:deepseek-ai/deepseek-coder-7b-instruct-v1.5
  - lora
  - sft
  - text-to-sql
  - trl
license: other
license_name: deepseek
license_link: https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/LICENSE-MODEL
pipeline_tag: text-generation
datasets:
  - b-mc2/sql-create-context
  - ise-uiuc/Magicoder-OSS-Instruct-75K

Model Card for deepseek-coder-7b-text2sql-magicoder-lora

LoRA adapter fine-tuned from deepseek-ai/deepseek-coder-7b-instruct-v1.5 for text-to-SQL generation, trained with TRL SFTTrainer.

Intended Use & Limitations

Intended use: generating a single SQL query from a natural-language question given a CREATE TABLE schema, in English, for single or lightly-joined relational databases similar in style to Spider / sql-create-context schemas.

Limitations:

Not evaluated on production/adversarial inputs, non-English questions, or dialects outside SQLite-compatible syntax.
Execution accuracy on Spider (60.8%) means roughly 2 in 5 generated queries on unseen schemas are still wrong — always validate generated SQL before running it against a real database, especially for destructive statements (this adapter was only trained/evaluated on read (SELECT) queries).
Code capability (HumanEval+) is retained better than the pure-SQL version but still ~4pp below the un-finetuned base model — for general-purpose coding tasks unrelated to SQL, the base model remains the stronger choice. See Results below.

What's on `main` vs `pure-sql`

main (this version) — trained on a 50/50 mix of SQL and general code-instruction data. Better SQL generalization to unseen schemas and better retention of general code ability than the pure-SQL version.
pure-sql branch — the original version trained on 100% SQL data. Higher in-distribution SQL accuracy on the training-like distribution, but noticeably worse code capability retention and worse generalization to unseen database schemas.

# to load the pure-SQL version instead:
PeftModel.from_pretrained(model, adapter_id, revision="pure-sql")

Why mix in code data

The pure-SQL version showed catastrophic forgetting of general code generation ability:

Metric	Base	Pure-SQL SFT	Δ
HumanEval pass@1	52.0%	40.0%	-12.0pp
HumanEval+ (999 edge cases)	46.0%	34.0%	-12.0pp

Training data was rebalanced to 50% b-mc2/sql-create-context + 50% ise-uiuc/Magicoder-OSS-Instruct-75K (interleaved batch-wise via datasets.interleave_datasets), 1 epoch.

Results

Code capability retention (n=50, HumanEval/HumanEval+)

Metric	Base	Pure-SQL SFT	This version (mixed)
HumanEval pass@1	52.0%	40.0% (-12.0pp)	44.0% (-8.0pp)
HumanEval+ (plus)	46.0%	34.0% (-12.0pp)	42.0% (-4.0pp)

SQL generalization on unseen schemas (Spider 1.0, n=1034, real databases + official eval)

Metric	Base	Pure-SQL SFT	This version (mixed)
Official Execution Accuracy	39.9%	50.4%	60.8%
Official Exact Match (structural)	32.1%	37.4%	47.3%

This version beats the pure-SQL version on every axis measured — code capability retention, SQL generalization to unseen schemas, and in-distribution SQL accuracy (see below). Mixing in code data appears to act as a regularizer against overfitting to the narrow single-domain SQL distribution.

In-distribution SQL accuracy (b-mc2/sql-create-context, same distribution as training)

Model	Exact Match
Base	4% (2/50)
Pure-SQL SFT	78% (39/50)
This version (mixed)	70% (14/20, different eval run — see repo `text-to-sql/docs/experiment_log.md` for full methodology)

LoRA configuration

r: 16
lora_alpha: 32
lora_dropout: 0.05
target_modules: [q_proj, k_proj, v_proj, o_proj]
quantization: 4-bit NF4 (QLoRA), bf16 compute

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "deepseek-ai/deepseek-coder-7b-instruct-v1.5"
adapter_id = "albertkingdom/deepseek-coder-7b-text2sql-magicoder-lora"

model = AutoModelForCausalLM.from_pretrained(base_model_id, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(model, adapter_id)  # main = mixed training version

messages = [{
    "role": "user",
    "content": """Given the database schema below, write a SQL query that answers the user's question.
Only output the SQL query. Do not add any explanation.

### Schema
CREATE TABLE users (id INT, name VARCHAR(100), email VARCHAR(100))

### Question
Find all users with gmail addresses"""
}]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training procedure

Trained with SFT (TRL SFTTrainer) on an interleaved SQL + code instruction dataset, 1 epoch, effective batch size 16, bf16, 4-bit QLoRA, on a rented RTX 5090.

Framework versions

PEFT 0.18.0
TRL: 0.26.2
Transformers: 4.57.3
Datasets: 4.4.2

License

This adapter is a derivative of deepseek-ai/deepseek-coder-7b-instruct-v1.5, which is released under the DeepSeek Model License rather than a standard open-source license. Per that license, derivative models must carry forward at least the same use-based restrictions, so this adapter — and any model merged/derived from it — inherits them:

No use for military purposes.
No use that harms minors.
No generation of false information intended to harm others.
No creation of non-consensual personal identifiable information.
No fully automated decision-making that adversely affects an individual's legal rights.
No discrimination based on protected characteristics.
See the full license text (Attachment A) for the complete list.

Commercial use is otherwise permitted, consistent with the base model's license.

Credits & Data Provenance

Base model: deepseek-ai/deepseek-coder-7b-instruct-v1.5 (DeepSeek Model License)
b-mc2/sql-create-context (CC-BY-4.0) — itself derived from WikiSQL and Spider; credit to both original sources per CC-BY-4.0 attribution terms.
ise-uiuc/Magicoder-OSS-Instruct-75K (MIT) — generated via the OSS-Instruct method using gpt-3.5-turbo-1106. Outputs are subject to OpenAI's usage policies in addition to the dataset's own MIT license.

Citations

@misc{vonwerra2022trl,
    title        = {{TRL: Transformer Reinforcement Learning}},
    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
    year         = 2020,
    journal      = {GitHub repository},
    publisher    = {GitHub},
    howpublished = {\url{https://github.com/huggingface/trl}}
}