Upload model weights with Git LFS

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,189 @@
+---
+base_model: Qwen/Qwen2.5-Coder-0.5B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+  - "base_model:adapter:Qwen/Qwen2.5-Coder-0.5B-Instruct"
+  - lora
+  - sft
+  - text-to-sql
+---
+
+# **SQLQwen - Qwen2.5 Text-to-SQL Fine-Tuning with LoRA**
+
+LoRA fine-tuning of **Qwen2.5-Coder-0.5B-Instruct** for **Text-to-SQL generation** using the **Gretel synthetic_text_to_sql** dataset.
+
+Text-to-SQL systems translate natural-language questions into executable SQL using a provided database schema or SQL context. SQLQwen is designed to specialize a lightweight code-focused language model for this task while keeping the fine-tuning process parameter-efficient and practical on consumer hardware.
+
+Given:
+
+1. a database schema or SQL context
+2. a natural-language request
+
+the model generates the relevant SQL query
+
+### Example
+
+**Database context:**
+
+```sql
+CREATE TABLE customers (
+    id INTEGER PRIMARY KEY,
+    name TEXT,
+    country TEXT,
+    revenue DECIMAL(12, 2)
+);
+```
+
+**Natural-language request:**
+
+```text
+Find the five customers with the highest revenue.
+```
+
+**Expected output:**
+
+```sql
+SELECT id, name, country, revenue
+FROM customers
+ORDER BY revenue DESC
+LIMIT 5;
+```
+
+## Why LoRA
+
+LoRA (**Low-Rank Adaptation**) provides a parameter-efficient alternative to full fine-tuning.
+
+Instead of updating all parameters in the pretrained model, LoRA freezes the original model weights and introduces small trainable low-rank matrices into selected Transformer layers.
+
+for this project, LoRA adapters are applied to the attention projection layers:
+
+```text
+q_proj
+k_proj
+v_proj
+o_proj
+```
+
+this reduces the number of trainable parameters, GPU memory requirements, and adapter storage size while preserving the capabilities of the original pretrained model.
+
+Only **2,162,688 of 496,195,456 parameters**, or approximately **0.4359%**, were trainable during fine-tuning.
+
+## Model
+
+* **Base model:** [`Qwen/Qwen2.5-Coder-0.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct)
+* **Fine-tuning:** PEFT LoRA
+* **Training:** TRL `SFTTrainer`
+* **LoRA rank:** `16`
+* **LoRA alpha:** `32`
+* **LoRA dropout:** `0.05`
+* **Target modules:** `q_proj`, `k_proj`, `v_proj`, `o_proj`
+* **Training objective:** Completion-only supervised fine-tuning
+
+Qwen2.5-Coder was selected because SQL generation is fundamentally a structured code-generation task rather than conventional natural-language classification.
+
+## Dataset
+
+### Gretel Synthetic Text-to-SQL
+
+[`gretelai/synthetic_text_to_sql`](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql)
+
+Gretel Synthetic Text-to-SQL dataset provides natural-language SQL requests, database context, target SQL queries, and supporting metadata.
+
+3 fields are used directly during training:
+
+| Dataset field | Purpose                        |
+| ------------- | ------------------------------ |
+| `sql_context` | Database schema or SQL context |
+| `sql_prompt`  | Natural-language request       |
+| `sql`         | Ground-truth SQL completion    |
+
+Training pipeline converts each example into an instruction-style prompt and trains the model specifically on the assistant SQL completion.
+
+## Training Configuration
+
+| Setting                 |                              Value |
+| ----------------------- | ---------------------------------: |
+| Base model              | `Qwen/Qwen2.5-Coder-0.5B-Instruct` |
+| LoRA rank               |                               `16` |
+| LoRA alpha              |                               `32` |
+| LoRA dropout            |                             `0.05` |
+| Epochs                  |                              **2** |
+| Training examples       |                         **20,000** |
+| Validation examples     |                          **1,000** |
+| Train batch size        |                              **4** |
+| Gradient accumulation   |                              **4** |
+| Effective batch size    |                   **16 sequences** |
+| Learning rate           |                           **2e-4** |
+| Warmup ratio            |                           **0.03** |
+| Weight decay            |                           **0.01** |
+| LR scheduler            |                         **Cosine** |
+| Maximum sequence length |                           **2048** |
+| Gradient checkpointing  |                        **Enabled** |
+| Training objective      |            **Completion-only SFT** |
+
+Training automatically uses **BF16** where supported, falling back to FP16 on CUDA or FP32 when CUDA is unavailable.
+
+## Training Run
+
+| Metric                          |      Result |
+| ------------------------------- | ----------: |
+| Training examples               |  **20,000** |
+| Validation examples             |   **1,000** |
+| Epochs                          |       **2** |
+| Optimizer steps                 |   **2,500** |
+| Final training loss             |  **0.2381** |
+| Best evaluation loss            |  **0.2198** |
+| Final evaluation loss           |  **0.2200** |
+| Final evaluation token accuracy |  **93.54%** |
+| Best checkpoint                 |   **2,400** |
+| Training runtime                | **~1h 35m** |
+
+Training was completed locally on an **NVIDIA GeForce RTX 3050**.
+
+Validation loss decreased from **0.2839** at the first logged evaluation to a best value of **0.2198** at checkpoint 2,400. The final checkpoint produced an evaluation loss of **0.2200**, indicating that training had largely converged by the end of the second epoch.
+
+## Held-Out Generation Evaluation
+
+A held-out generation benchmark of **50 examples** was used to evaluate SQL generation code quality.
+
+| Metric                   |      Result |
+| ------------------------ | ----------: |
+| Evaluation examples      |      **50** |
+| Exact-match accuracy     |   **28.0%** |
+| SQL syntax validity      |  **100.0%** |
+| Exact matches            | **14 / 50** |
+| Syntax-valid generations | **50 / 50** |
+
+All **50 generated SQL queries were successfully parsed by SQLGlot**, resulting in a **100% syntax-validity rate**.
+
+**28% exact-match score** uses strict string-level comparison. Semantically or execution-equivalent SQL queries may differ from the reference query while still producing the correct result, so exact match should not be interpreted as the model's full semantic accuracy.
+
+
+## Model Limitations:
+
+* model may hallucinate tables or columns when the supplied database context is incomplete.
+* **0.5B parameter** base model prioritizes lightweight training and inference over maximum reasoning capacity.
+* Training uses synthetic Text-to-SQL examples, which may not represent every real-world database schema or production SQL workload.
+* Exact-match evaluation does not account for all semantically equivalent SQL formulations.
+* SQL execution accuracy against live databases was not measured in the reported benchmark.
+
+## Results:
+
+completed fine-tuning run shows:
+
+* **parameter-efficient adaptation**, with approximately **0.4359%** of model parameters trainable
+* **stable convergence**, with validation loss reaching approximately **0.22**
+* **100% syntax-valid SQL generation** across the 50-example held-out benchmark
+* successful local fine-tuning of a code-focused language model using an **NVIDIA GeForce RTX 3050**
+
+## References:
+- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. *LoRA: Low-Rank Adaptation of Large Language Models*. arXiv:2106.09685, 2021.](https://arxiv.org/abs/2106.09685)
+
+- Hugging Face. *PEFT LoRA Documentation*. Parameter-Efficient Fine-Tuning documentation. (https://huggingface.co/docs/transformers/en/peft) (https://huggingface.co/docs/peft/en/package_reference/lora)
+
+- Qwen Team. Qwen2.5-Coder-0.5B-Instruct Model Card
+
+- Hui, B. et al. Qwen2.5-Coder Technical Report. arXiv:2409.12186, 2024
+
+- Gretel.ai. synthetic_text_to_sql Dataset Card. Hugging Face Datasets

adapter_config.json

@@ -0,0 +1,47 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-Coder-0.5B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "k_proj",
+    "q_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82501f27a261361484b8c67bcb6d87b8c861f4076c83def5570b46c9dff746b7
+size 8676008

chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d48beb5dc8838419f34acc78392a7acab120ac7d4425e4eb021c0b5604ee3e0f
+size 12179335

tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78391a5f3c7906f77f20a11da12769dbf82969852f48b5b7e857521e8cad37ae
+size 5713