Upload MODEL_CARD.md

MODEL_CARD.md

@@ -11,7 +11,7 @@ tags:
 - fine-tuned
 base_model: unsloth/Llama-3.2-3B-Instruct
 datasets:
-- [your-username]/mobile-forensics-sql
+- pawlaszc/mobile-forensics-sql
 metrics:
 - accuracy
 model-index:
@@ -22,19 +22,19 @@ model-index:
       name: Text-to-SQL Generation
     dataset:
       type: mobile-forensics
-      name: Mobile Forensics SQL Dataset
+      name: SQLiteDS — Mobile Forensics SQL Dataset (corrected)
     metrics:
     - type: accuracy
-      value: 79.0
-      name: Overall Accuracy
+      value: 91.0
+      name: Overall Accuracy (without app name)
     - type: accuracy
-      value: 94.3
+      value: 95.1
       name: Easy Queries Accuracy
     - type: accuracy
-      value: 80.6
+      value: 87.5
       name: Medium Queries Accuracy
     - type: accuracy
-      value: 61.8
+      value: 88.9
       name: Hard Queries Accuracy
 ---
 
@@ -42,55 +42,121 @@ model-index:
 
 ## Model Description
 
-**ForensicSQL** is a fine-tuned Llama 3.2 3B model specialized for generating SQLite queries for mobile forensics databases. The model converts natural language forensic investigation requests into executable SQL queries across various mobile app databases (WhatsApp, Signal, iOS Health, Android SMS, etc.).
+**ForSQLiteLM** (ForensicSQL-Llama-3.2-3B) is a fine-tuned Llama 3.2-3B model specialized
+for generating SQLite queries from natural language requests against mobile forensic databases.
+The model converts investigative questions into executable SQL queries across a wide range of
+forensic artifact databases — WhatsApp, Signal, iMessage, Android SMS, iOS Health, WeChat,
+Instagram, blockchain wallets, and many more.
 
-This model was developed as part of a master's thesis investigating LLM fine-tuning for forensic database analysis.
+This model was developed as part of a master's thesis and accompanying journal paper
+investigating LLM fine-tuning for forensic database analysis, and is integrated into
+[FQLite](https://github.com/pawlaszczyk/fqlite), an established open-source forensic
+analysis tool.
+
+> **Key result:** 91.0% execution accuracy on a 100-example held-out test set — within
+> 4 percentage points of GPT-4o (95.0%) evaluated under identical conditions
+> (McNemar test: p ≈ 0.39, not significant at α = 0.05), while running fully locally
+> with no internet connectivity required.
 
 ## Model Details
 
-- **Base Model:** Llama 3.2 3B Instruct
-- **Fine-tuning Method:** LoRA (Low-Rank Adaptation)
-- **Training Dataset:** 768 forensic SQL examples across 148 categories
-- **Training Framework:** Hugging Face Transformers + PEFT
-- **Model Size:** 
-  - Full (FP16): ~6 GB
-  - GGUF Q4_K_M: ~2.3 GB
-  - GGUF Q5_K_M: ~2.8 GB
-  - GGUF Q8_0: ~3.8 GB
+| Property | Value |
+|---|---|
+| **Base Model** | meta-llama/Llama-3.2-3B-Instruct |
+| **Fine-tuning Method** | Full fine-tune (bf16) |
+| **Training Dataset** | SQLiteDS — 800 training examples, 191 forensic artifact categories |
+| **Training Framework** | Hugging Face Transformers |
+| **Best Val Loss** | 0.3043 (7 epochs) |
+| **Model Size (bf16)** | ~6 GB |
+| **Hardware Required** | 16 GB unified memory (Apple M-series) or equivalent GPU |
 
 ## Performance
 
-### Overall Results
-- **Overall Accuracy:** 79.0%
-- **Schema Generation Errors:** 0% (completely eliminated)
-- **Executable Queries:** 79%
-
-### Breakdown by Difficulty
-| Difficulty             | Accuracy | Examples |
-|------------------------|----------|----------|
-| Easy (single-table)    | 94.3%    | 33/35    |
-| Medium (simple joins)  | 80.6%    | 25/31    |
-| Hard (complex queries) | 61.8%    | 21/34    |
-
-### Error Analysis
-| Error Type           | Percentage | Description                        |
-|----------------------|------------|------------------------------------|
-| Column Hallucination |        18% | References non-existent columns    |
-| Syntax Errors        |         3% | Invalid SQL syntax                 |
-| Schema Generation    |         0% | Eliminated through proper training |
+### Overall Results (fixed dataset, n=100, best configuration)
+
+| Metric | Value |
+|---|---|
+| **Overall Accuracy** | **91.0%** (91/100) |
+| 95% CI (Wilson) | [83.8%, 95.2%] |
+| Executable Queries | 92/100 |
+| GPT-4o Accuracy | 95.0% (gap: 4 pp, p ≈ 0.39) |
+| Base Model (no fine-tuning) | 35.0% |
+| Improvement over base | +56 pp |
+
+### Accuracy by Query Difficulty
+
+| Difficulty | Accuracy | n | 95% CI | vs. GPT-4o |
+|---|---|---|---|---|
+| Easy (single-table) | **95.1%** | 39/41 | [83.9%, 98.7%] | 0.0 pp |
+| Medium (joins, aggregation) | **87.5%** | 28/32 | [71.9%, 95.0%] | 0.0 pp |
+| Hard (CTEs, window functions) | **88.9%** | 24/27 | [71.9%, 96.1%] | −3.7 pp |
+
+ForSQLiteLM matches GPT-4o exactly on Easy and Medium queries. The remaining gap
+is concentrated on Hard queries (complex CTEs, window functions, multi-table joins).
+
+### Accuracy by Forensic Domain
+
+| Domain | Accuracy | n | 95% CI |
+|---|---|---|---|
+| Messaging & Social | **100.0%** | 28/28 | [87.9%, 100.0%] |
+| Android Artifacts | **94.4%** | 17/18 | [74.2%, 99.0%] |
+| Productivity & Other | **88.9%** | 16/18 | [67.2%, 96.9%] |
+| iOS CoreData | **84.0%** | 21/25 | [65.3%, 93.6%] |
+| Finance & Crypto | **81.8%** | 9/11 | [52.3%, 94.9%] |
+
+### Prompt Configuration Ablation
+
+| Configuration | Overall | Easy | Medium | Hard | iOS |
+|---|---|---|---|---|---|
+| **WITHOUT App Name** ★ | **91.0%** | **95.1%** | 87.5% | **88.9%** | 84.0% |
+| WITH App Name | 88.0% | 92.7% | 87.5% | 81.5% | **88.0%** |
+
+★ Primary configuration — omitting the application name from the prompt yields
+3 pp higher overall accuracy. Interestingly, including the app name helps iOS
+CoreData schemas (+4 pp) but hurts Hard queries (−7.4 pp); the primary
+configuration without app name is recommended for general use.
+
+### Post-Processing Pipeline Contribution
+
+| Component | Queries saved |
+|---|---|
+| Execution feedback (retry) | 7 |
+| Alias normalization | 18 |
+| Column corrections (Levenshtein) | 2 |
+
+### Training Progression
+
+| Configuration | Val Loss | Accuracy | Δ |
+|---|---|---|---|
+| Base model (no fine-tuning) | — | 35.0% | — |
+| Fine-tuned, no augmentation | — | 68.0% | +33 pp |
+| + Data augmentation (3.4×) | — | 74.0% | +6 pp |
+| + Extended training (7 epochs) | 0.3617 | 84.0% | +10 pp |
+| + Post-processing pipeline | 0.3617 | 87.0% | +3 pp |
+| + Execution feedback | 0.3617 | 90.0% | +3 pp |
+| + Corrected training dataset (v5) | **0.3043** | **91.0%** | +1 pp |
 
 ## Intended Use
 
 ### Primary Use Cases
-- Mobile forensics investigations
-- Automated SQL query generation for forensic databases
-- Educational tool for learning forensic database analysis
-- Research in text-to-SQL for specialized domains
+- Mobile forensics investigations: automated SQL query drafting against seized device databases
+- Integration into forensic tools (FQLite, Autopsy, ALEAPP/iLEAPP workflows)
+- Research in domain-specific Text-to-SQL
+- Educational use for learning forensic database analysis
+
+### Important: This Model is a Drafting Assistant
+
+> **ForSQLiteLM is not a replacement for SQL expertise.** It generates candidate queries
+> that require review by a practitioner with sufficient SQL knowledge before any reliance
+> is placed on their results. The 91.0% accuracy means approximately **1 in 11 queries
+> contains an error**. In court-admissible or case-critical work, all outputs must be
+> independently validated.
 
 ### Out-of-Scope Use
-- General-purpose SQL generation (use specialized models)
-- Production systems requiring >95% accuracy
-- Real-time critical forensic decisions without human review
+- Autonomous forensic decision-making without human review
+- Production systems requiring >95% guaranteed accuracy
+- General-purpose SQL generation outside the forensic domain
+- Non-SQLite databases (PostgreSQL, MySQL, etc.)
 
 ## How to Use
 
@@ -100,28 +166,34 @@ This model was developed as part of a master's thesis investigating LLM fine-tun
 from transformers import AutoModelForCausalLM, AutoTokenizer
 import torch
 
-# Load model and tokenizer
 model_name = "pawlaszc/ForensicSQL-Llama-3.2-3B"
 tokenizer = AutoTokenizer.from_pretrained(model_name)
 model = AutoModelForCausalLM.from_pretrained(
     model_name,
-    torch_dtype=torch.float16,
+    torch_dtype=torch.bfloat16,
     device_map="auto"
 )
+model.eval()
 
-# Prepare input
 schema = """
-CREATE TABLE messages (
-    _id INTEGER PRIMARY KEY,
-    address TEXT,
-    body TEXT,
+CREATE TABLE message (
+    ROWID INTEGER PRIMARY KEY,
+    text TEXT,
+    handle_id INTEGER,
     date INTEGER,
-    read INTEGER
+    is_from_me INTEGER,
+    cache_has_attachments INTEGER
+);
+CREATE TABLE handle (
+    ROWID INTEGER PRIMARY KEY,
+    id TEXT,
+    service TEXT
 );
 """
 
-request = "Find all unread messages from yesterday"
+request = "Find all messages received in the last 7 days that contain attachments"
 
+# Note: do NOT use apply_chat_template — use plain-text prompt
 prompt = f"""Generate a valid SQLite query for this forensic database request.
 
 Database Schema:
@@ -132,204 +204,200 @@ Request: {request}
 SQLite Query:
 """
 
-# Generate SQL
 inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=2048)
 inputs = {k: v.to(model.device) for k, v in inputs.items()}
 
 with torch.no_grad():
     outputs = model.generate(
         **inputs,
-        max_new_tokens=200,
-        do_sample=False,
+        max_new_tokens=300,
+        do_sample=False,        # greedy decoding — do not change
     )
 
-# Decode only the generated part
 input_length = inputs['input_ids'].shape[1]
-generated_tokens = outputs[0][input_length:]
-sql = tokenizer.decode(generated_tokens, skip_special_tokens=True)
-
+sql = tokenizer.decode(outputs[0][input_length:], skip_special_tokens=True)
 print(sql.strip())
-# Output: SELECT * FROM messages WHERE read = 0 AND date > ...
-```
-
-### Using GGUF Files (llama.cpp / Ollama)
-
-**With llama.cpp:**
-```bash
-# Download GGUF file
-wget https://huggingface.co/pawlaszc/ForensicSQL-Llama-3.2-3B/resolve/main/forensic-sql-q4_k_m.gguf
-
-# Run inference
-./llama-cli -m forensic-sql-q4_k_m.gguf -p "Generate SQL..."
 ```
 
-**With Ollama:**
-```bash
-# Create Modelfile
-FROM ./forensic-sql-q4_k_m.gguf
-PARAMETER temperature 0
-PARAMETER top_p 0.9
-
-# Import
-ollama create forensic-sql -f Modelfile
-
-# Use
-ollama run forensic-sql "Schema: ...\nRequest: Find messages\nSQL:"
-```
+> **Important:** Use plain-text tokenization (do **not** call `apply_chat_template`).
+> The model was trained and evaluated with a plain-text prompt format.
+> Use `do_sample=False` (greedy decoding) for reproducible results.
 
 ### Python Helper Class
 
 ```python
 class ForensicSQLGenerator:
-    def __init__(self, model_name="pawalaszc/ForensicSQL-Llama-3.2-3B"):
+    def __init__(self, model_name="pawlaszc/ForensicSQL-Llama-3.2-3B"):
         from transformers import AutoModelForCausalLM, AutoTokenizer
         import torch
-        
+
         self.tokenizer = AutoTokenizer.from_pretrained(model_name)
         self.model = AutoModelForCausalLM.from_pretrained(
             model_name,
-            torch_dtype=torch.float16,
+            torch_dtype=torch.bfloat16,
             device_map="auto"
         )
         self.model.eval()
-    
-    def generate_sql(self, schema: str, request: str) -> str:
-        prompt = f"""Generate a valid SQLite query for this forensic database request.
-
-Database Schema:
-{schema}
-
-Request: {request}
 
-SQLite Query:
-"""
+    def generate_sql(self, schema: str, request: str) -> str:
+        prompt = (
+            "Generate a valid SQLite query for this forensic database request.\n\n"
+            f"Database Schema:\n{schema}\n\n"
+            f"Request: {request}\n\n"
+            "SQLite Query:\n"
+        )
         inputs = self.tokenizer(
-            prompt, 
-            return_tensors="pt", 
-            truncation=True, 
-            max_length=2048
+            prompt, return_tensors="pt", truncation=True, max_length=2048
         )
         inputs = {k: v.to(self.model.device) for k, v in inputs.items()}
-        
-        input_length = inputs['input_ids'].shape[1]
-        
+        input_length = inputs["input_ids"].shape[1]
+
         with torch.no_grad():
             outputs = self.model.generate(
-                **inputs,
-                max_new_tokens=200,
-                do_sample=False,
+                **inputs, max_new_tokens=300, do_sample=False
             )
-        
-        generated_tokens = outputs[0][input_length:]
-        sql = self.tokenizer.decode(generated_tokens, skip_special_tokens=True)
-        
+
+        sql = self.tokenizer.decode(
+            outputs[0][input_length:], skip_special_tokens=True
+        )
+        # Return first statement only, normalized
         return sql.strip().split("\n")[0].strip().rstrip(";") + ";"
 
+
 # Usage
 generator = ForensicSQLGenerator()
-sql = generator.generate_sql(schema, request)
+sql = generator.generate_sql(schema, "Find all unread messages from the last 24 hours")
+print(sql)
 ```
 
-## Training Details
+### With Ollama / llama.cpp (GGUF)
 
-### Training Data
-- **Size:** 768 examples (original) → 2,304 examples (with augmentation)
-- **Categories:** 148 forensic database categories
-- **Sources:** WhatsApp, Signal, iMessage, SMS, iOS apps, Android apps
-- **Augmentation:** 3x paraphrase augmentation per example
+```bash
+# With llama.cpp
+./llama-cli -m forensic-sql-q4_k_m.gguf \
+  --temp 0 \
+  -p "Generate a valid SQLite query for this forensic database request.
+
+Database Schema:
+CREATE TABLE sms (_id INTEGER PRIMARY KEY, address TEXT, body TEXT, date INTEGER);
 
-### Training Procedure
-- **Method:** LoRA fine-tuning
-- **LoRA Rank:** 16
-- **LoRA Alpha:** 32
-- **Target Modules:** q_proj, k_proj, v_proj, o_proj
-- **Epochs:** 5
-- **Learning Rate:** 2e-5
-- **Batch Size:** 1 (gradient accumulation: 4)
-- **Max Sequence Length:** 2048 (critical for preventing truncation)
-- **Optimizer:** AdamW
-- **Scheduler:** Cosine with warmup (10%)
-- **Hardware:** Apple M2 (MPS)
-- **Training Time:** ~3.5 hours
+Request: Find all messages sent after midnight
 
-### Key Training Insights
+SQLite Query:"
 
-**Critical Discovery: Sequence Length Matters**
+# With Ollama — create a Modelfile
+cat > Modelfile << 'EOF'
+FROM ./forensic-sql-q4_k_m.gguf
+PARAMETER temperature 0
+PARAMETER num_predict 300
+EOF
 
-Initial training attempts with `max_seq_length=512` resulted in only 50% accuracy because 92% of training examples were truncated. The model learned to generate schema definitions (CREATE TABLE) instead of queries.
+ollama create forensic-sql -f Modelfile
+ollama run forensic-sql
+```
 
-Increasing to `max_seq_length=2048` eliminated truncation and improved accuracy from 50% to 79% (+29pp).
+## Training Details
 
-**Lesson:** Data preprocessing and proper sequence length configuration are critical for fine-tuning success.
+### Dataset — SQLiteDS
+
+- **Total examples:** 1,000 (800 train / 100 val / 100 test), fixed random seed 42
+- **Forensic artifact categories:** 191
+- **Reference query validation:** All 1,000 reference queries validated for execution
+  correctness against in-memory SQLite; 50 queries (5%) corrected before final training
+- **Augmentation:** 3.4× expansion via instruction paraphrasing, WHERE clause reordering,
+  and LIMIT injection — augmented examples confined to training split only
+- **Dataset:** [pawlaszc/mobile-forensics-sql](https://huggingface.co/datasets/pawlaszc/mobile-forensics-sql)
+- **License:** CC BY 4.0
+
+### Hyperparameters
+
+| Parameter | Value |
+|---|---|
+| Training method | Full fine-tune (no LoRA) |
+| Precision | bfloat16 |
+| Epochs | 7 |
+| Learning rate | 2e-5 (peak) |
+| LR scheduler | Cosine with warmup |
+| Batch size | 1 + gradient accumulation 4 |
+| Max sequence length | 2048 |
+| Optimizer | AdamW |
+| Hardware | Apple M-series, 16 GB unified memory |
+| Training time | ~17.6 hours |
+| Best val loss | 0.3043 (epoch 7) |
+
+### Key Training Insight: Sequence Length
+
+Early training runs with `max_seq_length=512` truncated 92% of examples, causing
+the model to learn schema generation (CREATE TABLE) instead of queries — resulting
+in only ~50% accuracy. Setting `max_seq_length=2048` eliminated truncation and
+improved accuracy from 50% to 68% before augmentation, and to 91% after all
+training components were applied.
 
 ## Limitations
 
 ### Known Issues
-1. **Column Hallucination (18%):** Model sometimes references non-existent columns
-2. **Complex Joins:** Performance drops on multi-table queries requiring JOINs (62%)
-3. **Schema Understanding:** Limited understanding of foreign key relationships
 
-### When to Use Human Review
-- Complex multi-table queries
-- Critical forensic investigations
-- Queries involving data deletion or modification
-- When accuracy >95% is required
+1. **iOS CoreData Schemas (84.0%):** The Z-prefix column naming convention
+   (e.g., `ZISFROMME`, `ZTIMESTAMP`) provides no semantic signal from column
+   names alone, making these schemas harder to reason about.
+2. **Hard Queries — 3.7 pp gap to GPT-4o:** Complex CTEs, recursive queries,
+   and window functions are the primary remaining challenge.
+3. **Finance & Crypto (81.8%, n=11):** Small test set; confidence intervals are
+   wide. Interpret with caution.
+4. **~1 in 11 error rate:** Approximately 9% of generated queries will contain
+   errors. Expert review of all outputs is required before use in investigations.
+
+### When Human Review is Especially Important
+- Complex multi-table queries with CTEs or window functions
+- Case-critical or court-admissible investigations
+- Any query that will be used to draw conclusions about a suspect
+- Queries involving rare or unusual forensic artifact schemas
 
 ## Evaluation
 
-### Test Set
-- **Size:** 100 queries (random sample from held-out data)
-- **Seed:** 42 (reproducible)
-- **Evaluation Metric:** Exact match (query results must match expected results)
-
-### Ablation Studies
-
-| Configuration                  | Accuracy | Notes                |
-|--------------------------------|----------|----------------------|
-| Zero-shot baseline             | 45%      | No fine-tuning       |
-| Final training (max_len=2048)  | 79%      | No truncation        |
-
+- **Test set:** 100 examples, held-out, seed=42, non-augmented
+- **Metric:** Execution accuracy — query is correct iff it executes without error
+  AND returns a result set identical to the reference query
+- **Reference validation:** All reference queries validated for execution correctness
+  before evaluation; 5 broken queries in the test set were corrected
+- **Evaluation script:** Available in the dataset repository on Zenodo ([DOI])
 
 ## Citation
 
-If you use this model in your research, please cite:
+If you use this model or the SQLiteDS dataset in your research, please cite:
 
 ```bibtex
-@mastersthesis{forensicsql2025,
-  author = {Dirk Pawlaszczyk},
-  title = {Fine-Tuning Large Language Models for Forensic SQL Query Generation},
-  school = {[Hochschule Mittweida University of Applied Sciences]},
-  year = {2026},
-  type = {Journal}
+@article{pawlaszczyk2026forsqlitelm,
+  author  = {Dirk Pawlaszczyk},
+  title   = {AI-Based Automated SQL Query Generation for SQLite Databases
+             in Mobile Forensics},
+  journal = {Forensic Science International: Digital Investigation},
+  year    = {2026},
+  note    = {FSIDI-D-26-00029}
 }
 ```
 
-## Model Card Authors
-
-Dirk Pawlaszczyk
-
-## Model Card Contact
-
-For questions or issues, please open an issue on the (https://github.com/pawlaszczyk/forensic-sql) or contact pawlaszc@hs-mittweida.de.
-
 ## License
 
-This model is released under the Apache 2.0 License, following the base Llama 3.2 license.
+Apache 2.0 — following the base Llama 3.2 license terms.
 
 ## Acknowledgments
 
-- Base model: Meta's Llama 3.2 3B Instruct
-- Training framework: Hugging Face Transformers, PEFT
-- Dataset creation: Custom forensic database schemas
-- Inspiration: Text-to-SQL research community
+- Base model: Meta's Llama 3.2-3B-Instruct
+- Training framework: Hugging Face Transformers
+- Forensic tool integration: [FQLite](https://github.com/pawlaszczyk/fqlite)
+- Schema sources: iLEAPP, ALEAPP, Autopsy (used under their respective open-source licenses)
 
 ## Additional Resources
 
-- **Dataset:** pawlaszc/mobile-forensics-sql
-- **GitHub:** https://github.com/pawlaszc/forensic-sql
-- **Paper:** [Link when published]
-- **Demo:** [HuggingFace Space if you create one]
+- **Dataset (Zenodo):** [SQLiteDS — DOI to be added on publication]
+- **Dataset (HuggingFace):** [pawlaszc/mobile-forensics-sql](https://huggingface.co/datasets/pawlaszc/mobile-forensics-sql)
+- **FQLite integration:** [github.com/pawlaszczyk/fqlite](https://github.com/pawlaszczyk/fqlite)
+- **Paper:** FSIDI-D-26-00029 (under review)
 
 ---
 
-**Disclaimer:** This model is intended for research and educational purposes. Always validate generated SQL queries before execution in production forensic investigations. The model may produce incorrect queries that could lead to data loss or incorrect conclusions if used without proper review.
+**Disclaimer:** ForSQLiteLM is intended for research and forensic practitioner use.
+All generated SQL queries must be reviewed by a qualified practitioner before
+execution in live forensic investigations. The authors accept no liability for
+incorrect conclusions drawn from unvalidated model outputs.