| ---
|
| license: cc-by-nc-sa-4.0
|
| language:
|
| - en
|
| base_model: meta-llama/Llama-3.1-8B-Instruct
|
| tags:
|
| - code
|
| - security
|
| - python
|
| - lora
|
| - qlora
|
| - fine-tuned
|
| - cybersecurity
|
| - secure-coding
|
| - vulnerability
|
| - cwe
|
| - peft
|
| - transformers
|
| task_categories:
|
| - text-generation
|
| task_ids:
|
| - language-modeling
|
| ---
|
|
|
| # SecuCoder
|
|
|
| SecuCoder is a fine-tuned version of [Llama 3.1 8B Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) trained to generate secure Python code and remediate security vulnerabilities. It is part of a research pipeline that combines supervised fine-tuning (SFT), structured prompting, and retrieval-augmented generation (RAG) to reduce the number of vulnerabilities in automatically generated Python code.
|
|
|
| ---
|
|
|
| ## Model Description
|
|
|
| | Field | Details |
|
| |---|---|
|
| | **Base model** | `meta-llama/Llama-3.1-8B-Instruct` |
|
| | **Fine-tuning method** | QLoRA (NF4 4-bit) + LoRA adapters |
|
| | **Training dataset** | [SecuCoder Messages Corpus](https://huggingface.co/datasets/ivitopow/secucoder) |
|
| | **Training examples** | 5,708 (train) + 317 (validation) |
|
| | **Epochs** | 2 |
|
| | **Format** | Merged safetensors (bfloat16) |
|
| | **Language** | English |
|
| | **Domain** | Python secure coding |
|
|
|
| ---
|
|
|
| ## Intended Use
|
|
|
| SecuCoder is designed for:
|
|
|
| - **Vulnerability remediation** — given a Python snippet with a security flaw, produce a corrected version.
|
| - **Secure code generation** — generate Python code from a natural language specification, avoiding common weaknesses.
|
| - **Vulnerability classification** — identify whether a Python snippet is secure or vulnerable.
|
|
|
| The model has been evaluated against the untuned Llama 3.1 8B Instruct baseline using static analysis tools (Bandit + Semgrep) and shows meaningful improvement in security metrics.
|
|
|
| > This model is intended for research and educational purposes. It should not be used as the sole security review mechanism in production systems.
|
|
|
| ---
|
|
|
| ## Usage
|
|
|
| ```python
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
| import torch
|
|
|
| model_id = "ivitopow/secucoder"
|
|
|
| tokenizer = AutoTokenizer.from_pretrained(model_id)
|
| model = AutoModelForCausalLM.from_pretrained(
|
| model_id,
|
| torch_dtype=torch.bfloat16,
|
| device_map="auto",
|
| )
|
|
|
| messages = [
|
| {
|
| "role": "system",
|
| "content": "You are a secure Python assistant. Help identify, explain, and fix security issues in Python code. Prefer safe, practical, and production-ready solutions."
|
| },
|
| {
|
| "role": "user",
|
| "content": "Fix the security vulnerability in this Python code.\n\n```python\nname = request.args.get('name')\nresp = make_response(\"Your name is \" + name)\n```\n\nCWE: CWE-079"
|
| }
|
| ]
|
|
|
| input_ids = tokenizer.apply_chat_template(
|
| messages,
|
| tokenize=True,
|
| add_generation_prompt=True,
|
| return_tensors="pt"
|
| ).to(model.device)
|
|
|
| output = model.generate(
|
| input_ids,
|
| max_new_tokens=512,
|
| temperature=0.1,
|
| top_p=0.9,
|
| do_sample=True,
|
| )
|
|
|
| response = tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True)
|
| print(response)
|
| ```
|
|
|
| ### Usage with Ollama
|
|
|
| A quantized GGUF version (Q4_K_M, ~4.6 GB) is available at [`ivitopow/secucoder-GGUF`](https://huggingface.co/ivitopow/secucoder-GGUF):
|
|
|
| ```bash
|
| ollama create secucoder -f Modelfile
|
| ollama run secucoder
|
| ```
|
|
|
| ---
|
|
|
| ## Training Details
|
|
|
| ### Method
|
|
|
| The model was trained using **QLoRA** (Quantized Low-Rank Adaptation): the base model is loaded in 4-bit NF4 precision via BitsAndBytes, and low-rank adapters are attached to all projection layers. After training, the adapters are merged back into the base model and saved as standard safetensors.
|
|
|
| ### LoRA Configuration
|
|
|
| | Parameter | Value |
|
| |---|---|
|
| | Rank (`r`) | 16 |
|
| | Alpha | 32 |
|
| | Dropout | 0.05 |
|
| | Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
|
|
|
| ### Training Hyperparameters
|
|
|
| | Parameter | Value |
|
| |---|---|
|
| | Epochs | 2 |
|
| | Learning rate | 2e-4 |
|
| | LR scheduler | Cosine with 3% warmup |
|
| | Optimizer | `paged_adamw_8bit` |
|
| | Gradient checkpointing | Enabled |
|
| | Precision | bfloat16 compute, NF4 storage |
|
| | Sequence length | 2048 tokens |
|
|
|
| ### Training Data
|
|
|
| Trained on the [SecuCoder Messages Corpus](https://huggingface.co/datasets/ivitopow/secucoder), a dataset of 6,342 Python security examples in chat format covering:
|
|
|
| - **Vulnerability fix** (`fix`) — 4,037 examples across 20+ CWE categories
|
| - **Security conversations** (`conversation`) — 2,210 multi-turn examples
|
| - **Vulnerability classification** (`classify`) — 52 examples
|
| - **Secure code generation** (`prompt_to_code`) — 43 examples
|
|
|
| ---
|
|
|
| ## Evaluation
|
|
|
| SecuCoder was evaluated as part of a 5-variant ablation study. Each variant adds one technique over the previous one:
|
|
|
| | Variant | Technique | Overall Score |
|
| |---|---|---|
|
| | `llama31_8b` | Baseline (no fine-tuning) | 60.34 |
|
| | `secucoder_v1` | + SFT (LoRA, FP16) | 60.43 |
|
| | `secucoder_v1-q4` | + Q4_K_M quantization | 61.46 |
|
| | `secucoder_v1-q4_prompting` | + Structured security prompt | 64.46 |
|
| | `secucoder_v1-q4_prompting_rag` | + RAG (OWASP, CWE, Python docs) | **77.11** |
|
|
|
| Overall score = mean `sample_score` over non-truncated samples (higher is better, max 100). The full SecuCoder system (`secucoder_v1-q4_prompting_rag`) achieves a **+27.8% improvement** over the untuned baseline.
|
|
|
| ### Evaluation Methodology
|
|
|
| Generated code was scanned with **Bandit** and **Semgrep** using weighted severity scores:
|
|
|
| ```
|
| penalty = Σ(bandit_high × 2.0 + bandit_medium × 1.25 + bandit_low × 0.75)
|
| + Σ(semgrep_error × 5.0 + semgrep_warning × 3.0 + semgrep_info × 1.0)
|
|
|
| sample_score = 100 / (1 + 8 × penalty_per_loc)
|
| ```
|
|
|
| Samples with invalid syntax score 0. Truncated samples are excluded from the overall score.
|
|
|
| ---
|
|
|
| ## Related Resources
|
|
|
| | Resource | Link |
|
| |---|---|
|
| | Training dataset | [ivitopow/secucoder](https://huggingface.co/datasets/ivitopow/secucoder) |
|
| | GGUF / Ollama version | [ivitopow/secucoder-GGUF](https://huggingface.co/ivitopow/secucoder-GGUF) |
|
| | Base model | [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) |
|
|
|
| ---
|
|
|
| ## Limitations
|
|
|
| - The model only covers **Python**. It has not been evaluated on other languages.
|
| - Static analysis tools (Bandit, Semgrep) do not detect all vulnerability types. Logic flaws or runtime-dependent issues may not be caught.
|
| - The model was fine-tuned on a specific set of CWE categories. It may underperform on vulnerability types not well represented in the training data.
|
| - As with all generative models, outputs should be reviewed by a developer before use in production.
|
|
|
| ---
|
|
|
| ## License
|
|
|
| This model is released under the [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/) license.
|
|
|
| It is built on top of Llama 3.1, which is subject to [Meta's Llama 3 Community License](https://llama.meta.com/llama3/license/). Please review both licenses before use.
|
|
|
| ---
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @misc{secucoder2025,
|
| title = {SecuCoder: Fine-tuning Llama 3.1 8B for Secure Python Code Generation},
|
| author = {SecuCoder Project},
|
| year = {2025},
|
| url = {https://huggingface.co/ivitopow/secucoder},
|
| note = {CC-BY-NC-SA-4.0}
|
| }
|
| ``` |
