cycloevan/vuln_detector

Model Card for merged-vuln-detector

Model Details

• Base Model: llama-3.2-1B-Instruct
• Fine-tuned Model: merged-vuln-detector
• Model Type: Causal Language Model fine-tuned for vulnerability detection in code.

Model Description

This model is a fine-tuned version of llama-3.2-1B-Instruct on a dataset of code snippets and their corresponding vulnerability analyses. The model is intended to be used as a security expert that can analyze code and identify potential vulnerabilities.

Training Data

The model was fine-tuned on the CyberNative/Code_Vulnerability_Security_DPO dataset, which can be found on Hugging Face at https://huggingface.co/datasets/CyberNative/Code_Vulnerability_Security_DPO.

The data is formatted as follows, where the model is prompted to analyze the security of a given code snippet:

Analyze the security vulnerabilities in the following code.

[CODE SNIPPET]

Analysis:
[VULNERABILITY DESCRIPTION]

Training Procedure

The model was fine-tuned using QLoRA on a single GPU. The training script uses the trl library's SFTTrainer.

Hyperparameters:

• Quantization: 4-bit (nf4)
• LoRA r: 16
• LoRA alpha: 32
• LoRA dropout: 0.1
• Target Modules: q_proj, k_proj, v_proj, o_proj
• Batch Size: 1 (with gradient accumulation steps of 8)
• Optimizer: paged_adamw_8bit
• Precision: fp16
• Max Steps: 240
• Learning Rate: 2e-4
• Max Sequence Length: 1024

Evaluation Results

The model was evaluated on the doss1232/vulnerable-code dataset against the base model. The results are as follows:

Model	ROUGE-L F1	BLEU
llama-3.2-1B-Instruct	0.0933	0.0061
merged-vuln-detector	0.1335	0.0219

How to use

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "merged-vuln-detector"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

code = """
#include <cstring>

void copyString(char* dest, const char* src) {
    while (*src != '\0') {
        *dest = *src;
        dest++;
        src++;
    }
}

int main() {
    char source[10] = "Hello!";
    char destination[5];
    copyString(destination, source);
    return 0;
}
"""

prompt = f"Analyze the security vulnerabilities in the following code.\n\n{code}\n\nAnalysis:\n"

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=512)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Example Output

Input Code:

#include <cstring>

void copyString(char* dest, const char* src) {
    while (*src != '\0') {
        *dest = *src;
        dest++;
        src++;
    }
}

int main() {
    char source[10] = "Hello!";
    char destination[5];
    copyString(destination, source);
    return 0;
}

Model Output:

The code has a buffer overflow vulnerability due to the lack of bounds checking on the destination buffer size.

GGUF / llama.cpp (On-device Inference)

Quantized GGUF builds of merged-vuln-detector are provided for on-device inference with llama.cpp. They were converted from merged-vuln-detector/model.safetensors with convert_hf_to_gguf.py and quantized with llama-quantize.

File	Quantization	Size	Notes
vuln_detector-Q4_K_M.gguf	Q4_K_M	0.81 GB	Recommended for on-device use
vuln_detector-Q8_0.gguf	Q8_0	1.32 GB	Near-lossless

Quantization quality

Measured on 100 samples from doss1232/vulnerable-code (shuffled with seed 42, 500-row held-out pool, first 100 evaluated; fine-tuning prompt format; greedy decoding, max_new_tokens=128; identical harness for every variant):

Variant	ROUGE-L F1	BLEU	Exact output match vs original
transformers (original, fp16)	0.1962	0.0645	—
GGUF F16	0.1963	0.0645	98%
GGUF Q8_0	0.1987	0.0675	89%
GGUF Q4_K_M	0.2325	0.0916	36%

Quantization does not degrade benchmark quality — Q8_0 and Q4_K_M match or slightly exceed the original model's reference metrics (differences within noise for a 1B model). Q4_K_M's token-level outputs diverge from the fp16 model on many samples while remaining equivalent in quality; choose Q8_0 when close output fidelity to the fp16 model matters.

Note: these absolute numbers are not directly comparable to the "Evaluation Results" table above, which used a different sampling/decoding protocol.

Measured performance (Apple M1, Metal)

llama-bench (pp512 = prompt processing, tg128 = generation):

Variant	Prompt processing (pp512)	Generation (tg128)
GGUF F16	941 t/s	19.2 t/s
GGUF Q8_0	618 t/s	29.4 t/s
GGUF Q4_K_M	504 t/s	42.3 t/s

Generation is memory-bandwidth-bound, so smaller weights decode faster: Q4_K_M generates 2.2× faster than F16. Prompt processing is compute-bound and favors F16 (dequantization overhead), but at 500+ t/s a typical prompt is prefilled in well under a second — end-to-end latency is dominated by generation, so Q4_K_M is the fastest overall.

End-to-end wall-clock on the identical 100-sample eval batch (greedy, n_predict=128, same prompts):

Runtime	Batch time	Speedup
transformers fp16 (PyTorch, MPS)	138 s	1.0×
llama.cpp GGUF F16	96 s	1.4×
llama.cpp GGUF Q8_0	57 s	2.4×
llama.cpp GGUF Q4_K_M	53 s	2.6×

Weight memory footprint: 2.48 GB (F16) → 1.32 GB (Q8_0) → 0.81 GB (Q4_K_M).

Run with llama.cpp

llama-cli -hf cycloevan/vuln_detector:Q4_K_M \
  -p "Analyze the security vulnerabilities in the following code.\n\n<CODE>\n\nAnalysis:\n" \
  -n 256 --temp 0

Run with llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained("cycloevan/vuln_detector", filename="vuln_detector-Q4_K_M.gguf")

code = "def login(u, p): cursor.execute(f\"SELECT * FROM users WHERE name='{u}' AND pw='{p}'\")"
prompt = f"Analyze the security vulnerabilities in the following code.\n\n{code}\n\nAnalysis:\n"
out = llm(prompt, max_tokens=256, temperature=0)
print(out["choices"][0]["text"])

Model Card Authors

[Seokhee Chang]

Model Card Contact

[cycloevan97@gmail.com]