cyber-cve2cwe-0.6b

Overview

cyber-cve2cwe-0.6b is a lightweight LoRA adapter that transforms the Qwen3-0.6B causal language model into a specialist for mapping software vulnerability descriptions (CVE entries) to a single Common Weakness Enumeration (CWE) identifier. The model follows an instruction-following paradigm: given a CVE description, it returns the most appropriate CWE code in the format CWE-<number> and nothing else.

Prompt format

The model expects a chat-style conversation with a fixed system message and a user message containing the CVE description. The system prompt is:

You are a security expert that classifies software vulnerabilities using the Common Weakness Enumeration (CWE). Given a CVE description, respond with the single most appropriate CWE identifier in the form CWE-<number> and nothing else.

The user message is the raw CVE description. The assistant's reply should be the CWE identifier. The "thinking" mode is disabled (enable_thinking: false).

Usage

Below is a minimal Python example that loads the base model, applies the LoRA adapter, builds the required chat messages, and generates a short answer.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load the base Qwen3-0.6B model and tokenizer
base_name = "unsloth/Qwen3-0.6B"
model = AutoModelForCausalLM.from_pretrained(
    base_name,
    device_map="auto",
    dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(base_name)

# Load the LoRA adapter
adapter_name = "luca-software-developer/cyber-cve2cwe-0.6b"
model = PeftModel.from_pretrained(model, adapter_name)

# Build chat messages
system_prompt = ("You are a security expert that classifies software vulnerabilities "
                 "using the Common Weakness Enumeration (CWE). Given a CVE description, "
                 "respond with the single most appropriate CWE identifier in the form "
                 "CWE-<number> and nothing else.")
cve_description = "Buffer overflow in the XYZ library allows remote attackers to execute arbitrary code via a crafted network packet."

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": cve_description},
]

# Convert messages to the model's chat format (add_generation_prompt=True)
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

# Generate the answer (greedy decoding)
output_ids = model.generate(
    **inputs,
    max_new_tokens=8,
    do_sample=False,
)

# Decode only the newly generated tokens (the CWE identifier)
new_tokens = output_ids[0][inputs["input_ids"].shape[1]:]
answer = tokenizer.decode(new_tokens, skip_special_tokens=True).strip()
print(answer)   # Expected output: e.g., CWE-120

Training data

The primary dataset is xamxte/cve-to-cwe, which contains 205 CWE classes derived from the National Vulnerability Database (NVD) with AI-assisted relabeling. A frozen test split is reserved for evaluation. To improve coverage of under-represented CWE categories, the training split was extended with 153 real NVD records from luca-software-developer/cyber-cve2cwe-extension. After this extension, class oversampling ensured that every CWE class had at least 500 training examples, achieved by sampling with replacement.

Training procedure

Fine-tuning was performed with Parameter Efficient Fine-Tuning (PEFT) using LoRA (rank 32, alpha 32, dropout 0) applied to the attention and MLP projection layers. The Unsloth library handled training in bfloat16 precision. Key hyper-parameters:

• Task: completion only (loss computed on the assistant answer, prompt tokens masked)
• Epochs: 1
• Effective batch size: 32 (per-device batch size 8, gradient accumulation 4)
• Learning rate: 2e-4 with a cosine decay schedule
• Optimizer: AdamW 8bit
• Maximum sequence length: 512 tokens
• Parameters tuned: ~20 M LoRA parameters out of ~616 M total model parameters

Evaluation

Evaluation used greedy decoding on the frozen test split. Metrics reported are accuracy, macro-averaged F1, and weighted F1.

split	accuracy	macro F1	weighted F1
full test (27,780 ex.)	0.858	0.593	0.862
leakage-free subset (26,213 ex.)	0.861	0.591	0.866

For comparison, a fully fine-tuned RoBERTa-base model (~125 M parameters) trained on the same data achieved accuracy 0.874 and macro F1 0.607.

Intended uses

The model is intended to assist security analysts during triage and research by providing a quick, single-label suggestion for the CWE category of a given CVE description. It can also be used in academic studies of CVE-to-CWE mapping. Outputs are advisory and should be validated against authoritative sources such as the CVE Program and the NVD before any operational decision is made.

Limitations and biases

• The model performs well on frequent CWE categories but shows reduced recall on rare classes, which lowers macro-averaged F1 relative to accuracy.
• Occasionally the model may generate a CWE identifier that does not belong to the 205-class label set.
• Classification is based solely on the textual description.

License

The LoRA adapter is released under the Apache-2.0 license, matching the licensing terms of the Qwen3-0.6B base model. Users must also respect the licenses of the underlying datasets (xamxte/cve-to-cwe and luca-software-developer/cyber-cve2cwe-extension).

References

• Base model: Qwen/Qwen3-0.6B, loaded through unsloth/Qwen3-0.6B.
• Base dataset: xamxte/cve-to-cwe.
• Extension dataset: luca-software-developer/cyber-cve2cwe-extension.
• Common Weakness Enumeration (CWE).
• National Vulnerability Database (NVD).

Citation

If you use this model, please cite it using the following reference.

@misc{dellorusso2026cve2cwe06b,
  title        = {cyber-cve2cwe-0.6b: a specialized small language model for CVE-to-CWE classification},
  author       = {Dello Russo, Luca},
  year         = {2026},
  howpublished = {\url{https://hf.co/luca-software-developer/cyber-cve2cwe-0.6b}}
}