Rewrite model card: usage, intended use, data, protocol, artifacts

README.md

@@ -2,42 +2,185 @@
 library_name: transformers
 license: cc-by-4.0
 base_model: roberta-base
+pipeline_tag: text-classification
+language:
+- en
+datasets:
+- CIRCL/vulnerability-attack-techniques
 tags:
+- security
+- vulnerability
+- cve
+- mitre-attack
+- cti
+- multi-label-classification
 - generated_from_trainer
 model-index:
 - name: vulnerability-attack-technique-classification-roberta-base
-  results: []
+  results:
+  - task:
+      type: text-classification
+      name: Multi-label MITRE ATT&CK technique classification
+    dataset:
+      name: CIRCL/vulnerability-attack-techniques
+      type: CIRCL/vulnerability-attack-techniques
+      split: test
+    metrics:
+    - type: recall
+      name: Recall@5
+      value: 0.6440
+    - type: recall
+      name: Recall@3
+      value: 0.5181
+    - type: f1
+      name: F1 micro
+      value: 0.3899
+    - type: f1
+      name: F1 macro
+      value: 0.1910
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 # vulnerability-attack-technique-classification-roberta-base
 
-This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.6410
-- F1 Micro: 0.3899
-- F1 Macro: 0.1910
-- Precision Micro: 0.2740
-- Recall Micro: 0.6756
-- Recall At 3: 0.5181
-- Recall At 5: 0.6440
+Suggests [MITRE ATT&CK](https://attack.mitre.org/) (Enterprise) techniques
+from a free-text vulnerability description. This is a multi-label classifier
+([roberta-base](https://huggingface.co/roberta-base) with a sigmoid head, one
+output per technique) fine-tuned on
+[CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques),
+a gold dataset of 1,207 CVEs whose labels come from expert
+[MITRE CTID](https://ctid.mitre.org/) mappings. Given a CVE description it
+scores 53 parent techniques (e.g. T1190 *Exploit Public-Facing Application*,
+T1505 *Server Software Component*), for use as a ranked list of candidate
+techniques for analyst review.
+
+The model is trained with [VulnTrain](https://github.com/vulnerability-lookup/VulnTrain)
+and runs in production on the public [Vulnerability-Lookup](https://vulnerability.circl.lu)
+instance operated by CIRCL, served locally by
+[ML-Gateway](https://github.com/vulnerability-lookup/ML-Gateway): every
+vulnerability page has an ATT&CK tab with the model's suggestions (example:
+[CVE-2021-44077](https://vulnerability.circl.lu/vuln/CVE-2021-44077#attack)).
+
+The methodology, evaluation protocol, and the negative result on
+LLM-assisted label expansion are described in the paper
+[*Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and
+the Limits of LLM-Assisted Label Expansion*](https://github.com/vulnerability-lookup/cve-attack-mapping-paper).
+
+DOI: [10.57967/hf/9623](https://doi.org/10.57967/hf/9623)
+
+## How to use
+
+With VulnTrain, against a live CVE:
+
+```bash
+vulntrain-infer-attack-classification --cve CVE-2021-44077 \
+    --model CIRCL/vulnerability-attack-technique-classification-roberta-base
+```
 
-## Model description
+With plain Transformers:
 
-More information needed
+```python
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+model_id = "CIRCL/vulnerability-attack-technique-classification-roberta-base"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSequenceClassification.from_pretrained(model_id)
+model.eval()
+
+description = (
+    "Zoho ManageEngine ServiceDesk Plus before 11306, ServiceDesk Plus MSP "
+    "before 10530, and SupportCenter Plus before 11014 are vulnerable to "
+    "unauthenticated remote code execution."
+)
+inputs = tokenizer(description, truncation=True, max_length=512, return_tensors="pt")
+with torch.no_grad():
+    probs = torch.sigmoid(model(**inputs).logits)[0]
+
+for i in probs.argsort(descending=True)[:5]:
+    print(f"{model.config.id2label[int(i)]}  {probs[i]:.4f}")
+```
+
+Technique IDs map to names via the
+[ATT&CK Enterprise matrix](https://attack.mitre.org/techniques/enterprise/);
+`sigmoid >= 0.5` is the prediction threshold used in evaluation, but the
+model is most useful as a ranker (see the recall@k numbers below).
 
 ## Intended uses & limitations
 
-More information needed
+**Intended**: triage assistance — given a vulnerability description, surface
+a short ranked list of candidate ATT&CK techniques for a human analyst to
+confirm or reject. This is how it is deployed in Vulnerability-Lookup, where
+the UI explicitly flags the output as unverified AI-generated suggestions.
+
+**Limitations**:
+
+- The label space is the 53 parent techniques with at least 5 training
+  examples; sub-techniques are collapsed to their parent, and techniques
+  outside this vocabulary can never be suggested.
+- The gold set skews toward exploited-in-the-wild CVEs (CTID's corpus and
+  the KEV catalog), so coverage is best for the techniques common there.
+- Trained on 972 CVEs — deliberately label-quality-bound rather than
+  data-bound (see the paper's gold-size scaling curve: every metric still
+  improves monotonically with more curated rows).
+- English descriptions only; input is truncated at 512 tokens.
+- Scores are not calibrated probabilities.
+- Suggestions are **not** verified mappings; treat them as guidance, never
+  as authoritative CTI.
 
 ## Training and evaluation data
 
-More information needed
+Labels come from the two public expert sources of CVE→ATT&CK mappings, both
+following the CTID [*Mapping ATT&CK to CVE for
+Impact*](https://ctid.mitre.org/projects/mapping-attck-to-cve-for-impact)
+methodology: the CTID `attack_to_cve` mappings and the CTID Mappings
+Explorer KEV mappings (ATT&CK 16.1). Technique IDs revoked since publication
+are remapped to their successors via the ATT&CK STIX `revoked-by`
+relationships. Descriptions are joined from
+[CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores)
+(Vulnerability-Lookup). Full details in the
+[dataset card](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques)
+and the [VulnTrain documentation](https://github.com/vulnerability-lookup/VulnTrain/blob/main/docs/attack-techniques-dataset.md).
+
+Splits: 972 train / 106 validation / 118 test examples. **Checkpoint
+selection uses the validation split; the test split is touched once, for the
+numbers reported here.** The paper documents why this matters: selecting the
+best checkpoint on the test split inflates and destabilizes small-test-split
+metrics enough to produce spurious conclusions.
+
+Notably, this model was **not** trained on LLM-generated labels: the paper's
+controlled experiments show that folding in LLM-labeled CVEs at ≈0.39
+agreement with the experts yields no reliable ranking improvement and
+measurably degrades rare-technique coverage at scale (the comparison
+checkpoint is published as
+[...-llm-expanded](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded)).
+
+## Evaluation
+
+Held-out test split, this checkpoint (seed 42):
+
+| Metric | Value |
+|--------|-------|
+| Recall@5 | 0.6440 |
+| Recall@3 | 0.5181 |
+| F1 micro | 0.3899 |
+| F1 macro | 0.1910 |
+| Precision micro | 0.2740 |
+| Recall micro | 0.6756 |
+
+Recall@5 = 0.64 means that on average 64% of an unseen CVE's expert-assigned
+techniques appear in the model's top five suggestions — roughly double the
+zero-shot embedding-similarity baseline reported in the paper. Across five
+seeds under the identical protocol, the numbers of record are recall@5
+0.673 ± 0.019, recall@3 0.536 ± 0.032, micro-F1 0.410 ± 0.006, macro-F1
+0.177 ± 0.014. The complete trainer logs are published in the
+[paper repository](https://github.com/vulnerability-lookup/cve-attack-mapping-paper/tree/main/trainer-logs).
 
 ## Training procedure
 
+Binary cross-entropy over 53 sigmoid outputs, with per-label `pos_weight`
+balancing (capped at 20) to keep rare techniques trainable. Trained with
+`vulntrain-train-attack-classification` (VulnTrain).
+
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
@@ -48,6 +191,9 @@ The following hyperparameters were used during training:
 - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: linear
 - num_epochs: 40
+- max_length: 512
+- loss: BCEWithLogitsLoss, balanced pos_weight (min 2.447, max 20.0)
+- checkpoint selection: best macro-F1 on the validation split
 
 ### Training results
 
@@ -94,10 +240,37 @@ The following hyperparameters were used during training:
 | 0.4570        | 39.0  | 1209 | 0.6206          | 0.4350   | 0.1932   | 0.3326          | 0.6287       | 0.5281      | 0.6910      |
 | 0.4436        | 40.0  | 1240 | 0.6199          | 0.4325   | 0.1916   | 0.3296          | 0.6287       | 0.5329      | 0.6882      |
 
-
 ### Framework versions
 
 - Transformers 5.13.0
 - Pytorch 2.12.1+cu130
 - Datasets 4.8.5
 - Tokenizers 0.22.2
+
+## Related artifacts
+
+| Artifact | Location | DOI |
+|----------|----------|-----|
+| Gold dataset (1,207 CVEs, CTID-curated labels) | [CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques) | [10.57967/hf/9621](https://doi.org/10.57967/hf/9621) |
+| LLM expansion dataset (negative result) | [CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling) | [10.57967/hf/9622](https://doi.org/10.57967/hf/9622) |
+| LLM-expanded comparison model | [CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded) | [10.57967/hf/9624](https://doi.org/10.57967/hf/9624) |
+| Code | [vulnerability-lookup/VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) | — |
+| Paper + trainer logs | [vulnerability-lookup/cve-attack-mapping-paper](https://github.com/vulnerability-lookup/cve-attack-mapping-paper) | — |
+
+## Citation
+
+```bibtex
+@misc{bonhomme2026cveattack,
+  title  = {Mapping CVEs to MITRE ATT\&CK Techniques: A Curated Gold-Set
+            Classifier and the Limits of LLM-Assisted Label Expansion},
+  author = {Bonhomme, C{\'e}dric},
+  year   = {2026},
+  note   = {Preprint},
+}
+```
+
+## Acknowledgements
+
+Developed at [CIRCL](https://www.circl.lu) in the context of the
+[AIPITCH](https://www.science.nask.pl/en/research-areas/projects/12456)
+project, co-funded by the European Union.