Update README.md

README.md

@@ -10,19 +10,81 @@ datasets:
 - naufalso/cybersec-topic-classification-dataset-filtered
 ---
 
-# Model Trained Using AutoTrain
+# Model Card: Cybersecurity Text Classifier (ModernBERT-base)
 
-- Problem type: Text Classification
+<p align="center">
+  <b> "RedSage: A Cybersecurity Generalist LLM" (ICLR 2026) </b>
+  <br>
+  <b>Authors:</b> Naufal Suryanto<sup>1*</sup>, Muzammal Naseer<sup>1</sup>, Pengfei Li<sup>1</sup>, Syed Talal Wasim<sup>2</sup>, Jinhui Yi<sup>2</sup>, Juergen Gall<sup>2</sup>, Paolo Ceravolo<sup>3</sup>, Ernesto Damiani<sup>3</sup>
+  <br>
+  <sup>1</sup>Khalifa University, <sup>2</sup>University of Bonn, <sup>3</sup>University of Milan
+  <br>
+  <sup>*</sup>Project Lead
+  <br>
+  <br>
+  <a href="https://openreview.net/forum?id=W4FAenIrQ2"><img src="https://img.shields.io/badge/Paper-OpenReview-B31B1B.svg"></a>
+  <a href="https://huggingface.co/RISys-Lab"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-RISys--Lab-orange"></a>
+</p>
 
-## Validation Metrics
-loss: 0.07552551478147507
+---
+
+## Model Details
+
+* **Model Type**: Binary text classification model developed for domain-specific content filtering.
+* **Architecture**: Based on **ModernBERT-base**, a bidirectional transformer encoder optimized for efficiency and long-context performance.
+* **Domain**: Cybersecurity vs. Non-Cybersecurity.
+* **Developers**: Researchers affiliated with Khalifa University, the University of Bonn, the Lamarr Institute, and the University of Milan.
+* **License**: Released as part of the open-source RedSage project resources.
+
+## Intended Use
+
+* **Primary Use Case**: Identifying cybersecurity-relevant documents within large-scale, unstructured web corpora such as FineWeb.
+* **Application**: Filtering approximately 17.2 trillion tokens from Common Crawl subsets (2013–2024) to curate the 11.7B-token CyberFineWeb corpus.
+* **Intended Users**: Researchers and developers focused on domain continual pretraining for cybersecurity LLMs.
+
+## Training Data
+
+* **Source Dataset**: Cybersecurity Topic Classification dataset.
+* **Data Origin**: Labeled samples collected from Reddit, StackExchange, and arXiv, alongside web articles.
+* **Dataset Size**:
+    * **Pre-processing**: 9.27M training samples and 459K validation samples.
+    * **Post-filtering**: Reduced to 4.62M training samples and 2.46K validation samples after removing very short texts to minimize ambiguity.
+* **Labeling Method**: Derived from forum categories, tags, and keyword metadata rather than LLM-generated annotations.
 
-f1: 0.9142125480153649
+## Training Procedure
 
-precision: 0.9272727272727272
+* **Optimizer**: Adam optimizer.
+* **Learning Rate**: 2e-5.
+* **Schedule**: 10% warmup ratio over 2 training epochs.
+* **Hardware**: Implementation utilized the ModernBERT-base encoder as the foundation for the binary head.
 
-recall: 0.9015151515151515
+## Evaluation Results
+
+The model was evaluated on a validation set of 2,460 samples derived from web articles, achieving the following metrics:
+
+| Metric | Score |
+| :--- | :--- |
+| **Accuracy** | 97.3% |
+| **Precision** | 92.8% |
+| **Recall** | 90.2% |
+| **F1 Score** | 91.4% |
+
+## Limitations & Risks
+
+* **Context Sensitivity**: While highly accurate, the model was specifically filtered to exclude very short texts to avoid context ambiguity.
+* **Temporal Bias**: The model identifies cybersecurity content based on trends observed in web data up to late 2024; emerging threats post-2024 may not be represented.
+* **Dual-Use Concerns**: The classifier is designed to identify offensive security technical content, which carries an inherent risk of misuse if applied outside of defensive or educational research.
+
+---
 
-auc: 0.9906997543262077
+## Citation
 
-accuracy: 0.9727642276422764
+```bibtex
+@inproceedings{suryanto2026redsage,
+  title={RedSage: A Cybersecurity Generalist {LLM}},
+  author={Naufal Suryanto and Muzammal Naseer and Pengfei Li and Syed Talal Wasim and Jinhui Yi and Juergen Gall and Paolo Ceravolo and Ernesto Damiani},
+  booktitle={The Fourteenth International Conference on Learning Representations},
+  year={2026},
+  url={https://openreview.net/forum?id=W4FAenIrQ2}
+}
+```