Feature Extraction
Transformers
PyTorch
English
fill-mask
genomics
virology
dnabert
foundation-model
hvilm
viral-genomics
pathogenicity
transmissibility
host-tropism
hvue-v2
custom_code
Instructions to use duttaprat/HViLM-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use duttaprat/HViLM-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="duttaprat/HViLM-base", trust_remote_code=True)# Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("duttaprat/HViLM-base", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,307 +1,307 @@
|
|
| 1 |
---
|
| 2 |
language:
|
| 3 |
-
- en
|
| 4 |
license: apache-2.0
|
| 5 |
library_name: transformers
|
|
|
|
| 6 |
tags:
|
| 7 |
-
- genomics
|
| 8 |
-
- virology
|
| 9 |
-
- dnabert
|
| 10 |
-
- foundation-model
|
| 11 |
-
- hvilm
|
| 12 |
-
-
|
| 13 |
-
-
|
| 14 |
-
-
|
| 15 |
-
-
|
|
|
|
| 16 |
datasets:
|
| 17 |
-
-
|
| 18 |
-
- BV-BRC
|
| 19 |
-
- VHDB
|
| 20 |
-
- duttaprat/HVUE
|
| 21 |
pipeline_tag: feature-extraction
|
| 22 |
widget:
|
| 23 |
-
- text: "ATGCGTACGTTAGCCGATCG"
|
| 24 |
-
|
| 25 |
---
|
| 26 |
|
| 27 |
# HViLM-base: A Foundation Model for Viral Genomics
|
| 28 |
|
| 29 |
<div align="center">
|
| 30 |
|
| 31 |
-
[](LICENSE)
|
| 34 |
-
[](https://huggingface.co/duttaprat/HViLM-base)
|
| 35 |
|
| 36 |
</div>
|
| 37 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
## Model Description
|
| 39 |
|
| 40 |
-
**HViLM (Human Virome Language Model)** is
|
|
|
|
|
|
|
| 41 |
|
| 42 |
-
**
|
|
|
|
| 43 |
|
| 44 |
-
**Authors**
|
| 45 |
|
| 46 |
-
**Code
|
| 47 |
|
| 48 |
---
|
| 49 |
|
| 50 |
-
##
|
| 51 |
|
| 52 |
-
-
|
| 53 |
-
- 🎯 **Multi-task predictions** across 3 epidemiologically critical tasks:
|
| 54 |
-
- **Pathogenicity classification**: 95.32% average accuracy
|
| 55 |
-
- **Host tropism prediction**: 96.25% accuracy
|
| 56 |
-
- **Transmissibility assessment**: 97.36% average accuracy
|
| 57 |
-
- 📊 **[HVUE Benchmark](https://huggingface.co/datasets/duttaprat/HVUE)**: 7 curated datasets totaling 60K+ viral sequences
|
| 58 |
-
- 🔍 **Mechanistic interpretability**: Identifies transcription factor binding site mimicry (42 conserved motifs)
|
| 59 |
-
- ⚡ **Parameter-efficient fine-tuning**: LoRA adaptation (~0.3M trainable parameters per task)
|
| 60 |
-
- 🚀 **State-of-the-art performance**: Outperforms Nucleotide Transformer, GENA-LM, and DNABERT-MB
|
| 61 |
|
| 62 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
|
| 64 |
-
|
| 65 |
|
| 66 |
-
|
| 67 |
-
- **Tokenization**: Byte Pair Encoding (BPE) with vocabulary size 4,096
|
| 68 |
-
- **Max sequence length**: 1,000 base pairs
|
| 69 |
-
- **Hidden size**: 768
|
| 70 |
-
- **Attention heads**: 12
|
| 71 |
-
- **Layers**: 12
|
| 72 |
-
- **Positional encoding**: Attention with Linear Biases (ALiBi)
|
| 73 |
|
| 74 |
-
|
| 75 |
-
- **Objective**: Masked Language Modeling (MLM)
|
| 76 |
-
- **Training data**: 5M viral sequence chunks (non-overlapping, 1000 bp)
|
| 77 |
-
- **Data source**: VIRION database (clustered at 80% identity with MMseqs2)
|
| 78 |
-
- **Training**: 10 epochs, AdamW optimizer, learning rate 5e-5
|
| 79 |
-
- **Hardware**: 4x NVIDIA A100 GPUs (72 hours)
|
| 80 |
-
- **Performance**: 94.2% MLM accuracy on validation set
|
| 81 |
|
| 82 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
|
| 84 |
-
|
| 85 |
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
---
|
| 91 |
|
| 92 |
## Quick Start
|
| 93 |
|
| 94 |
-
###
|
| 95 |
|
| 96 |
```python
|
| 97 |
-
from transformers import AutoTokenizer, AutoModel
|
| 98 |
import torch
|
|
|
|
|
|
|
|
|
|
| 99 |
|
| 100 |
-
# Load model and tokenizer
|
| 101 |
tokenizer = AutoTokenizer.from_pretrained(
|
| 102 |
-
|
| 103 |
-
trust_remote_code=True
|
| 104 |
)
|
|
|
|
| 105 |
model = AutoModel.from_pretrained(
|
| 106 |
-
|
| 107 |
-
trust_remote_code=True
|
| 108 |
)
|
| 109 |
|
| 110 |
-
|
| 111 |
-
viral_sequence = "ATGCGTACGTTAGCCGATCGATTACGCGTACGTAGCTAGCTAGCT"
|
| 112 |
|
| 113 |
-
# Tokenize
|
| 114 |
inputs = tokenizer(
|
| 115 |
-
|
| 116 |
return_tensors="pt",
|
| 117 |
truncation=True,
|
| 118 |
-
|
| 119 |
-
padding=True
|
| 120 |
)
|
| 121 |
|
| 122 |
-
# Generate embeddings
|
| 123 |
with torch.no_grad():
|
| 124 |
outputs = model(**inputs)
|
| 125 |
-
embeddings = outputs.last_hidden_state # [batch_size, seq_len, 768]
|
| 126 |
|
| 127 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
|
| 129 |
-
|
| 130 |
-
attention_mask = inputs['attention_mask']
|
| 131 |
-
mask_expanded = attention_mask.unsqueeze(-1).expand(embeddings.size()).float()
|
| 132 |
-
sum_embeddings = torch.sum(embeddings * mask_expanded, dim=1)
|
| 133 |
-
sum_mask = torch.clamp(mask_expanded.sum(dim=1), min=1e-9)
|
| 134 |
-
mean_embeddings = sum_embeddings / sum_mask
|
| 135 |
|
| 136 |
-
|
| 137 |
-
```
|
| 138 |
|
| 139 |
-
|
| 140 |
|
| 141 |
-
|
| 142 |
|
| 143 |
```python
|
| 144 |
-
|
| 145 |
-
from transformers import AutoModel, TrainingArguments, Trainer
|
| 146 |
-
from peft import LoraConfig, get_peft_model
|
| 147 |
-
|
| 148 |
-
# Load base model
|
| 149 |
-
model = AutoModel.from_pretrained("duttaprat/HViLM-base", trust_remote_code=True)
|
| 150 |
-
|
| 151 |
-
# Configure LoRA for parameter-efficient fine-tuning
|
| 152 |
-
lora_config = LoraConfig(
|
| 153 |
-
r=8, # rank
|
| 154 |
-
lora_alpha=16, # scaling factor
|
| 155 |
-
target_modules=["query", "value"], # attention layers
|
| 156 |
-
lora_dropout=0.1,
|
| 157 |
-
bias="none"
|
| 158 |
-
)
|
| 159 |
|
| 160 |
-
|
| 161 |
-
model = get_peft_model(model, lora_config)
|
| 162 |
|
| 163 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
```
|
| 165 |
|
| 166 |
-
|
| 167 |
|
| 168 |
-
|
|
|
|
| 169 |
|
| 170 |
-
###
|
| 171 |
|
| 172 |
-
|
| 173 |
-
|---------|-----------|----------|----------|-----|
|
| 174 |
-
| CINI | 159 | **87.74%** | 86.98 | 74.48 |
|
| 175 |
-
| BVBRC-CoV | 18,066 | **98.26%** | 98.26 | 96.52 |
|
| 176 |
-
| BVBRC-Calici | 31,089 | **99.95%** | 99.93 | 99.90 |
|
| 177 |
-
| **Average** | **49,314** | **95.32%** | **95.06** | **90.30** |
|
| 178 |
|
| 179 |
-
|
|
|
|
|
|
|
| 180 |
|
| 181 |
-
|
| 182 |
-
|---------|-----------|----------|----------|-----|
|
| 183 |
-
| VHDB | 9,428 | **96.25%** | 91.34 | 91.24 |
|
| 184 |
|
| 185 |
-
|
|
|
|
| 186 |
|
| 187 |
-
|
| 188 |
-
|--------------|-----------|----------|----------|-----|
|
| 189 |
-
| Coronaviridae | ~3,000 | **97.45%** | 97.37 | 93.43 |
|
| 190 |
-
| Orthomyxoviridae | ~2,500 | **95.62%** | 95.44 | 91.07 |
|
| 191 |
-
| Caliciviridae | ~1,800 | **99.95%** | 99.95 | 99.90 |
|
| 192 |
-
| **Average** | **~7,300** | **97.36%** | **97.59** | **94.80** |
|
| 193 |
|
| 194 |
-
|
| 195 |
|
| 196 |
-
|
|
|
|
|
|
|
| 197 |
|
| 198 |
-
|
| 199 |
|
| 200 |
-
|
|
|
|
| 201 |
|
| 202 |
-
|
| 203 |
-
- **10 vertebrate transcription factors** targeted, including:
|
| 204 |
-
- **Irf1** (Interferon Regulatory Factor 1): 8 convergent motifs for immune evasion
|
| 205 |
-
- **Foxq1**: Multiple motifs for epithelial cell tropism
|
| 206 |
-
- **ZNF354A**: 6 motifs for chromatin regulation
|
| 207 |
-
|
| 208 |
-
This demonstrates that HViLM captures genuine biological mechanisms rather than spurious correlations.
|
| 209 |
|
| 210 |
---
|
| 211 |
|
| 212 |
-
##
|
| 213 |
|
| 214 |
-
|
| 215 |
|
| 216 |
-
|
| 217 |
-
- **Genomes**: 10,817,265 unique NCBI accession numbers
|
| 218 |
-
- **Processing**:
|
| 219 |
-
- Segmented into non-overlapping 1000 bp chunks
|
| 220 |
-
- Clustered with MMseqs2 at 80% identity threshold
|
| 221 |
-
- **Final dataset**: 5 million unique sequences
|
| 222 |
-
- **Coverage**: 45+ viral families across all Baltimore classification groups
|
| 223 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 224 |
|
| 225 |
-
|
| 226 |
|
| 227 |
-
|
|
|
|
|
|
|
| 228 |
|
| 229 |
-
|
| 230 |
|
| 231 |
-
###
|
| 232 |
-
- **CINI**: 159 sequences, 4 viral families, manual literature curation
|
| 233 |
-
- **BVBRC-CoV**: 18,066 coronaviruses
|
| 234 |
-
- **BVBRC-Calici**: 31,089 caliciviruses
|
| 235 |
|
| 236 |
-
|
| 237 |
-
- **VHDB**: 9,428 sequences, 30 viral families
|
| 238 |
-
- Binary classification: human-tropic (13.1%) vs non-human-tropic (86.9%)
|
| 239 |
|
| 240 |
-
|
| 241 |
-
-
|
| 242 |
-
|
| 243 |
-
|
|
|
|
| 244 |
|
| 245 |
-
|
| 246 |
|
| 247 |
-
###
|
| 248 |
-
```python
|
| 249 |
-
from datasets import load_dataset
|
| 250 |
|
| 251 |
-
|
| 252 |
-
host_tropism = load_dataset("duttaprat/HVUE", data_dir="Host_Tropism")
|
| 253 |
-
pathogenicity = load_dataset("duttaprat/HVUE", data_dir="Pathogenecity")
|
| 254 |
-
transmissibility = load_dataset("duttaprat/HVUE", data_dir="Transmissibility")
|
| 255 |
|
| 256 |
-
|
| 257 |
-
|
| 258 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 259 |
|
| 260 |
---
|
| 261 |
|
| 262 |
-
##
|
| 263 |
|
| 264 |
-
###
|
| 265 |
-
```python
|
| 266 |
-
from datasets import load_dataset
|
| 267 |
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
|
| 271 |
-
transmissibility = load_dataset("duttaprat/HVUE", data_dir="Transmissibility")
|
| 272 |
-
```
|
| 273 |
|
| 274 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 275 |
|
| 276 |
-
|
| 277 |
|
|
|
|
| 278 |
|
|
|
|
| 279 |
|
| 280 |
-
|
| 281 |
-
# Clone repository
|
| 282 |
-
git clone https://github.com/duttaprat/HViLM.git
|
| 283 |
-
cd HViLM
|
| 284 |
|
| 285 |
-
|
| 286 |
-
pip install -r requirements.txt
|
| 287 |
|
| 288 |
-
|
| 289 |
-
cd finetune
|
| 290 |
-
bash scripts/run_patho_cini.sh
|
| 291 |
|
| 292 |
-
#
|
| 293 |
-
bash scripts/run_tropism_vhdb.sh
|
| 294 |
|
| 295 |
-
|
| 296 |
-
|
| 297 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 298 |
|
| 299 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 300 |
|
| 301 |
---
|
| 302 |
|
| 303 |
## Citation
|
| 304 |
-
|
|
|
|
| 305 |
|
| 306 |
```bibtex
|
| 307 |
@article{dutta2026hvilm,
|
|
@@ -313,37 +313,37 @@ If you use HViLM in your research, please cite our paper:
|
|
| 313 |
publisher={Cold Spring Harbor Laboratory}
|
| 314 |
}
|
| 315 |
```
|
|
|
|
|
|
|
|
|
|
| 316 |
---
|
| 317 |
|
| 318 |
## Model Card Authors
|
| 319 |
|
| 320 |
-
- **Pratik Dutta**
|
| 321 |
-
- **Ramana V. Davuluri**
|
| 322 |
|
| 323 |
---
|
| 324 |
|
| 325 |
## Contact
|
| 326 |
|
| 327 |
-
- **
|
| 328 |
-
- **Lab**
|
| 329 |
-
- **GitHub Issues**: [Report bugs or request features](https://github.com/duttaprat/HViLM/issues)
|
| 330 |
|
| 331 |
---
|
| 332 |
|
| 333 |
## Acknowledgments
|
| 334 |
|
| 335 |
-
|
| 336 |
-
|
| 337 |
-
|
| 338 |
|
| 339 |
---
|
| 340 |
|
| 341 |
## License
|
| 342 |
|
| 343 |
-
|
| 344 |
|
| 345 |
---
|
| 346 |
|
| 347 |
## Disclaimer
|
| 348 |
|
| 349 |
-
HViLM is a research
|
|
|
|
| 1 |
---
|
| 2 |
language:
|
| 3 |
+
- en
|
| 4 |
license: apache-2.0
|
| 5 |
library_name: transformers
|
| 6 |
+
base_model: zhihan1996/DNABERT-2-117M
|
| 7 |
tags:
|
| 8 |
+
- genomics
|
| 9 |
+
- virology
|
| 10 |
+
- dnabert
|
| 11 |
+
- foundation-model
|
| 12 |
+
- hvilm
|
| 13 |
+
- viral-genomics
|
| 14 |
+
- pathogenicity
|
| 15 |
+
- transmissibility
|
| 16 |
+
- host-tropism
|
| 17 |
+
- hvue-v2
|
| 18 |
datasets:
|
| 19 |
+
- duttaprat/HVUE-v2
|
|
|
|
|
|
|
|
|
|
| 20 |
pipeline_tag: feature-extraction
|
| 21 |
widget:
|
| 22 |
+
- text: "ATGCGTACGTTAGCCGATCG"
|
| 23 |
+
example_title: "Virus sequence example"
|
| 24 |
---
|
| 25 |
|
| 26 |
# HViLM-base: A Foundation Model for Viral Genomics
|
| 27 |
|
| 28 |
<div align="center">
|
| 29 |
|
| 30 |
+
[](https://www.biorxiv.org/content/10.64898/2026.03.18.712700v1)
|
| 31 |
+
[](https://github.com/duttaprat/HViLM)
|
| 32 |
+
[](https://huggingface.co/datasets/duttaprat/HVUE-v2)
|
| 33 |
[](LICENSE)
|
|
|
|
| 34 |
|
| 35 |
</div>
|
| 36 |
|
| 37 |
+
> [!IMPORTANT]
|
| 38 |
+
> **HVUE v2 supersedes the original HVUE benchmark.**
|
| 39 |
+
> The original HVUE v1 benchmark contained substantial cross-split sequence similarity that could inflate held-out performance estimates. HVUE v2 was rebuilt using source-sequence clustering **before** train/validation/test assignment and chunking, followed by exact- and near-match leakage auditing.
|
| 40 |
+
> **Use [duttaprat/HVUE-v2](https://huggingface.co/datasets/duttaprat/HVUE-v2) for current HViLM evaluation.**
|
| 41 |
+
|
| 42 |
## Model Description
|
| 43 |
|
| 44 |
+
**HViLM (Human Virome Language Model)** is a genomic foundation model adapted to virus sequences through continued pre-training of [DNABERT-2](https://huggingface.co/zhihan1996/DNABERT-2-117M). HViLM-base was trained on approximately **5 million non-redundant virus-derived sequence fragments** from the [VIRION](https://virion.verena.org/) resource, representing approximately 9,000 virus species across 45+ families.
|
| 45 |
+
|
| 46 |
+
The architecture and tokenizer remain those of DNABERT-2; continued pre-training updates the model weights using a masked-language-modeling objective on the virus-focused corpus.
|
| 47 |
|
| 48 |
+
**Preprint:** *HViLM: A Foundation Model for Viral Genomics Enables Multi-Task Prediction of Pathogenicity, Transmissibility, and Host Tropism*
|
| 49 |
+
[bioRxiv 2026.03.18.712700](https://www.biorxiv.org/content/10.64898/2026.03.18.712700v1)
|
| 50 |
|
| 51 |
+
**Authors:** Pratik Dutta, Jack Vaska, Pallavi Surana, Rekha Sathian, Max Chao, Zhihan Zhou, Han Liu, and Ramana V. Davuluri
|
| 52 |
|
| 53 |
+
**Code:** [github.com/duttaprat/HViLM](https://github.com/duttaprat/HViLM)
|
| 54 |
|
| 55 |
---
|
| 56 |
|
| 57 |
+
## HViLM Model Family
|
| 58 |
|
| 59 |
+
HViLM-base is the continued-pretrained foundation model. Official task-specific models fine-tuned on HVUE v2 are released as standalone checkpoints:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
+
| Resource | Purpose |
|
| 62 |
+
|---|---|
|
| 63 |
+
| [HViLM-base](https://huggingface.co/duttaprat/HViLM-base) | Continued-pretrained foundation model / sequence representations |
|
| 64 |
+
| [HViLM-Patho](https://huggingface.co/duttaprat/HViLM-Patho) | Pathogenicity classification |
|
| 65 |
+
| [HViLM-R0](https://huggingface.co/duttaprat/HViLM-R0) | Transmissibility classification |
|
| 66 |
+
| [HViLM-Tropism](https://huggingface.co/duttaprat/HViLM-Tropism) | Human host-tropism classification |
|
| 67 |
+
| [HVUE-v2](https://huggingface.co/datasets/duttaprat/HVUE-v2) | Leakage-controlled benchmark |
|
| 68 |
|
| 69 |
+
The complete project is also grouped in the **HViLM: Human Virome Language Model** collection on the [duttaprat Collections page](https://huggingface.co/duttaprat/collections).
|
| 70 |
|
| 71 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
|
| 73 |
+
## Key Features
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
+
- **Virus-focused continued pre-training:** approximately 5M non-redundant fragments derived from VIRION-linked virus sequences.
|
| 76 |
+
- **DNABERT-2 initialization:** preserves the DNABERT-2 architecture and BPE tokenizer while adapting model weights to virus sequence data.
|
| 77 |
+
- **Three official downstream models:** pathogenicity, transmissibility, and host tropism.
|
| 78 |
+
- **HVUE v2 evaluation:** cluster-aware splitting before chunking, with multiple similarity stringencies and sequence lengths.
|
| 79 |
+
- **Parameter-efficient downstream adaptation:** official task models were trained with LoRA.
|
| 80 |
+
- **Public reproducibility resources:** base model, three task-specific checkpoints, HVUE v2 benchmark, and project code are released publicly.
|
| 81 |
|
| 82 |
+
---
|
| 83 |
|
| 84 |
+
## Model Architecture and Continued Pre-training
|
| 85 |
+
|
| 86 |
+
HViLM-base is derived from **DNABERT-2 (117M parameters)**.
|
| 87 |
+
|
| 88 |
+
| Property | Value |
|
| 89 |
+
|---|---|
|
| 90 |
+
| Architecture | MosaicBERT / DNABERT-2 |
|
| 91 |
+
| Parameters | ~117M |
|
| 92 |
+
| Hidden size | 768 |
|
| 93 |
+
| Transformer layers | 12 |
|
| 94 |
+
| Attention heads | 12 |
|
| 95 |
+
| Tokenization | Byte Pair Encoding (BPE) |
|
| 96 |
+
| Positional method | ALiBi |
|
| 97 |
+
| Continued-pretraining objective | Masked Language Modeling |
|
| 98 |
+
| Pretraining fragment length | 1000 nt |
|
| 99 |
+
| Final virus-focused corpus | ~5M non-redundant fragments |
|
| 100 |
+
| Redundancy reduction | MMseqs2 clustering at 80% identity / 80% coverage |
|
| 101 |
+
| Optimizer | AdamW |
|
| 102 |
+
| Learning rate | 5e-5 |
|
| 103 |
+
| Training | 10 epochs |
|
| 104 |
+
| Hardware | 4 × NVIDIA A100 GPUs |
|
| 105 |
+
| Approximate training time | 72 hours |
|
| 106 |
+
| Held-out MLM accuracy | 94.2% |
|
| 107 |
+
|
| 108 |
+
**Sequence-length note:** HViLM uses BPE tokenization, so nucleotide length and model-token length are not equivalent. The continued-pretraining corpus used 1000-nt sequence fragments; downstream configurations are described by nucleotide length in HVUE v2.
|
| 109 |
|
| 110 |
---
|
| 111 |
|
| 112 |
## Quick Start
|
| 113 |
|
| 114 |
+
### Extract sequence representations from HViLM-base
|
| 115 |
|
| 116 |
```python
|
|
|
|
| 117 |
import torch
|
| 118 |
+
from transformers import AutoTokenizer, AutoModel
|
| 119 |
+
|
| 120 |
+
model_id = "duttaprat/HViLM-base"
|
| 121 |
|
|
|
|
| 122 |
tokenizer = AutoTokenizer.from_pretrained(
|
| 123 |
+
model_id,
|
| 124 |
+
trust_remote_code=True,
|
| 125 |
)
|
| 126 |
+
|
| 127 |
model = AutoModel.from_pretrained(
|
| 128 |
+
model_id,
|
| 129 |
+
trust_remote_code=True,
|
| 130 |
)
|
| 131 |
|
| 132 |
+
sequence = "ATGCGTACGTTAGCCGATCGATTACGCGTACGTAGCTAGCTAGCT"
|
|
|
|
| 133 |
|
|
|
|
| 134 |
inputs = tokenizer(
|
| 135 |
+
sequence,
|
| 136 |
return_tensors="pt",
|
| 137 |
truncation=True,
|
| 138 |
+
padding=True,
|
|
|
|
| 139 |
)
|
| 140 |
|
|
|
|
| 141 |
with torch.no_grad():
|
| 142 |
outputs = model(**inputs)
|
|
|
|
| 143 |
|
| 144 |
+
token_embeddings = outputs.last_hidden_state
|
| 145 |
+
print(token_embeddings.shape)
|
| 146 |
+
```
|
| 147 |
+
|
| 148 |
+
For sequence-level representations, pooling strategy should be chosen according to the downstream task rather than treated as a fixed property of HViLM-base.
|
| 149 |
|
| 150 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
|
| 152 |
+
## Use the Official Fine-tuned Models
|
|
|
|
| 153 |
|
| 154 |
+
If the goal is one of the three HVUE v2 tasks, users can load the corresponding task model directly; `HViLM-base` does not need to be loaded separately.
|
| 155 |
|
| 156 |
+
### Pathogenicity
|
| 157 |
|
| 158 |
```python
|
| 159 |
+
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
|
| 161 |
+
model_id = "duttaprat/HViLM-Patho"
|
|
|
|
| 162 |
|
| 163 |
+
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
| 164 |
+
model = AutoModelForSequenceClassification.from_pretrained(
|
| 165 |
+
model_id,
|
| 166 |
+
trust_remote_code=True,
|
| 167 |
+
)
|
| 168 |
```
|
| 169 |
|
| 170 |
+
Labels:
|
| 171 |
|
| 172 |
+
- `0` → `NON_PATHOGENIC`
|
| 173 |
+
- `1` → `PATHOGENIC`
|
| 174 |
|
| 175 |
+
### Transmissibility
|
| 176 |
|
| 177 |
+
Use:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 178 |
|
| 179 |
+
```text
|
| 180 |
+
duttaprat/HViLM-R0
|
| 181 |
+
```
|
| 182 |
|
| 183 |
+
Labels:
|
|
|
|
|
|
|
| 184 |
|
| 185 |
+
- `0` → `R0_LT_1`
|
| 186 |
+
- `1` → `R0_GE_1`
|
| 187 |
|
| 188 |
+
### Host Tropism
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 189 |
|
| 190 |
+
Use:
|
| 191 |
|
| 192 |
+
```text
|
| 193 |
+
duttaprat/HViLM-Tropism
|
| 194 |
+
```
|
| 195 |
|
| 196 |
+
Labels:
|
| 197 |
|
| 198 |
+
- `0` → `NON_HUMAN_TROPIC`
|
| 199 |
+
- `1` → `HUMAN_TROPIC`
|
| 200 |
|
| 201 |
+
See the individual model cards for full usage examples and task-specific limitations.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 202 |
|
| 203 |
---
|
| 204 |
|
| 205 |
+
## HVUE v2 Benchmark
|
| 206 |
|
| 207 |
+
[HVUE v2](https://huggingface.co/datasets/duttaprat/HVUE-v2) is the current evaluation benchmark for HViLM. It replaces HVUE v1.
|
| 208 |
|
| 209 |
+
The benchmark was reconstructed to reduce supervised train-test leakage using the following ordering:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 210 |
|
| 211 |
+
1. consolidate and deduplicate source sequences;
|
| 212 |
+
2. cluster source sequences with MMseqs2;
|
| 213 |
+
3. assign complete clusters to train/validation/test splits;
|
| 214 |
+
4. chunk sequences only **after** split assignment;
|
| 215 |
+
5. remove exact duplicate chunks;
|
| 216 |
+
6. audit cross-split exact and near matches.
|
| 217 |
|
| 218 |
+
HVUE v2 includes:
|
| 219 |
|
| 220 |
+
- **Pathogenicity**
|
| 221 |
+
- **Transmissibility**
|
| 222 |
+
- **Host Tropism**
|
| 223 |
|
| 224 |
+
Across the benchmark, configurations evaluate different sequence lengths (500, 1000, and 2000 nt where applicable), sequence-similarity stringencies, and temporal generalization where reliable collection-date metadata are available.
|
| 225 |
|
| 226 |
+
### Primary HViLM Results
|
|
|
|
|
|
|
|
|
|
| 227 |
|
| 228 |
+
The primary results below use the **standard 1000-nt configuration** for each task.
|
|
|
|
|
|
|
| 229 |
|
| 230 |
+
| Task | HVUE v2 configuration | Accuracy | F1 | MCC | Official model |
|
| 231 |
+
|---|---|---:|---:|---:|---|
|
| 232 |
+
| Pathogenicity | `standard_capped_1000bp` | **92.39** | **91.32** | **83.10** | [HViLM-Patho](https://huggingface.co/duttaprat/HViLM-Patho) |
|
| 233 |
+
| Transmissibility | `standard_capped_1000bp` | **87.50** | **86.16** | **72.66** | [HViLM-R0](https://huggingface.co/duttaprat/HViLM-R0) |
|
| 234 |
+
| Host Tropism | `standard_95_1000bp` | **96.49** | **74.49** | **48.99** | [HViLM-Tropism](https://huggingface.co/duttaprat/HViLM-Tropism) |
|
| 235 |
|
| 236 |
+
The directory/configuration identifiers retain `bp` for release stability; manuscript and descriptive text use **nt** for nucleotide sequence length.
|
| 237 |
|
| 238 |
+
### Interpretation of the HVUE v2 Results
|
|
|
|
|
|
|
| 239 |
|
| 240 |
+
Under leakage-controlled evaluation, the effect of virus-focused continued pre-training is **task dependent**:
|
|
|
|
|
|
|
|
|
|
| 241 |
|
| 242 |
+
- **Pathogenicity:** HViLM improves F1 by 1.28 points over vanilla DNABERT-2 (91.32 vs. 90.04).
|
| 243 |
+
- **Transmissibility:** HViLM and DNABERT-2 are close (86.16 vs. 85.81 F1), and HViLM is essentially tied with DNABERT-MB (86.16 vs. 86.15 F1).
|
| 244 |
+
- **Host Tropism:** HViLM shows the largest F1 improvement, reaching 74.49 compared with 64.82 for class-balanced DNABERT-2.
|
| 245 |
+
|
| 246 |
+
These results support a more specific conclusion than the original HVUE v1 evaluation: virus-focused continued pre-training provides its clearest benefit on the more challenging Host Tropism task, while gains on Pathogenicity and Transmissibility are smaller.
|
| 247 |
+
|
| 248 |
+
For complete baseline comparisons, hard-split evaluations, temporal evaluations, and sequence-length analyses, see the [HViLM GitHub repository](https://github.com/duttaprat/HViLM) and [HVUE-v2](https://huggingface.co/datasets/duttaprat/HVUE-v2).
|
| 249 |
|
| 250 |
---
|
| 251 |
|
| 252 |
+
## Training Data
|
| 253 |
|
| 254 |
+
### Continued-pretraining corpus
|
|
|
|
|
|
|
| 255 |
|
| 256 |
+
HViLM-base was trained using virus sequences associated with the **VIRION** resource.
|
| 257 |
+
|
| 258 |
+
Processing included:
|
|
|
|
|
|
|
| 259 |
|
| 260 |
+
- retrieval and quality control of VIRION-linked nucleotide sequences;
|
| 261 |
+
- removal of short sequences and exact duplicates;
|
| 262 |
+
- segmentation into non-overlapping 1000-nt fragments;
|
| 263 |
+
- MMseqs2 clustering at 80% sequence identity and 80% coverage;
|
| 264 |
+
- selection of approximately 5M representative fragments for continued pre-training.
|
| 265 |
|
| 266 |
+
The corpus spans approximately 9,000 virus species and 45+ virus families across the Baltimore classification groups.
|
| 267 |
|
| 268 |
+
---
|
| 269 |
|
| 270 |
+
## Interpretability
|
| 271 |
|
| 272 |
+
Attention-guided analyses associated with the HViLM study identified **candidate sequence motifs** in pathogenic coronavirus sequences, including motifs with similarity to vertebrate transcription-factor binding motifs such as IRF1, FOXQ1, and ZNF354A.
|
|
|
|
|
|
|
|
|
|
| 273 |
|
| 274 |
+
These observations are **hypothesis-generating**. Sequence similarity between virus motifs and host transcription-factor binding motifs does not by itself establish molecular mimicry, causal regulation, immune evasion, or another biological mechanism. Experimental validation and additional controls are required for mechanistic interpretation.
|
|
|
|
| 275 |
|
| 276 |
+
---
|
|
|
|
|
|
|
| 277 |
|
| 278 |
+
## Limitations
|
|
|
|
| 279 |
|
| 280 |
+
- HVUE v2 controls supervised split leakage through source-level clustering and auditing, but sequence-similarity thresholds cannot eliminate every form of biological relatedness.
|
| 281 |
+
- The complete historical training exposure of the original DNABERT-2 model cannot be reconstructed; therefore, absence of all possible ancestral pretraining exposure to benchmark-related sequences cannot be guaranteed.
|
| 282 |
+
- Host association is biologically context-dependent and may include multi-host, zoonotic, and reverse-zoonotic relationships; the benchmark uses a simplified binary formulation.
|
| 283 |
+
- R₀-based transmissibility labels simplify a continuous, context-dependent epidemiological quantity into a binary benchmark task.
|
| 284 |
+
- Performance differences between closely matched models should not be interpreted as statistically meaningful without uncertainty estimates or repeated evaluations.
|
| 285 |
+
- Attention-based motif analyses should be considered exploratory rather than direct evidence of mechanism.
|
| 286 |
+
- HViLM predictions are research outputs and are not intended to replace experimental, clinical, epidemiological, or public-health assessment.
|
| 287 |
+
|
| 288 |
+
---
|
| 289 |
+
|
| 290 |
+
## Reproducibility and Resources
|
| 291 |
|
| 292 |
+
- **Base model:** [duttaprat/HViLM-base](https://huggingface.co/duttaprat/HViLM-base)
|
| 293 |
+
- **Pathogenicity model:** [duttaprat/HViLM-Patho](https://huggingface.co/duttaprat/HViLM-Patho)
|
| 294 |
+
- **Transmissibility model:** [duttaprat/HViLM-R0](https://huggingface.co/duttaprat/HViLM-R0)
|
| 295 |
+
- **Host Tropism model:** [duttaprat/HViLM-Tropism](https://huggingface.co/duttaprat/HViLM-Tropism)
|
| 296 |
+
- **Benchmark:** [duttaprat/HVUE-v2](https://huggingface.co/datasets/duttaprat/HVUE-v2)
|
| 297 |
+
- **Code:** [github.com/duttaprat/HViLM](https://github.com/duttaprat/HViLM)
|
| 298 |
+
- **Collections:** [duttaprat's Hugging Face Collections](https://huggingface.co/duttaprat/collections)
|
| 299 |
|
| 300 |
---
|
| 301 |
|
| 302 |
## Citation
|
| 303 |
+
|
| 304 |
+
If you use HViLM in your research, please cite:
|
| 305 |
|
| 306 |
```bibtex
|
| 307 |
@article{dutta2026hvilm,
|
|
|
|
| 313 |
publisher={Cold Spring Harbor Laboratory}
|
| 314 |
}
|
| 315 |
```
|
| 316 |
+
|
| 317 |
+
If you use DNABERT-2 directly or build on its architecture, please also cite the DNABERT-2 publication.
|
| 318 |
+
|
| 319 |
---
|
| 320 |
|
| 321 |
## Model Card Authors
|
| 322 |
|
| 323 |
+
- **Pratik Dutta** — Stony Brook University
|
| 324 |
+
- **Ramana V. Davuluri** — Stony Brook University
|
| 325 |
|
| 326 |
---
|
| 327 |
|
| 328 |
## Contact
|
| 329 |
|
| 330 |
+
- **GitHub Issues:** [github.com/duttaprat/HViLM/issues](https://github.com/duttaprat/HViLM/issues)
|
| 331 |
+
- **Lab:** [Davuluri Lab, Stony Brook University](https://davulurilab.github.io/)
|
|
|
|
| 332 |
|
| 333 |
---
|
| 334 |
|
| 335 |
## Acknowledgments
|
| 336 |
|
| 337 |
+
HViLM builds on DNABERT-2 by Zhou et al. Continued-pretraining data were derived from the VIRION resource maintained by the Viral Emergence Research Initiative (Verena).
|
|
|
|
|
|
|
| 338 |
|
| 339 |
---
|
| 340 |
|
| 341 |
## License
|
| 342 |
|
| 343 |
+
HViLM-base is released under the **Apache License 2.0**.
|
| 344 |
|
| 345 |
---
|
| 346 |
|
| 347 |
## Disclaimer
|
| 348 |
|
| 349 |
+
HViLM is a research model for computational biology. It should not be used as the sole basis for clinical, diagnostic, epidemiological, biosurveillance, or public-health decisions. Model outputs should be interpreted alongside appropriate biological evidence and expert assessment.
|