DermFM-Zero: A Vision-Language Foundation Model for Zero-Shot Clinical Collaboration and Automated Concept Discovery in Dermatology

📢 This is the public release checkpoint of DermFM-Zero. It was retrained on 517,455 publicly available image–text pairs using improved training approaches and performs on par with the paper checkpoint (mean zero-shot score 0.691 versus 0.675 across seven benchmarks). Training details and full results: technical report. Contact: siyuan.yan@monash.edu.

Model Description

DermFM-Zero is a dermatology vision–language foundation model. It pairs a PanDerm ViT-L/16 vision encoder with native-resolution input (NaViT patch-and-pack) and a PubMedBERT-256 text encoder pretrained on a Derm1M knowledge tree (KEP), aligned on 517,455 public dermatology image–text pairs with multi-aspect knowledge contrastive learning (MAKE) and knowledge distillation. It performs zero-shot diagnosis and cross-modal retrieval, serves as a backbone for multimodal fine-tuning and VQA, and yields interpretable concepts through sparse autoencoders.

Across extensive benchmarks and three multinational reader studies, DermFM-Zero achieves state-of-the-art zero-shot performance while improving clinician decision-making in primary care and specialist settings.

Model Details

• Model Type: Pretrained Vision-Language Foundation Model (knowledge-enhanced contrastive alignment)
• Architecture:
  • Vision encoder (PanDerm-Large, ViT-L/16): initialised from the publicly released PanDerm checkpoint and extended with NaViT-style patch-and-pack so it accepts native-resolution input
  • Text encoder: PubMedBERT-256, a biomedical-domain encoder with an extended token window for detailed clinical descriptions. We initialize it by pretraining on Derm1M following KEP.
• Resolution: native-resolution input during training (ScaleJitter, long side ≤ 448 px); all reported results use 224 × 224
• Manuscript: Currently under review at Nature Biomedical Engineering (ID: nBME-26-0776)
• Code repository: https://github.com/SiyuanYan1/DermFM-Zero
• License: CC-BY-NC-ND 4.0 (non-commercial academic research use only)

Training Details

Pretraining corpus

517,455 dermatological image-text pairs covering 400+ skin conditions. By modality the corpus is 406,831 clinical (79%) and 110,624 dermoscopic (21%) images.

Public dermatology datasets. ISIC Archive, BCN20000, MSKCC, DermNet, Fitzpatrick17K, Derm12345, and HIBA. The 38,404 ISIC Archive images we used are listed in pretrain_image_lists/isic_image_ids.txt in this repository, so that anyone evaluating on ISIC-derived benchmarks can check for overlap.

Web and educational sources. The remainder is internet data collected from Derm1M, spanning medical literature, textbooks, clinical forums and video lectures.

Training

1. Text encoder pretraining (KEP). PubMedBERT-256 is trained with the AdaSP metric-learning loss on a Derm1M knowledge tree of 556,372 attribute texts (raw captions, ontology captions, visual-concept captions and sub-captions), following KEP. The resulting encoder initialises the trainable text tower and is also kept frozen as a knowledge encoder.
2. Vision–language alignment (MAKE + knowledge distillation). Starting from the PanDerm vision encoder and the KEP text encoder, the model is trained on the 517,455 pairs above with the MAKE multi-aspect contrastive objective (raw caption, disease aspect, concept aspect and two sub-captions per image, with fine-grained patch–subtext alignment) plus a knowledge-distillation term that anchors the trainable text tower to the frozen knowledge encoder. Images are fed at native resolution through patch-and-pack with ScaleJitter multi-scale augmentation.
   • AdamW (lr 1×10⁻⁴, weight decay 0.1), 200 warm-up steps, cosine decay; effective batch size 2,048; 15 epochs; bf16; 2 × NVIDIA H200, ~1 d 9 h.

Compute environment

• Python 3.10.13, PyTorch 2.4.1, CUDA 11.8, open_clip
• NumPy 2.2.6, SciPy 1.15.2, scikit-learn 1.6.1

Intended Uses

Primary Use Cases

• Zero-shot dermatological diagnosis across 200+ skin conditions
• Cross-modal retrieval (image ↔ clinical text)
• Few-shot / label-efficient learning via linear probing
• Multimodal fine-tuning with dermoscopy + clinical photography + patient metadata
• Automated concept discovery via sparse autoencoders (SAE-CBM)
• Artifact-aware diagnosis with concept-level intervention (ruler / pen / hair neuron suppression)

Out-of-Scope / Not Recommended

• Not for clinical deployment without further validation. All clinical evaluation was retrospective and conducted in store-and-forward teledermatology settings.
• Not optimized for in-person dermatology, full-body imaging, or histopathology.
• Skin-tone fairness within human-AI workflows has not been formally tested.
• Not for commercial use under the CC-BY-NC-ND 4.0 license.

How to Use

Installation

git clone https://github.com/SiyuanYan1/DermFM-Zero.git
cd DermFM-Zero

conda create -n dermfm-zero python=3.9.20
conda activate dermfm-zero
pip install -r requirements.txt

Quick Start: Zero-shot Classification

import open_clip
from PIL import Image
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

# Load model from the Hugging Face Hub (public)
model, _, preprocess = open_clip.create_model_and_transforms(
    'hf-hub:redlessone/DermFM-Zero', device=device
)
model.eval()

# Tokenizer
tokenizer = open_clip.get_tokenizer('hf-hub:redlessone/DermFM-Zero')

# Read example image
image = preprocess(Image.open("your_skin_image.png")).unsqueeze(0).to(device)

# Define disease labels (example: PAD-UFES-20 classes)
PAD_CLASSNAMES = [
    "nevus",
    "basal cell carcinoma",
    "actinic keratosis",
    "seborrheic keratosis",
    "squamous cell carcinoma",
    "melanoma",
]

# Build text prompts
template = lambda c: f'This is a skin image of {c}.'
text = tokenizer([template(c) for c in PAD_CLASSNAMES]).to(device)

# Inference
with torch.no_grad(), torch.autocast(device):
    image_features = model.encode_image(image)
    text_features  = model.encode_text(text)

    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features  /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

final_prediction = PAD_CLASSNAMES[torch.argmax(text_probs[0])]
print(f"This image is diagnosed as: {final_prediction}")
print("Label probabilities:", text_probs)

Multi-prompt ensembling (recommended for best results)

DermFM-Zero was evaluated with 7 prompt templates (Supplementary Table 41). Averaging text features across templates improves robustness:

PROMPT_TEMPLATES = [
    "This is a skin image of {}.",
    "A skin image of {}.",
    "An image of {}, a skin condition.",
    "{}, a skin disorder, is shown in this image.",
    "The skin lesion depicted is {}.",
    "The skin cancer in this image is {}.",
    "This image depicts {}, a type of skin cancer.",
]

def build_classifier(class_names, templates, tokenizer, model):
    """Average text features across prompt templates for each class."""
    weights = []
    for cname in class_names:
        prompts = [t.format(cname) for t in templates]
        tokens  = tokenizer(prompts).to(device)
        with torch.no_grad():
            feats = model.encode_text(tokens)
            feats = feats / feats.norm(dim=-1, keepdim=True)
            feats = feats.mean(dim=0)
            feats = feats / feats.norm()
        weights.append(feats)
    return torch.stack(weights, dim=0)  # [num_classes, dim]

classifier = build_classifier(PAD_CLASSNAMES, PROMPT_TEMPLATES, tokenizer, model)

with torch.no_grad(), torch.autocast(device):
    image_features = model.encode_image(image)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_probs = (100.0 * image_features @ classifier.T).softmax(dim=-1)

final_prediction = PAD_CLASSNAMES[torch.argmax(text_probs[0])]
print(f"This image is diagnosed as: {final_prediction}")

See examples/zero-shot-classification.ipynb in the repo for a runnable demo (includes batched evaluation on a toy dataset).

Other downstream tasks

Task	Script
Cross-modal retrieval	script/zero-shot-eval/DermFM-Zero-zs-retrieval.sh
Linear probing (few-shot)	script/linear-probe/DermFM-Zero-lp-eval.sh
Multimodal fine-tuning	script/multimodal_finetune/*.sh
SAE concept discovery	script/automated-concept-discovery/SAE-training/
Concept-level artifact intervention	script/automated-concept-discovery/ISIC-intervention/
Reader study replication	reader_studies/ (RS1 / RS2A / RS2B with real de-identified data)

Limitations

• Disease coverage: ~400 skin conditions in pretraining; rare tropical diseases, complex systemic dermatoses, and rare genetic disorders remain underrepresented.
• Source balance: Most of the corpus comes from web and literature sources rather than consecutive clinical cohorts, with curated clinical archives making up a minority of pairs. The corpus is therefore not population-representative, and performance on any specific patient population should be empirically validated before use.
• Retrospective evaluation only: All clinical validation was conducted in store-and-forward teledermatology workflows, not in live patient encounters.
• Skin-tone fairness in collaboration: Standalone fairness was characterized, but skin-tone-stratified fairness within human-AI workflows was not formally analyzed.
• Data overlap: Despite source-level separation and SSCD image-level deduplication, some images may appear in both pretraining and downstream benchmarks via published literature in our PubMed and web corpora. Image-level overlap rates against the zero-shot benchmarks range from 0.14% (SNU) to 13.51% (Daffodil) at a cosine threshold of 0.75. That threshold is permissive: above 0.85 the rates fall sharply (e.g. ISIC2020 1.57% → 0.08%), and no zero-shot benchmark retains a near-exact duplicate (≥ 0.99). Per-dataset counts and the flagged image lists are published in the code repository under data_deduplication/results/.

Ethical Considerations

• License restriction: CC-BY-NC-ND 4.0 — non-commercial academic research only. No clinical deployment.
• Reader study ethics: All three reader studies were approved by relevant institutional review boards; readers participated voluntarily under signed agreements.

Citation

If you use DermFM-Zero, please cite:

@article{yan2026dermfmzero,
  title   = {A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology},
  author  = {Yan, Siyuan and Li, Xieji and Mo, Dan and Tschandl, Philipp and Jiang, Yiwen and Wang, Zhonghua and Hu, Ming and Ju, Lie and Vico-Alonso, Cristina and Zheng, Yizhen and Liu, Jiahe and Zhou, Juexiao and Chello, Camilla and Cheung, Jen G. and Anriot, Julien and Thomas, Luc and Primiero, Clare and Tan, Gin and Ng, Aik Beng and See, Simon and Tang, Xiaoying and Ip, Albert and Liao, Xiaoyang and Bowling, Adrian and Haskett, Martin and Zhao, Shuang and Janda, Monika and Soyer, H. Peter and Mar, Victoria and Kittler, Harald and Ge, Zongyuan},
  journal = {Nature Biomedical Engineering},
  year    = {2026},
  note    = {Under review, manuscript ID nBME-26-0776}
}

Related work:

@article{yan2025multimodal,
  title   = {A multimodal vision foundation model for clinical dermatology},
  author  = {Yan, Siyuan and Yu, Zhen and Primiero, Clare and Vico-Alonso, Cristina and Wang, Zhonghua and Yang, Litao and Tschandl, Philipp and Hu, Ming and Ju, Lie and Tan, Gin and others},
  journal = {Nature Medicine},
  year    = {2025},
  pages   = {1--12}
}

@inproceedings{yan2025derm1m,
  title     = {Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology},
  author    = {Yan, Siyuan and Hu, Ming and Jiang, Yiwen and Li, Xieji and Fei, Hao and Tschandl, Philipp and Kittler, Harald and Ge, Zongyuan},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year      = {2025}
}

Contact

Siyuan Yan — Research Fellow, Monash University 📧 siyuan.yan@monash.edu