README.md

---
license: cc-by-nc-sa-4.0
language:
- en
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- Qwen/Qwen2.5-3B-Instruct
tags:
- pathology
- computational-pathology
- digital-pathology
- histopathology
- whole-slide-image
- vision-language-model
- report-generation
- synoptic-report
- case-level
- conch
- qwen2.5
datasets:
- david4real/HistGen
- histai/HISTAI-metadata
metrics:
- rouge
- meteor
- bleu
- bertscore
arxiv: 2605.30716
model-index:
- name: PathoSynVLM
  results:
  - task:
      type: image-text-to-text
      name: Case-level pathology synoptic report generation
    dataset:
      name: HISTAI case-report pairs
      type: histai/HISTAI-metadata
    metrics:
    - type: rouge
      name: ROUGE-L
      value: 0.2495
    - type: meteor
      name: METEOR
      value: 0.1988
    - type: bleu
      name: BLEU-4
      value: 0.0525
    - type: bertscore
      name: BERTScore F1
      value: 0.3018
---

# PathoSynVLM: Case-Level Pathology Synoptic Report Generation

[![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b.svg)](https://arxiv.org/abs/2605.30716)
[![Code](https://img.shields.io/badge/Code-GitHub-181717.svg)](https://github.com/AtlasAnalyticsLab/PathoSynVLM)
[![Project Page](https://img.shields.io/badge/Project-Page-0e8a9c.svg)](https://atlasanalyticslab.github.io/PathoSynVLM/)

PathoSynVLM is a token-efficient vision-language model for generating
case-level pathology synoptic reports from one or more whole-slide
images represented as precomputed CONCHv1.5 patch embeddings.

**Code and complete documentation:** [https://github.com/AtlasAnalyticsLab/PathoSynVLM](https://github.com/AtlasAnalyticsLab/PathoSynVLM)

This repository provides the trained model package. Use it with the
PathoSynVLM code repository for embedding preparation, case-level
inference, evaluation, and training.

![PathoSynVLM architecture](assets/paper_architecture.png)

## What This Repository Contains

| Path | Purpose |
|---|---|
| `llm/model.safetensors` | Merged language-model weights for the selected Stage 2 checkpoint. |
| `vlm_state.pt` | Vision-language aligner, WSI marker, and WSI index tensors. |
| `tokenizer/` | Tokenizer configuration and chat template used for inference. |
| `config.json` | PathoSynVLM architecture and inference settings. |
| `labels.json` | Input contract and generated report-field schema. |
| `best_checkpoint_summary.json` | Selected checkpoint and validation summary. |
| `model_index.json` | Machine-readable task, dataset, and metric metadata. |
| `examples/` | Example case-level input manifest. |

The paper run used `unfreeze_llm_base=true`, so the release package
includes the merged/full language-model weights under `llm/`, not only a
LoRA adapter.

## Quick Start

The release is loaded through the PathoSynVLM inference code rather than
directly through `transformers.AutoModel`. A CUDA-capable GPU is
recommended for normal use; CPU execution is intended for smoke tests.

### 1. Install the code

```bash
git clone https://github.com/AtlasAnalyticsLab/PathoSynVLM PathoSynVLM
cd PathoSynVLM
conda create -n pathosynvlm python=3.11 -y
conda activate pathosynvlm
export PYTHONNOUSERSITE=1
pip install -e .
```

### 2. Download the model

```bash
source configs/paths.example.env
hf download AtlasAnalyticsLab/PathoSynVLM \
  --local-dir "$PATHOSYNVLM_WEIGHTS_ROOT/pathosynvlm-stage2-main"
```

### 3. Generate a report for one case

Pass every WSI embedding file belonging to the case in the desired slide
order:

```bash
# Optional when the paths below are relative.
export PATHOSYNVLM_EMBEDDINGS_ROOT=/path/to/conch_v15/embeddings

python scripts/generate_case_report.py \
  --embeddings case_001/slide_1.h5 case_001/slide_2.h5 \
  --output_json report.json
```

Relative `--embeddings` paths are resolved under
`PATHOSYNVLM_EMBEDDINGS_ROOT`; absolute `.h5` paths work without setting
that variable. The JSON output records the generated report, resolved
slide paths, per-WSI patch counts, and feature key.

Generated report text follows:

```text
Diagnosis: ...
Certainty: ...
Conclusion: ...
```

## Input Format

PathoSynVLM runs on precomputed WSI patch embeddings, not raw WSIs. Each
`.h5` file should contain:

```text
/features/conch_v15  # shape: (num_patches, 768)
```

See the GitHub [embedding guide](https://github.com/AtlasAnalyticsLab/PathoSynVLM/blob/main/docs/embeddings.md)
for patch extraction, feature generation, H5 validation, and configurable
storage paths. Dataset placement and access requirements are documented in
the [data guide](https://github.com/AtlasAnalyticsLab/PathoSynVLM/blob/main/docs/data.md).

### From Precomputed H5 Feature Files

This is the fastest path. Put one or more WSI embedding files for a case
into the `--embeddings` argument:

```bash
python scripts/generate_case_report.py \
  --embeddings case_slide_1.h5 case_slide_2.h5 \
  --output_json report.json
```

### From Raw Whole-Slide Images

First extract tissue patches and CONCHv1.5 patch embeddings using a WSI
preprocessing pipeline that writes the H5 layout above. Then pass the
resulting H5 files to `scripts/generate_case_report.py`. PathoSynVLM does
not send raw WSI pixels directly to the language model.

## Running the Paper Pipeline

Follow the GitHub [paper pipeline](https://github.com/AtlasAnalyticsLab/PathoSynVLM/blob/main/docs/paper_pipeline.md)
for the complete sequence: dataset setup, CONCHv1.5 embedding generation,
metadata preparation, Stage 1 alignment, Stage 2 case-level fine-tuning,
and evaluation. Machine-readable paper configurations and reported values
are maintained in the same code repository.

## Training Data

The released Stage 2 checkpoint was fine-tuned on case-report pairs from
[HISTAI](https://huggingface.co/datasets/histai/HISTAI-metadata). The
official metadata repository is the starting point for dataset access and
links to the organ-specific WSI repositories used by HISTAI. See the
[HISTAI source documentation](https://github.com/HistAI/HISTAI) for the
dataset structure, subsets, citation, and access instructions.

HISTAI data remain subject to the dataset's CC BY-NC 4.0 license and
current access requirements.

## Training Recipe

- Stage 1: train the two-layer MLP aligner on HistGen + REG2025 while
  keeping the CONCHv1.5 patch encoder and LLM frozen.
- Stage 2: fine-tune on HISTAI case-report pairs with WSI marker tokens.

Checkpoint selected for release:

- checkpoint step: `30400`
- checkpoint epoch: `7`
- validation loss: `1.010892`
- prompt style: `double`
- patch level: `5x_512`
- max vision tokens: `4096`

## Reported Metrics

Stage 1 aligner-only training:

| ROUGE-L | METEOR | BLEU-4 | BERTScore F1 |
|---:|---:|---:|---:|
| 0.4743 | 0.4810 | 0.1247 | 0.4253 |

Stage 2 HISTAI main result:

| ROUGE-L | METEOR | BLEU-4 | BERTScore F1 | Diagnosis Exact | Diagnosis Relaxed | Certainty |
|---:|---:|---:|---:|---:|---:|---:|
| 0.2495 | 0.1988 | 0.0525 | 0.3018 | 0.1667 | 0.3333 | 0.9000 |

## Intended Use

This model is intended for research on pathology report generation from
precomputed WSI patch embeddings.

It is not a clinical diagnostic device and should not be used for patient
care without appropriate validation, regulatory review, and expert
oversight.

## License and Commercial Use

This repository uses CC BY-NC-SA 4.0. Research and non-commercial use only.
Dataset access, pretrained third-party models, and any externally hosted
model weights remain subject to their own terms.

## Citation

```bibtex
@inproceedings{yang2026simpletokenvlm,
  title     = {Simple Token-Efficient Vision-Language Model for Case-Level Pathology Synoptic Report Generation},
  author    = {Yang, Zhiyuan and Cheng, Jiahao and Trinh, Vincent Quoc-Huy and Hosseini, Mahdi S.},
  booktitle = {Proceedings of the 7th International Conference on Deep Learning Theory and Applications},
  pages     = {514--537},
  year      = {2026},
  issn      = {2184-9277}
}
```