Image-Text-to-Text
Transformers
Safetensors
English
pathology
computational-pathology
digital-pathology
histopathology
whole-slide-image
vision-language-model
report-generation
synoptic-report
case-level
conch
qwen2.5
Eval Results (legacy)
Instructions to use AtlasAnalyticsLab/PathoSynVLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AtlasAnalyticsLab/PathoSynVLM with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="AtlasAnalyticsLab/PathoSynVLM")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("AtlasAnalyticsLab/PathoSynVLM", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use AtlasAnalyticsLab/PathoSynVLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AtlasAnalyticsLab/PathoSynVLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AtlasAnalyticsLab/PathoSynVLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/AtlasAnalyticsLab/PathoSynVLM
- SGLang
How to use AtlasAnalyticsLab/PathoSynVLM with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "AtlasAnalyticsLab/PathoSynVLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AtlasAnalyticsLab/PathoSynVLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "AtlasAnalyticsLab/PathoSynVLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AtlasAnalyticsLab/PathoSynVLM", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use AtlasAnalyticsLab/PathoSynVLM with Docker Model Runner:
docker model run hf.co/AtlasAnalyticsLab/PathoSynVLM
Jiahao Cheng commited on
Commit ·
aa0ba11
1
Parent(s): aa50ff8
docs: expand model usage guide
Browse filesMake the GitHub code link prominent and document the model package, embedding contract, case-level inference workflow, outputs, and paper pipeline.
README.md
CHANGED
|
@@ -52,45 +52,46 @@ model-index:
|
|
| 52 |
|
| 53 |
# PathoSynVLM: Case-Level Pathology Synoptic Report Generation
|
| 54 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
PathoSynVLM is a token-efficient vision-language model for generating
|
| 56 |
case-level pathology synoptic reports from one or more whole-slide
|
| 57 |
images represented as precomputed CONCHv1.5 patch embeddings.
|
| 58 |
|
| 59 |
-
|
| 60 |
-
Pathology Synoptic Report Generation](https://arxiv.org/abs/2605.30716)
|
| 61 |
-
|
| 62 |
-
Code: [https://github.com/AtlasAnalyticsLab/PathoSynVLM](https://github.com/AtlasAnalyticsLab/PathoSynVLM)
|
| 63 |
|
| 64 |
-
|
|
|
|
|
|
|
| 65 |
|
| 66 |

|
| 67 |
|
| 68 |
## What This Repository Contains
|
| 69 |
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
``
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
tokenizer/
|
| 81 |
-
llm/
|
| 82 |
-
best_checkpoint_summary.json
|
| 83 |
-
assets/
|
| 84 |
-
examples/
|
| 85 |
-
```
|
| 86 |
|
| 87 |
The paper run used `unfreeze_llm_base=true`, so the release package
|
| 88 |
includes the merged/full language-model weights under `llm/`, not only a
|
| 89 |
LoRA adapter.
|
| 90 |
|
| 91 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
|
| 93 |
-
Install the code
|
| 94 |
|
| 95 |
```bash
|
| 96 |
git clone https://github.com/AtlasAnalyticsLab/PathoSynVLM PathoSynVLM
|
|
@@ -101,25 +102,34 @@ export PYTHONNOUSERSITE=1
|
|
| 101 |
pip install -e .
|
| 102 |
```
|
| 103 |
|
| 104 |
-
Download
|
| 105 |
|
| 106 |
```bash
|
| 107 |
source configs/paths.example.env
|
| 108 |
-
hf download AtlasAnalyticsLab/PathoSynVLM
|
|
|
|
| 109 |
```
|
| 110 |
|
| 111 |
-
Generate a
|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
```bash
|
|
|
|
|
|
|
|
|
|
| 114 |
python scripts/generate_case_report.py \
|
| 115 |
-
--embeddings slide_1.h5 slide_2.h5 \
|
| 116 |
--output_json report.json
|
| 117 |
```
|
| 118 |
|
| 119 |
Relative `--embeddings` paths are resolved under
|
| 120 |
-
`PATHOSYNVLM_EMBEDDINGS_ROOT`
|
|
|
|
|
|
|
| 121 |
|
| 122 |
-
|
| 123 |
|
| 124 |
```text
|
| 125 |
Diagnosis: ...
|
|
@@ -136,8 +146,10 @@ PathoSynVLM runs on precomputed WSI patch embeddings, not raw WSIs. Each
|
|
| 136 |
/features/conch_v15 # shape: (num_patches, 768)
|
| 137 |
```
|
| 138 |
|
| 139 |
-
|
| 140 |
-
|
|
|
|
|
|
|
| 141 |
|
| 142 |
### From Precomputed H5 Feature Files
|
| 143 |
|
|
@@ -154,31 +166,22 @@ python scripts/generate_case_report.py \
|
|
| 154 |
|
| 155 |
First extract tissue patches and CONCHv1.5 patch embeddings using a WSI
|
| 156 |
preprocessing pipeline that writes the H5 layout above. Then pass the
|
| 157 |
-
resulting H5 files to `scripts/generate_case_report.py`.
|
|
|
|
| 158 |
|
| 159 |
-
## Running
|
| 160 |
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
export PYTHONNOUSERSITE=1
|
| 167 |
-
pip install -e .
|
| 168 |
-
|
| 169 |
-
# 1. Download HistGen, REG2025, and HISTAI metadata/WSIs.
|
| 170 |
-
# 2. Extract CONCHv1.5 H5 patch embeddings.
|
| 171 |
-
# 3. Prepare Stage 1 and Stage 2 metadata.
|
| 172 |
-
# 4. Train Stage 1 alignment.
|
| 173 |
-
# 5. Train Stage 2 HISTAI report generation.
|
| 174 |
-
# 6. Evaluate with scripts/evaluate_checkpoint.py.
|
| 175 |
-
```
|
| 176 |
|
| 177 |
## Training Recipe
|
| 178 |
|
| 179 |
- Stage 1: train the two-layer MLP aligner on HistGen + REG2025 while
|
| 180 |
keeping the CONCHv1.5 patch encoder and LLM frozen.
|
| 181 |
-
- Stage 2:
|
| 182 |
|
| 183 |
Checkpoint selected for release:
|
| 184 |
|
|
|
|
| 52 |
|
| 53 |
# PathoSynVLM: Case-Level Pathology Synoptic Report Generation
|
| 54 |
|
| 55 |
+
[](https://arxiv.org/abs/2605.30716)
|
| 56 |
+
[](https://github.com/AtlasAnalyticsLab/PathoSynVLM)
|
| 57 |
+
[](https://atlasanalyticslab.github.io/PathoSynVLM/)
|
| 58 |
+
|
| 59 |
PathoSynVLM is a token-efficient vision-language model for generating
|
| 60 |
case-level pathology synoptic reports from one or more whole-slide
|
| 61 |
images represented as precomputed CONCHv1.5 patch embeddings.
|
| 62 |
|
| 63 |
+
**Code and complete documentation:** [https://github.com/AtlasAnalyticsLab/PathoSynVLM](https://github.com/AtlasAnalyticsLab/PathoSynVLM)
|
|
|
|
|
|
|
|
|
|
| 64 |
|
| 65 |
+
This repository provides the trained model package. Use it with the
|
| 66 |
+
PathoSynVLM code repository for embedding preparation, case-level
|
| 67 |
+
inference, evaluation, and training.
|
| 68 |
|
| 69 |

|
| 70 |
|
| 71 |
## What This Repository Contains
|
| 72 |
|
| 73 |
+
| Path | Purpose |
|
| 74 |
+
|---|---|
|
| 75 |
+
| `llm/model.safetensors` | Merged language-model weights for the selected Stage 2 checkpoint. |
|
| 76 |
+
| `vlm_state.pt` | Vision-language aligner, WSI marker, and WSI index tensors. |
|
| 77 |
+
| `tokenizer/` | Tokenizer configuration and chat template used for inference. |
|
| 78 |
+
| `config.json` | PathoSynVLM architecture and inference settings. |
|
| 79 |
+
| `labels.json` | Input contract and generated report-field schema. |
|
| 80 |
+
| `best_checkpoint_summary.json` | Selected checkpoint and validation summary. |
|
| 81 |
+
| `model_index.json` | Machine-readable task, dataset, and metric metadata. |
|
| 82 |
+
| `examples/` | Example case-level input manifest. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
|
| 84 |
The paper run used `unfreeze_llm_base=true`, so the release package
|
| 85 |
includes the merged/full language-model weights under `llm/`, not only a
|
| 86 |
LoRA adapter.
|
| 87 |
|
| 88 |
+
## Quick Start
|
| 89 |
+
|
| 90 |
+
The release is loaded through the PathoSynVLM inference code rather than
|
| 91 |
+
directly through `transformers.AutoModel`. A CUDA-capable GPU is
|
| 92 |
+
recommended for normal use; CPU execution is intended for smoke tests.
|
| 93 |
|
| 94 |
+
### 1. Install the code
|
| 95 |
|
| 96 |
```bash
|
| 97 |
git clone https://github.com/AtlasAnalyticsLab/PathoSynVLM PathoSynVLM
|
|
|
|
| 102 |
pip install -e .
|
| 103 |
```
|
| 104 |
|
| 105 |
+
### 2. Download the model
|
| 106 |
|
| 107 |
```bash
|
| 108 |
source configs/paths.example.env
|
| 109 |
+
hf download AtlasAnalyticsLab/PathoSynVLM \
|
| 110 |
+
--local-dir "$PATHOSYNVLM_WEIGHTS_ROOT/pathosynvlm-stage2-main"
|
| 111 |
```
|
| 112 |
|
| 113 |
+
### 3. Generate a report for one case
|
| 114 |
+
|
| 115 |
+
Pass every WSI embedding file belonging to the case in the desired slide
|
| 116 |
+
order:
|
| 117 |
|
| 118 |
```bash
|
| 119 |
+
# Optional when the paths below are relative.
|
| 120 |
+
export PATHOSYNVLM_EMBEDDINGS_ROOT=/path/to/conch_v15/embeddings
|
| 121 |
+
|
| 122 |
python scripts/generate_case_report.py \
|
| 123 |
+
--embeddings case_001/slide_1.h5 case_001/slide_2.h5 \
|
| 124 |
--output_json report.json
|
| 125 |
```
|
| 126 |
|
| 127 |
Relative `--embeddings` paths are resolved under
|
| 128 |
+
`PATHOSYNVLM_EMBEDDINGS_ROOT`; absolute `.h5` paths work without setting
|
| 129 |
+
that variable. The JSON output records the generated report, resolved
|
| 130 |
+
slide paths, per-WSI patch counts, and feature key.
|
| 131 |
|
| 132 |
+
Generated report text follows:
|
| 133 |
|
| 134 |
```text
|
| 135 |
Diagnosis: ...
|
|
|
|
| 146 |
/features/conch_v15 # shape: (num_patches, 768)
|
| 147 |
```
|
| 148 |
|
| 149 |
+
See the GitHub [embedding guide](https://github.com/AtlasAnalyticsLab/PathoSynVLM/blob/main/docs/embeddings.md)
|
| 150 |
+
for patch extraction, feature generation, H5 validation, and configurable
|
| 151 |
+
storage paths. Dataset placement and access requirements are documented in
|
| 152 |
+
the [data guide](https://github.com/AtlasAnalyticsLab/PathoSynVLM/blob/main/docs/data.md).
|
| 153 |
|
| 154 |
### From Precomputed H5 Feature Files
|
| 155 |
|
|
|
|
| 166 |
|
| 167 |
First extract tissue patches and CONCHv1.5 patch embeddings using a WSI
|
| 168 |
preprocessing pipeline that writes the H5 layout above. Then pass the
|
| 169 |
+
resulting H5 files to `scripts/generate_case_report.py`. PathoSynVLM does
|
| 170 |
+
not send raw WSI pixels directly to the language model.
|
| 171 |
|
| 172 |
+
## Running the Paper Pipeline
|
| 173 |
|
| 174 |
+
Follow the GitHub [paper pipeline](https://github.com/AtlasAnalyticsLab/PathoSynVLM/blob/main/docs/paper_pipeline.md)
|
| 175 |
+
for the complete sequence: dataset setup, CONCHv1.5 embedding generation,
|
| 176 |
+
metadata preparation, Stage 1 alignment, Stage 2 case-level fine-tuning,
|
| 177 |
+
and evaluation. Machine-readable paper configurations and reported values
|
| 178 |
+
are maintained in the same code repository.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
|
| 180 |
## Training Recipe
|
| 181 |
|
| 182 |
- Stage 1: train the two-layer MLP aligner on HistGen + REG2025 while
|
| 183 |
keeping the CONCHv1.5 patch encoder and LLM frozen.
|
| 184 |
+
- Stage 2: fine-tune on HISTAI case-report pairs with WSI marker tokens.
|
| 185 |
|
| 186 |
Checkpoint selected for release:
|
| 187 |
|