Jiahao Cheng commited on
Commit
aa0ba11
·
1 Parent(s): aa50ff8

docs: expand model usage guide

Browse files

Make the GitHub code link prominent and document the model package, embedding contract, case-level inference workflow, outputs, and paper pipeline.

Files changed (1) hide show
  1. README.md +52 -49
README.md CHANGED
@@ -52,45 +52,46 @@ model-index:
52
 
53
  # PathoSynVLM: Case-Level Pathology Synoptic Report Generation
54
 
 
 
 
 
55
  PathoSynVLM is a token-efficient vision-language model for generating
56
  case-level pathology synoptic reports from one or more whole-slide
57
  images represented as precomputed CONCHv1.5 patch embeddings.
58
 
59
- Paper: [Simple Token-Efficient Vision-Language Model for Case-level
60
- Pathology Synoptic Report Generation](https://arxiv.org/abs/2605.30716)
61
-
62
- Code: [https://github.com/AtlasAnalyticsLab/PathoSynVLM](https://github.com/AtlasAnalyticsLab/PathoSynVLM)
63
 
64
- Model: [https://huggingface.co/AtlasAnalyticsLab/PathoSynVLM](https://huggingface.co/AtlasAnalyticsLab/PathoSynVLM)
 
 
65
 
66
  ![PathoSynVLM architecture](assets/paper_architecture.png)
67
 
68
  ## What This Repository Contains
69
 
70
- This Hugging Face repository contains the exported Stage 2 PathoSynVLM
71
- model package:
72
-
73
- ```text
74
- README.md
75
- LICENSE
76
- model_index.json
77
- config.json
78
- vlm_state.pt
79
- labels.json
80
- tokenizer/
81
- llm/
82
- best_checkpoint_summary.json
83
- assets/
84
- examples/
85
- ```
86
 
87
  The paper run used `unfreeze_llm_base=true`, so the release package
88
  includes the merged/full language-model weights under `llm/`, not only a
89
  LoRA adapter.
90
 
91
- ## Quickstart
 
 
 
 
92
 
93
- Install the code repository:
94
 
95
  ```bash
96
  git clone https://github.com/AtlasAnalyticsLab/PathoSynVLM PathoSynVLM
@@ -101,25 +102,34 @@ export PYTHONNOUSERSITE=1
101
  pip install -e .
102
  ```
103
 
104
- Download this model repository:
105
 
106
  ```bash
107
  source configs/paths.example.env
108
- hf download AtlasAnalyticsLab/PathoSynVLM --local-dir "$PATHOSYNVLM_WEIGHTS_ROOT/pathosynvlm-stage2-main"
 
109
  ```
110
 
111
- Generate a case-level report from one or more slide embedding files:
 
 
 
112
 
113
  ```bash
 
 
 
114
  python scripts/generate_case_report.py \
115
- --embeddings slide_1.h5 slide_2.h5 \
116
  --output_json report.json
117
  ```
118
 
119
  Relative `--embeddings` paths are resolved under
120
- `PATHOSYNVLM_EMBEDDINGS_ROOT`. Absolute `.h5` paths also work.
 
 
121
 
122
- The output follows:
123
 
124
  ```text
125
  Diagnosis: ...
@@ -136,8 +146,10 @@ PathoSynVLM runs on precomputed WSI patch embeddings, not raw WSIs. Each
136
  /features/conch_v15 # shape: (num_patches, 768)
137
  ```
138
 
139
- Use the code repository docs for dataset placement, metadata filtering,
140
- and embedding generation details.
 
 
141
 
142
  ### From Precomputed H5 Feature Files
143
 
@@ -154,31 +166,22 @@ python scripts/generate_case_report.py \
154
 
155
  First extract tissue patches and CONCHv1.5 patch embeddings using a WSI
156
  preprocessing pipeline that writes the H5 layout above. Then pass the
157
- resulting H5 files to `scripts/generate_case_report.py`.
 
158
 
159
- ## Running The Paper Pipeline
160
 
161
- To rerun training and evaluation, follow the code repository guide:
162
-
163
- ```bash
164
- git clone https://github.com/AtlasAnalyticsLab/PathoSynVLM PathoSynVLM
165
- cd PathoSynVLM
166
- export PYTHONNOUSERSITE=1
167
- pip install -e .
168
-
169
- # 1. Download HistGen, REG2025, and HISTAI metadata/WSIs.
170
- # 2. Extract CONCHv1.5 H5 patch embeddings.
171
- # 3. Prepare Stage 1 and Stage 2 metadata.
172
- # 4. Train Stage 1 alignment.
173
- # 5. Train Stage 2 HISTAI report generation.
174
- # 6. Evaluate with scripts/evaluate_checkpoint.py.
175
- ```
176
 
177
  ## Training Recipe
178
 
179
  - Stage 1: train the two-layer MLP aligner on HistGen + REG2025 while
180
  keeping the CONCHv1.5 patch encoder and LLM frozen.
181
- - Stage 2: finetune on HISTAI case-report pairs with WSI marker tokens.
182
 
183
  Checkpoint selected for release:
184
 
 
52
 
53
  # PathoSynVLM: Case-Level Pathology Synoptic Report Generation
54
 
55
+ [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b.svg)](https://arxiv.org/abs/2605.30716)
56
+ [![Code](https://img.shields.io/badge/Code-GitHub-181717.svg)](https://github.com/AtlasAnalyticsLab/PathoSynVLM)
57
+ [![Project Page](https://img.shields.io/badge/Project-Page-0e8a9c.svg)](https://atlasanalyticslab.github.io/PathoSynVLM/)
58
+
59
  PathoSynVLM is a token-efficient vision-language model for generating
60
  case-level pathology synoptic reports from one or more whole-slide
61
  images represented as precomputed CONCHv1.5 patch embeddings.
62
 
63
+ **Code and complete documentation:** [https://github.com/AtlasAnalyticsLab/PathoSynVLM](https://github.com/AtlasAnalyticsLab/PathoSynVLM)
 
 
 
64
 
65
+ This repository provides the trained model package. Use it with the
66
+ PathoSynVLM code repository for embedding preparation, case-level
67
+ inference, evaluation, and training.
68
 
69
  ![PathoSynVLM architecture](assets/paper_architecture.png)
70
 
71
  ## What This Repository Contains
72
 
73
+ | Path | Purpose |
74
+ |---|---|
75
+ | `llm/model.safetensors` | Merged language-model weights for the selected Stage 2 checkpoint. |
76
+ | `vlm_state.pt` | Vision-language aligner, WSI marker, and WSI index tensors. |
77
+ | `tokenizer/` | Tokenizer configuration and chat template used for inference. |
78
+ | `config.json` | PathoSynVLM architecture and inference settings. |
79
+ | `labels.json` | Input contract and generated report-field schema. |
80
+ | `best_checkpoint_summary.json` | Selected checkpoint and validation summary. |
81
+ | `model_index.json` | Machine-readable task, dataset, and metric metadata. |
82
+ | `examples/` | Example case-level input manifest. |
 
 
 
 
 
 
83
 
84
  The paper run used `unfreeze_llm_base=true`, so the release package
85
  includes the merged/full language-model weights under `llm/`, not only a
86
  LoRA adapter.
87
 
88
+ ## Quick Start
89
+
90
+ The release is loaded through the PathoSynVLM inference code rather than
91
+ directly through `transformers.AutoModel`. A CUDA-capable GPU is
92
+ recommended for normal use; CPU execution is intended for smoke tests.
93
 
94
+ ### 1. Install the code
95
 
96
  ```bash
97
  git clone https://github.com/AtlasAnalyticsLab/PathoSynVLM PathoSynVLM
 
102
  pip install -e .
103
  ```
104
 
105
+ ### 2. Download the model
106
 
107
  ```bash
108
  source configs/paths.example.env
109
+ hf download AtlasAnalyticsLab/PathoSynVLM \
110
+ --local-dir "$PATHOSYNVLM_WEIGHTS_ROOT/pathosynvlm-stage2-main"
111
  ```
112
 
113
+ ### 3. Generate a report for one case
114
+
115
+ Pass every WSI embedding file belonging to the case in the desired slide
116
+ order:
117
 
118
  ```bash
119
+ # Optional when the paths below are relative.
120
+ export PATHOSYNVLM_EMBEDDINGS_ROOT=/path/to/conch_v15/embeddings
121
+
122
  python scripts/generate_case_report.py \
123
+ --embeddings case_001/slide_1.h5 case_001/slide_2.h5 \
124
  --output_json report.json
125
  ```
126
 
127
  Relative `--embeddings` paths are resolved under
128
+ `PATHOSYNVLM_EMBEDDINGS_ROOT`; absolute `.h5` paths work without setting
129
+ that variable. The JSON output records the generated report, resolved
130
+ slide paths, per-WSI patch counts, and feature key.
131
 
132
+ Generated report text follows:
133
 
134
  ```text
135
  Diagnosis: ...
 
146
  /features/conch_v15 # shape: (num_patches, 768)
147
  ```
148
 
149
+ See the GitHub [embedding guide](https://github.com/AtlasAnalyticsLab/PathoSynVLM/blob/main/docs/embeddings.md)
150
+ for patch extraction, feature generation, H5 validation, and configurable
151
+ storage paths. Dataset placement and access requirements are documented in
152
+ the [data guide](https://github.com/AtlasAnalyticsLab/PathoSynVLM/blob/main/docs/data.md).
153
 
154
  ### From Precomputed H5 Feature Files
155
 
 
166
 
167
  First extract tissue patches and CONCHv1.5 patch embeddings using a WSI
168
  preprocessing pipeline that writes the H5 layout above. Then pass the
169
+ resulting H5 files to `scripts/generate_case_report.py`. PathoSynVLM does
170
+ not send raw WSI pixels directly to the language model.
171
 
172
+ ## Running the Paper Pipeline
173
 
174
+ Follow the GitHub [paper pipeline](https://github.com/AtlasAnalyticsLab/PathoSynVLM/blob/main/docs/paper_pipeline.md)
175
+ for the complete sequence: dataset setup, CONCHv1.5 embedding generation,
176
+ metadata preparation, Stage 1 alignment, Stage 2 case-level fine-tuning,
177
+ and evaluation. Machine-readable paper configurations and reported values
178
+ are maintained in the same code repository.
 
 
 
 
 
 
 
 
 
 
179
 
180
  ## Training Recipe
181
 
182
  - Stage 1: train the two-layer MLP aligner on HistGen + REG2025 while
183
  keeping the CONCHv1.5 patch encoder and LLM frozen.
184
+ - Stage 2: fine-tune on HISTAI case-report pairs with WSI marker tokens.
185
 
186
  Checkpoint selected for release:
187