Image Segmentation
PyTorch
sam2
semantic-segmentation
microstructure
materials-science
segment-anything-2
lora
Instructions to use WUT-AI-AI4Mat/microstructure-sam2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sam2
How to use WUT-AI-AI4Mat/microstructure-sam2 with sam2:
# Use SAM2 with images import torch from sam2.sam2_image_predictor import SAM2ImagePredictor predictor = SAM2ImagePredictor.from_pretrained(WUT-AI-AI4Mat/microstructure-sam2) with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): predictor.set_image(<your_image>) masks, _, _ = predictor.predict(<input_prompts>)# Use SAM2 with videos import torch from sam2.sam2_video_predictor import SAM2VideoPredictor predictor = SAM2VideoPredictor.from_pretrained(WUT-AI-AI4Mat/microstructure-sam2) with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): state = predictor.init_state(<your_video>) # add new prompts and instantly get the output on the same frame frame_idx, object_ids, masks = predictor.add_new_points(state, <your_prompts>): # propagate the prompts to get masklets throughout the video for frame_idx, object_ids, masks in predictor.propagate_in_video(state): ... - Notebooks
- Google Colab
- Kaggle
Upload model card and dataset checkpoints
Browse files- CHECKSUMS.sha256 +7 -0
- LICENSE +21 -0
- README.md +101 -0
- checkpoint_manifest.json +77 -0
- checkpoints/Aachen-Heerlen/sam2_lora_decoder_best.pth +3 -0
- checkpoints/EBC/semantic_sam2_best.pth +3 -0
- checkpoints/EMPS/sam2_lora_decoder_best.pth +3 -0
- checkpoints/Grain/sam2_lora_decoder_best.pth +3 -0
- checkpoints/MetalDAM/semantic_sam2_best.pth +3 -0
- checkpoints/Super/semantic_sam2_best.pth +3 -0
- checkpoints/UHCS/semantic_sam2_best.pth +3 -0
CHECKSUMS.sha256
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
da7ffbb7dcf2fa0ab7278fb88ed3654e90d1cef2fcba00b18b977ae4c5ced313 checkpoints/Aachen-Heerlen/sam2_lora_decoder_best.pth
|
| 2 |
+
33a37f95eb3d50485038da3e0dc8494f469bd11c5bd4e3c09ccd10956f150524 checkpoints/EBC/semantic_sam2_best.pth
|
| 3 |
+
3551a19ecc984fb999cdb7b1475ec258776e0dfdebdd4caedb91ad102df6eb39 checkpoints/EMPS/sam2_lora_decoder_best.pth
|
| 4 |
+
7bb0e3036b685eddb1c9b6d4ce3f62c3ce33d4cad971f9d6ae4913601566d37e checkpoints/Grain/sam2_lora_decoder_best.pth
|
| 5 |
+
5031b5040d1f05d65edd4b53ef8cddd135d7848ff57bab7db3ea2b6a90de7f0b checkpoints/MetalDAM/semantic_sam2_best.pth
|
| 6 |
+
d9b8fdc275efa1c58c40287777a7d2c72c0c4c9a66f71439fb946c4acfacac1d checkpoints/Super/semantic_sam2_best.pth
|
| 7 |
+
9188cfc3d8ffe9aeefa26b65129223726d2fa9742cd343b94578291e260bdd64 checkpoints/UHCS/semantic_sam2_best.pth
|
LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 The Material Image Segmentation Benchmark Authors
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
README.md
ADDED
|
@@ -0,0 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
pipeline_tag: image-segmentation
|
| 5 |
+
tags:
|
| 6 |
+
- semantic-segmentation
|
| 7 |
+
- microstructure
|
| 8 |
+
- materials-science
|
| 9 |
+
- segment-anything-2
|
| 10 |
+
- lora
|
| 11 |
+
- sam2
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# SAM2 Fine-Tuned Checkpoints for Microstructure Segmentation
|
| 15 |
+
|
| 16 |
+
## Model Description
|
| 17 |
+
|
| 18 |
+
This repository contains dataset-specific fine-tuned parameter checkpoints for
|
| 19 |
+
SAM2.1 Hiera Base+. The files are partial checkpoints and must be loaded with
|
| 20 |
+
the exact `sam2.1_hiera_base_plus.pt` base checkpoint and
|
| 21 |
+
`configs/sam2.1/sam2.1_hiera_b+.yaml`.
|
| 22 |
+
|
| 23 |
+
Source code:
|
| 24 |
+
https://github.com/WUT-AI-AI4Mat/microstructure-segmentation-benchmark
|
| 25 |
+
|
| 26 |
+
## Fine-Tuning Routes and Checkpoints
|
| 27 |
+
|
| 28 |
+
| Dataset | Route | Classes | File |
|
| 29 |
+
| --- | --- | ---: | --- |
|
| 30 |
+
| Aachen-Heerlen | LoRA plus mask decoder | 2 | `checkpoints/Aachen-Heerlen/sam2_lora_decoder_best.pth` |
|
| 31 |
+
| EMPS | LoRA plus mask decoder | 2 | `checkpoints/EMPS/sam2_lora_decoder_best.pth` |
|
| 32 |
+
| Grain | LoRA plus mask decoder | 2 | `checkpoints/Grain/sam2_lora_decoder_best.pth` |
|
| 33 |
+
| EBC | LoRA, mask decoder, and class tokens | 3 | `checkpoints/EBC/semantic_sam2_best.pth` |
|
| 34 |
+
| Super | LoRA, mask decoder, and class tokens | 3 | `checkpoints/Super/semantic_sam2_best.pth` |
|
| 35 |
+
| MetalDAM | LoRA, mask decoder, and class tokens | 5 | `checkpoints/MetalDAM/semantic_sam2_best.pth` |
|
| 36 |
+
| UHCS | LoRA, mask decoder, and class tokens | 7 | `checkpoints/UHCS/semantic_sam2_best.pth` |
|
| 37 |
+
|
| 38 |
+
Class counts include background.
|
| 39 |
+
|
| 40 |
+
## Base Model Requirement
|
| 41 |
+
|
| 42 |
+
Download the official SAM2.1 Hiera Base+ checkpoint from:
|
| 43 |
+
https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_base_plus.pt
|
| 44 |
+
|
| 45 |
+
Do not substitute another SAM2 architecture without changing the model config.
|
| 46 |
+
|
| 47 |
+
## Training Details
|
| 48 |
+
|
| 49 |
+
- Base architecture: SAM2.1 Hiera Base+
|
| 50 |
+
- Input preprocessing: longest side resized to 1024, padded to 1024 x 1024
|
| 51 |
+
- Training batch size: 1
|
| 52 |
+
- Epochs: 200
|
| 53 |
+
- Optimizer: AdamW
|
| 54 |
+
- Learning rate: 0.0001
|
| 55 |
+
- Weight decay: 0.0001
|
| 56 |
+
- Scheduler: CosineAnnealingLR with minimum learning rate 0.000001
|
| 57 |
+
- Early-stopping patience: 50
|
| 58 |
+
- LoRA rank: 8
|
| 59 |
+
- LoRA alpha: 16
|
| 60 |
+
- LoRA dropout: 0.05
|
| 61 |
+
- LoRA targets: `qkv` and `proj`
|
| 62 |
+
- Binary objective: BCE, Dice, and IoU MSE
|
| 63 |
+
- Multiclass objective: cross-entropy and Dice
|
| 64 |
+
|
| 65 |
+
## Usage
|
| 66 |
+
|
| 67 |
+
Example for binary EMPS segmentation:
|
| 68 |
+
|
| 69 |
+
```bash
|
| 70 |
+
hf download NAMESPACE/microstructure-sam2 checkpoints/EMPS/sam2_lora_decoder_best.pth --local-dir weights/sam2
|
| 71 |
+
python sam2/test_lora_decoder.py \
|
| 72 |
+
--dataset-root /path/to/EMPS \
|
| 73 |
+
--checkpoint weights/pretrained/sam2.1_hiera_base_plus.pt \
|
| 74 |
+
--finetuned-checkpoint weights/sam2/checkpoints/EMPS/sam2_lora_decoder_best.pth \
|
| 75 |
+
--output-dir results/sam2_emps
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
For multiclass checkpoints, use `sam2/test_semantic_sam2.py` and pass the
|
| 79 |
+
class count from the table.
|
| 80 |
+
|
| 81 |
+
## Inference Parameters
|
| 82 |
+
|
| 83 |
+
Automatic mask generation uses `points_per_side=32`, `points_per_batch=64`,
|
| 84 |
+
`pred_iou_thresh=0.8`, `stability_score_thresh=0.8`, and `crop_n_layers=0`.
|
| 85 |
+
Padded regions are removed before predictions are restored to the original
|
| 86 |
+
image size.
|
| 87 |
+
|
| 88 |
+
## Evaluation, Intended Use, and Limitations
|
| 89 |
+
|
| 90 |
+
Evaluation saves original-resolution masks and calculates the common benchmark
|
| 91 |
+
metrics through `Myutils/metrics.py`. These checkpoints require the exact
|
| 92 |
+
SAM2.1 Hiera Base+ base model and matching source code. They are intended for
|
| 93 |
+
research reproduction and may not generalize to unseen materials or imaging
|
| 94 |
+
conditions. They are not validated for safety-critical or industrial
|
| 95 |
+
quality-control decisions.
|
| 96 |
+
|
| 97 |
+
## License and Citation
|
| 98 |
+
|
| 99 |
+
The released experiment files are provided under the MIT license. SAM2 and its
|
| 100 |
+
base checkpoint remain subject to Meta's upstream terms and are not
|
| 101 |
+
redistributed here. A paper citation will be added after publication.
|
checkpoint_manifest.json
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "SAM2.1 Hiera Base+",
|
| 3 |
+
"source_code": "https://github.com/WUT-AI-AI4Mat/microstructure-segmentation-benchmark",
|
| 4 |
+
"checkpoint_count": 7,
|
| 5 |
+
"checkpoints": [
|
| 6 |
+
{
|
| 7 |
+
"dataset": "Aachen-Heerlen",
|
| 8 |
+
"task": "binary_segmentation",
|
| 9 |
+
"num_classes": 2,
|
| 10 |
+
"fine_tuning_route": "lora_and_mask_decoder",
|
| 11 |
+
"path": "checkpoints/Aachen-Heerlen/sam2_lora_decoder_best.pth",
|
| 12 |
+
"size_bytes": 19068232,
|
| 13 |
+
"sha256": "da7ffbb7dcf2fa0ab7278fb88ed3654e90d1cef2fcba00b18b977ae4c5ced313",
|
| 14 |
+
"requires_base_checkpoint": "sam2.1_hiera_base_plus.pt"
|
| 15 |
+
},
|
| 16 |
+
{
|
| 17 |
+
"dataset": "EBC",
|
| 18 |
+
"task": "multiclass_semantic_segmentation",
|
| 19 |
+
"num_classes": 3,
|
| 20 |
+
"fine_tuning_route": "lora_mask_decoder_and_class_tokens",
|
| 21 |
+
"path": "checkpoints/EBC/semantic_sam2_best.pth",
|
| 22 |
+
"size_bytes": 19072973,
|
| 23 |
+
"sha256": "33a37f95eb3d50485038da3e0dc8494f469bd11c5bd4e3c09ccd10956f150524",
|
| 24 |
+
"requires_base_checkpoint": "sam2.1_hiera_base_plus.pt"
|
| 25 |
+
},
|
| 26 |
+
{
|
| 27 |
+
"dataset": "EMPS",
|
| 28 |
+
"task": "binary_segmentation",
|
| 29 |
+
"num_classes": 2,
|
| 30 |
+
"fine_tuning_route": "lora_and_mask_decoder",
|
| 31 |
+
"path": "checkpoints/EMPS/sam2_lora_decoder_best.pth",
|
| 32 |
+
"size_bytes": 19068232,
|
| 33 |
+
"sha256": "3551a19ecc984fb999cdb7b1475ec258776e0dfdebdd4caedb91ad102df6eb39",
|
| 34 |
+
"requires_base_checkpoint": "sam2.1_hiera_base_plus.pt"
|
| 35 |
+
},
|
| 36 |
+
{
|
| 37 |
+
"dataset": "Grain",
|
| 38 |
+
"task": "binary_segmentation",
|
| 39 |
+
"num_classes": 2,
|
| 40 |
+
"fine_tuning_route": "lora_and_mask_decoder",
|
| 41 |
+
"path": "checkpoints/Grain/sam2_lora_decoder_best.pth",
|
| 42 |
+
"size_bytes": 19068232,
|
| 43 |
+
"sha256": "7bb0e3036b685eddb1c9b6d4ce3f62c3ce33d4cad971f9d6ae4913601566d37e",
|
| 44 |
+
"requires_base_checkpoint": "sam2.1_hiera_base_plus.pt"
|
| 45 |
+
},
|
| 46 |
+
{
|
| 47 |
+
"dataset": "MetalDAM",
|
| 48 |
+
"task": "multiclass_semantic_segmentation",
|
| 49 |
+
"num_classes": 5,
|
| 50 |
+
"fine_tuning_route": "lora_mask_decoder_and_class_tokens",
|
| 51 |
+
"path": "checkpoints/MetalDAM/semantic_sam2_best.pth",
|
| 52 |
+
"size_bytes": 19075021,
|
| 53 |
+
"sha256": "5031b5040d1f05d65edd4b53ef8cddd135d7848ff57bab7db3ea2b6a90de7f0b",
|
| 54 |
+
"requires_base_checkpoint": "sam2.1_hiera_base_plus.pt"
|
| 55 |
+
},
|
| 56 |
+
{
|
| 57 |
+
"dataset": "Super",
|
| 58 |
+
"task": "multiclass_semantic_segmentation",
|
| 59 |
+
"num_classes": 3,
|
| 60 |
+
"fine_tuning_route": "lora_mask_decoder_and_class_tokens",
|
| 61 |
+
"path": "checkpoints/Super/semantic_sam2_best.pth",
|
| 62 |
+
"size_bytes": 19072973,
|
| 63 |
+
"sha256": "d9b8fdc275efa1c58c40287777a7d2c72c0c4c9a66f71439fb946c4acfacac1d",
|
| 64 |
+
"requires_base_checkpoint": "sam2.1_hiera_base_plus.pt"
|
| 65 |
+
},
|
| 66 |
+
{
|
| 67 |
+
"dataset": "UHCS",
|
| 68 |
+
"task": "multiclass_semantic_segmentation",
|
| 69 |
+
"num_classes": 7,
|
| 70 |
+
"fine_tuning_route": "lora_mask_decoder_and_class_tokens",
|
| 71 |
+
"path": "checkpoints/UHCS/semantic_sam2_best.pth",
|
| 72 |
+
"size_bytes": 19077069,
|
| 73 |
+
"sha256": "9188cfc3d8ffe9aeefa26b65129223726d2fa9742cd343b94578291e260bdd64",
|
| 74 |
+
"requires_base_checkpoint": "sam2.1_hiera_base_plus.pt"
|
| 75 |
+
}
|
| 76 |
+
]
|
| 77 |
+
}
|
checkpoints/Aachen-Heerlen/sam2_lora_decoder_best.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:da7ffbb7dcf2fa0ab7278fb88ed3654e90d1cef2fcba00b18b977ae4c5ced313
|
| 3 |
+
size 19068232
|
checkpoints/EBC/semantic_sam2_best.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:33a37f95eb3d50485038da3e0dc8494f469bd11c5bd4e3c09ccd10956f150524
|
| 3 |
+
size 19072973
|
checkpoints/EMPS/sam2_lora_decoder_best.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3551a19ecc984fb999cdb7b1475ec258776e0dfdebdd4caedb91ad102df6eb39
|
| 3 |
+
size 19068232
|
checkpoints/Grain/sam2_lora_decoder_best.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7bb0e3036b685eddb1c9b6d4ce3f62c3ce33d4cad971f9d6ae4913601566d37e
|
| 3 |
+
size 19068232
|
checkpoints/MetalDAM/semantic_sam2_best.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5031b5040d1f05d65edd4b53ef8cddd135d7848ff57bab7db3ea2b6a90de7f0b
|
| 3 |
+
size 19075021
|
checkpoints/Super/semantic_sam2_best.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d9b8fdc275efa1c58c40287777a7d2c72c0c4c9a66f71439fb946c4acfacac1d
|
| 3 |
+
size 19072973
|
checkpoints/UHCS/semantic_sam2_best.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9188cfc3d8ffe9aeefa26b65129223726d2fa9742cd343b94578291e260bdd64
|
| 3 |
+
size 19077069
|