Hugging Face's logo

qualcomm
/
MobileSam

Image Segmentation
PyTorch
foundation
android
Model card Files
xet
 Community
qaihm-bot commited on
Commit
03b0df1
·
verified ·
1 Parent(s): 94c0c53

v0.48.0

Browse files

See https://github.com/qualcomm/ai-hub-models/releases/v0.48.0 for changelog.

Files changed (1) hide show
  1. README.md +58 -58
README.md CHANGED
@@ -15,7 +15,7 @@ pipeline_tag: image-segmentation
15
  Transformer based encoder-decoder where prompts specify what to segment in an image thereby allowing segmentation without the need for additional training. The image encoder generates embeddings and the lightweight decoder operates on the embeddings for point and mask based image segmentation.
16
 
17
  This is based on the implementation of MobileSam found [here](https://github.com/facebookresearch/segment-anything).
18
- This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/quic/ai-hub-models/blob/main/qai_hub_models/models/mobilesam) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).
19
 
20
  Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.
21
 
@@ -28,23 +28,23 @@ Below are pre-exported model assets ready for deployment.
28
 
29
  | Runtime | Precision | Chipset | SDK Versions | Download |
30
  |---|---|---|---|---|
31
- | ONNX | float | Universal | QAIRT 2.42, ONNX Runtime 1.24.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mobilesam/releases/v0.47.0/mobilesam-onnx-float.zip)
32
- | QNN_DLC | float | Universal | QAIRT 2.43 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mobilesam/releases/v0.47.0/mobilesam-qnn_dlc-float.zip)
33
- | TFLITE | float | Universal | QAIRT 2.43, TFLite 2.17.0 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mobilesam/releases/v0.47.0/mobilesam-tflite-float.zip)
34
 
35
  For more device-specific assets and performance metrics, visit **[MobileSam on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/mobilesam)**.
36
 
37
 
38
  ### Option 2: Export with Custom Configurations
39
 
40
- Use the [Qualcomm® AI Hub Models](https://github.com/quic/ai-hub-models/blob/main/qai_hub_models/models/mobilesam) Python library to compile and export the model with your own:
41
  - Custom weights (e.g., fine-tuned checkpoints)
42
  - Custom input shapes
43
  - Target device and runtime configurations
44
 
45
  This option is ideal if you need to customize the model beyond the default configuration provided here.
46
 
47
- See our repository for [MobileSam on GitHub](https://github.com/quic/ai-hub-models/blob/main/qai_hub_models/models/mobilesam) for usage instructions.
48
 
49
  ## Model Details
50
 
@@ -61,58 +61,58 @@ See our repository for [MobileSam on GitHub](https://github.com/quic/ai-hub-mode
61
  ## Performance Summary
62
  | Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
63
  |---|---|---|---|---|---|---
64
- | MobileSAMDecoder | ONNX | float | Snapdragon® X Elite | 7.515 ms | 11 - 11 MB | NPU
65
- | MobileSAMDecoder | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 4.916 ms | 5 - 263 MB | NPU
66
- | MobileSAMDecoder | ONNX | float | Qualcomm® QCS8550 (Proxy) | 7.013 ms | 0 - 15 MB | NPU
67
- | MobileSAMDecoder | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 3.65 ms | 4 - 226 MB | NPU
68
- | MobileSAMDecoder | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.89 ms | 0 - 232 MB | NPU
69
- | MobileSAMDecoder | ONNX | float | Snapdragon® X2 Elite | 2.951 ms | 11 - 11 MB | NPU
70
- | MobileSAMDecoder | QNN_DLC | float | Snapdragon® X Elite | 5.286 ms | 4 - 4 MB | NPU
71
- | MobileSAMDecoder | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 3.36 ms | 4 - 232 MB | NPU
72
- | MobileSAMDecoder | QNN_DLC | float | Qualcomm® QCS8275 (Proxy) | 11.687 ms | 1 - 198 MB | NPU
73
- | MobileSAMDecoder | QNN_DLC | float | Qualcomm® QCS8550 (Proxy) | 4.86 ms | 4 - 36 MB | NPU
74
- | MobileSAMDecoder | QNN_DLC | float | Qualcomm® SA8775P | 5.887 ms | 0 - 200 MB | NPU
75
- | MobileSAMDecoder | QNN_DLC | float | Qualcomm® QCS8450 (Proxy) | 8.484 ms | 0 - 220 MB | NPU
76
- | MobileSAMDecoder | QNN_DLC | float | Qualcomm® SA7255P | 11.687 ms | 1 - 198 MB | NPU
77
- | MobileSAMDecoder | QNN_DLC | float | Qualcomm® SA8295P | 7.197 ms | 0 - 193 MB | NPU
78
- | MobileSAMDecoder | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 2.595 ms | 0 - 198 MB | NPU
79
- | MobileSAMDecoder | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.154 ms | 3 - 202 MB | NPU
80
- | MobileSAMDecoder | QNN_DLC | float | Snapdragon® X2 Elite | 2.662 ms | 4 - 4 MB | NPU
81
- | MobileSAMDecoder | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 3.875 ms | 0 - 235 MB | NPU
82
- | MobileSAMDecoder | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 13.024 ms | 0 - 203 MB | NPU
83
- | MobileSAMDecoder | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 5.6 ms | 0 - 26 MB | NPU
84
- | MobileSAMDecoder | TFLITE | float | Qualcomm® SA8775P | 6.7 ms | 0 - 204 MB | NPU
85
- | MobileSAMDecoder | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 9.491 ms | 0 - 225 MB | NPU
86
- | MobileSAMDecoder | TFLITE | float | Qualcomm® SA7255P | 13.024 ms | 0 - 203 MB | NPU
87
- | MobileSAMDecoder | TFLITE | float | Qualcomm® SA8295P | 8.115 ms | 0 - 201 MB | NPU
88
- | MobileSAMDecoder | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 2.955 ms | 0 - 205 MB | NPU
89
- | MobileSAMDecoder | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.353 ms | 0 - 206 MB | NPU
90
- | MobileSAMEncoder | ONNX | float | Snapdragon® X Elite | 172.158 ms | 12 - 12 MB | NPU
91
- | MobileSAMEncoder | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 125.387 ms | 20 - 3042 MB | NPU
92
- | MobileSAMEncoder | ONNX | float | Qualcomm® QCS8550 (Proxy) | 175.157 ms | 12 - 31 MB | NPU
93
- | MobileSAMEncoder | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 102.261 ms | 9 - 1390 MB | NPU
94
- | MobileSAMEncoder | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 83.472 ms | 20 - 1524 MB | NPU
95
- | MobileSAMEncoder | ONNX | float | Snapdragon® X2 Elite | 94.261 ms | 14 - 14 MB | NPU
96
- | MobileSAMEncoder | QNN_DLC | float | Snapdragon® X Elite | 100.191 ms | 12 - 12 MB | NPU
97
- | MobileSAMEncoder | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 66.146 ms | 12 - 1636 MB | NPU
98
- | MobileSAMEncoder | QNN_DLC | float | Qualcomm® QCS8275 (Proxy) | 209.193 ms | 0 - 1209 MB | NPU
99
- | MobileSAMEncoder | QNN_DLC | float | Qualcomm® QCS8550 (Proxy) | 96.518 ms | 13 - 15 MB | NPU
100
- | MobileSAMEncoder | QNN_DLC | float | Qualcomm® SA8775P | 103.049 ms | 0 - 1209 MB | NPU
101
- | MobileSAMEncoder | QNN_DLC | float | Qualcomm® QCS8450 (Proxy) | 470.295 ms | 13 - 1732 MB | NPU
102
- | MobileSAMEncoder | QNN_DLC | float | Qualcomm® SA7255P | 209.193 ms | 0 - 1209 MB | NPU
103
- | MobileSAMEncoder | QNN_DLC | float | Qualcomm® SA8295P | 474.267 ms | 0 - 1248 MB | NPU
104
- | MobileSAMEncoder | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 49.845 ms | 12 - 1217 MB | NPU
105
- | MobileSAMEncoder | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 48.339 ms | 10 - 1323 MB | NPU
106
- | MobileSAMEncoder | QNN_DLC | float | Snapdragon® X2 Elite | 51.495 ms | 12 - 12 MB | NPU
107
- | MobileSAMEncoder | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 127.145 ms | 2 - 1509 MB | NPU
108
- | MobileSAMEncoder | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 295.631 ms | 4 - 1269 MB | NPU
109
- | MobileSAMEncoder | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 175.109 ms | 4 - 8 MB | NPU
110
- | MobileSAMEncoder | TFLITE | float | Qualcomm® SA8775P | 180.289 ms | 4 - 1269 MB | NPU
111
- | MobileSAMEncoder | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 584.211 ms | 0 - 3174 MB | NPU
112
- | MobileSAMEncoder | TFLITE | float | Qualcomm® SA7255P | 295.631 ms | 4 - 1269 MB | NPU
113
- | MobileSAMEncoder | TFLITE | float | Qualcomm® SA8295P | 502.945 ms | 4 - 1893 MB | NPU
114
- | MobileSAMEncoder | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 95.905 ms | 3 - 1261 MB | NPU
115
- | MobileSAMEncoder | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 78.087 ms | 4 - 1349 MB | NPU
116
 
117
  ## License
118
  * The license for the original implementation of MobileSam can be found
 
15
  Transformer based encoder-decoder where prompts specify what to segment in an image thereby allowing segmentation without the need for additional training. The image encoder generates embeddings and the lightweight decoder operates on the embeddings for point and mask based image segmentation.
16
 
17
  This is based on the implementation of MobileSam found [here](https://github.com/facebookresearch/segment-anything).
18
+ This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/main/qai_hub_models/models/mobilesam) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).
19
 
20
  Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.
21
 
 
28
 
29
  | Runtime | Precision | Chipset | SDK Versions | Download |
30
  |---|---|---|---|---|
31
+ | ONNX | float | Universal | QAIRT 2.42, ONNX Runtime 1.24.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mobilesam/releases/v0.48.0/mobilesam-onnx-float.zip)
32
+ | QNN_DLC | float | Universal | QAIRT 2.43 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mobilesam/releases/v0.48.0/mobilesam-qnn_dlc-float.zip)
33
+ | TFLITE | float | Universal | QAIRT 2.43, TFLite 2.17.0 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mobilesam/releases/v0.48.0/mobilesam-tflite-float.zip)
34
 
35
  For more device-specific assets and performance metrics, visit **[MobileSam on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/mobilesam)**.
36
 
37
 
38
  ### Option 2: Export with Custom Configurations
39
 
40
+ Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/main/qai_hub_models/models/mobilesam) Python library to compile and export the model with your own:
41
  - Custom weights (e.g., fine-tuned checkpoints)
42
  - Custom input shapes
43
  - Target device and runtime configurations
44
 
45
  This option is ideal if you need to customize the model beyond the default configuration provided here.
46
 
47
+ See our repository for [MobileSam on GitHub](https://github.com/qualcomm/ai-hub-models/blob/main/qai_hub_models/models/mobilesam) for usage instructions.
48
 
49
  ## Model Details
50
 
 
61
  ## Performance Summary
62
  | Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
63
  |---|---|---|---|---|---|---
64
+ | MobileSAMDecoder | ONNX | float | Snapdragon® X2 Elite | 2.953 ms | 11 - 11 MB | NPU
65
+ | MobileSAMDecoder | ONNX | float | Snapdragon® X Elite | 7.521 ms | 11 - 11 MB | NPU
66
+ | MobileSAMDecoder | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 4.892 ms | 4 - 270 MB | NPU
67
+ | MobileSAMDecoder | ONNX | float | Qualcomm® QCS8550 (Proxy) | 6.988 ms | 0 - 15 MB | NPU
68
+ | MobileSAMDecoder | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 3.641 ms | 1 - 223 MB | NPU
69
+ | MobileSAMDecoder | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.886 ms | 0 - 232 MB | NPU
70
+ | MobileSAMDecoder | QNN_DLC | float | Snapdragon® X2 Elite | 2.609 ms | 4 - 4 MB | NPU
71
+ | MobileSAMDecoder | QNN_DLC | float | Snapdragon® X Elite | 5.309 ms | 4 - 4 MB | NPU
72
+ | MobileSAMDecoder | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 3.364 ms | 4 - 230 MB | NPU
73
+ | MobileSAMDecoder | QNN_DLC | float | Qualcomm® QCS8275 (Proxy) | 11.701 ms | 0 - 199 MB | NPU
74
+ | MobileSAMDecoder | QNN_DLC | float | Qualcomm® QCS8550 (Proxy) | 4.844 ms | 4 - 5 MB | NPU
75
+ | MobileSAMDecoder | QNN_DLC | float | Qualcomm® SA8775P | 5.875 ms | 0 - 200 MB | NPU
76
+ | MobileSAMDecoder | QNN_DLC | float | Qualcomm® QCS8450 (Proxy) | 8.445 ms | 0 - 220 MB | NPU
77
+ | MobileSAMDecoder | QNN_DLC | float | Qualcomm® SA7255P | 11.701 ms | 0 - 199 MB | NPU
78
+ | MobileSAMDecoder | QNN_DLC | float | Qualcomm® SA8295P | 7.169 ms | 0 - 193 MB | NPU
79
+ | MobileSAMDecoder | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 2.593 ms | 0 - 200 MB | NPU
80
+ | MobileSAMDecoder | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.151 ms | 2 - 201 MB | NPU
81
+ | MobileSAMDecoder | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 3.877 ms | 0 - 235 MB | NPU
82
+ | MobileSAMDecoder | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 13.033 ms | 0 - 202 MB | NPU
83
+ | MobileSAMDecoder | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 5.605 ms | 0 - 18 MB | NPU
84
+ | MobileSAMDecoder | TFLITE | float | Qualcomm® SA8775P | 6.698 ms | 0 - 203 MB | NPU
85
+ | MobileSAMDecoder | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 9.541 ms | 0 - 225 MB | NPU
86
+ | MobileSAMDecoder | TFLITE | float | Qualcomm® SA7255P | 13.033 ms | 0 - 202 MB | NPU
87
+ | MobileSAMDecoder | TFLITE | float | Qualcomm® SA8295P | 8.047 ms | 0 - 201 MB | NPU
88
+ | MobileSAMDecoder | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 2.958 ms | 0 - 207 MB | NPU
89
+ | MobileSAMDecoder | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.352 ms | 0 - 207 MB | NPU
90
+ | MobileSAMEncoder | ONNX | float | Snapdragon® X2 Elite | 94.358 ms | 14 - 14 MB | NPU
91
+ | MobileSAMEncoder | ONNX | float | Snapdragon® X Elite | 173.085 ms | 12 - 12 MB | NPU
92
+ | MobileSAMEncoder | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 125.467 ms | 20 - 3025 MB | NPU
93
+ | MobileSAMEncoder | ONNX | float | Qualcomm® QCS8550 (Proxy) | 175.267 ms | 12 - 32 MB | NPU
94
+ | MobileSAMEncoder | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 100.697 ms | 9 - 1389 MB | NPU
95
+ | MobileSAMEncoder | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 88.418 ms | 20 - 1523 MB | NPU
96
+ | MobileSAMEncoder | QNN_DLC | float | Snapdragon® X2 Elite | 51.746 ms | 12 - 12 MB | NPU
97
+ | MobileSAMEncoder | QNN_DLC | float | Snapdragon® X Elite | 100.135 ms | 12 - 12 MB | NPU
98
+ | MobileSAMEncoder | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 66.42 ms | 12 - 1635 MB | NPU
99
+ | MobileSAMEncoder | QNN_DLC | float | Qualcomm® QCS8275 (Proxy) | 209.122 ms | 0 - 1209 MB | NPU
100
+ | MobileSAMEncoder | QNN_DLC | float | Qualcomm® QCS8550 (Proxy) | 96.617 ms | 12 - 16 MB | NPU
101
+ | MobileSAMEncoder | QNN_DLC | float | Qualcomm® SA8775P | 103.074 ms | 0 - 1209 MB | NPU
102
+ | MobileSAMEncoder | QNN_DLC | float | Qualcomm® QCS8450 (Proxy) | 470.205 ms | 12 - 1731 MB | NPU
103
+ | MobileSAMEncoder | QNN_DLC | float | Qualcomm® SA7255P | 209.122 ms | 0 - 1209 MB | NPU
104
+ | MobileSAMEncoder | QNN_DLC | float | Qualcomm® SA8295P | 474.048 ms | 0 - 1249 MB | NPU
105
+ | MobileSAMEncoder | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 49.865 ms | 12 - 1217 MB | NPU
106
+ | MobileSAMEncoder | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 48.392 ms | 8 - 1319 MB | NPU
107
+ | MobileSAMEncoder | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 126.457 ms | 0 - 1496 MB | NPU
108
+ | MobileSAMEncoder | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 295.258 ms | 4 - 1269 MB | NPU
109
+ | MobileSAMEncoder | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 176.603 ms | 0 - 3 MB | NPU
110
+ | MobileSAMEncoder | TFLITE | float | Qualcomm® SA8775P | 179.751 ms | 0 - 1265 MB | NPU
111
+ | MobileSAMEncoder | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 590.208 ms | 0 - 3175 MB | NPU
112
+ | MobileSAMEncoder | TFLITE | float | Qualcomm® SA7255P | 295.258 ms | 4 - 1269 MB | NPU
113
+ | MobileSAMEncoder | TFLITE | float | Qualcomm® SA8295P | 502.789 ms | 0 - 1888 MB | NPU
114
+ | MobileSAMEncoder | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 97.396 ms | 4 - 1267 MB | NPU
115
+ | MobileSAMEncoder | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 78.068 ms | 4 - 1348 MB | NPU
116
 
117
  ## License
118
  * The license for the original implementation of MobileSam can be found