Hugging Face's logo

qualcomm
/
ResNet-Mixed-Convolution

Video Classification
PyTorch
backbone
android
Model card Files
xet
Community
1
qaihm-bot commited on
Commit
7be4d2d
·
verified ·
1 Parent(s): f23aa0f

v0.46.1

Browse files

See https://github.com/quic/ai-hub-models/releases/v0.46.1 for changelog.

Files changed (7) hide show
  1. README.md +95 -236
  2. ResNet-Mixed-Convolution_float.dlc +0 -3
  3. ResNet-Mixed-Convolution_float.onnx.zip +0 -3
  4. ResNet-Mixed-Convolution_float.tflite +0 -3
  5. ResNet-Mixed-Convolution_w8a16.dlc +0 -3
  6. ResNet-Mixed-Convolution_w8a16.onnx.zip +0 -3
  7. tool-versions.yaml +0 -4
README.md CHANGED
@@ -10,254 +10,113 @@ pipeline_tag: video-classification
10
 
11
  ![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/resnet_mixed/web-assets/model_demo.png)
12
 
13
- # ResNet-Mixed-Convolution: Optimized for Mobile Deployment
14
- ## Sports and human action recognition in videos
15
-
16
 
17
  ResNet Mixed Convolutions is a network with a mixture of 2D and 3D convolutions used for video understanding.
18
 
19
- This model is an implementation of ResNet-Mixed-Convolution found [here](https://github.com/pytorch/vision/blob/main/torchvision/models/video/resnet.py).
20
-
21
-
22
- This repository provides scripts to run ResNet-Mixed-Convolution on Qualcomm® devices.
23
- More details on model performance across various devices, can be found
24
- [here](https://aihub.qualcomm.com/models/resnet_mixed).
25
-
26
-
27
-
28
- ### Model Details
29
-
30
- - **Model Type:** Model_use_case.video_classification
31
- - **Model Stats:**
32
- - Model checkpoint: Kinetics-400
33
- - Input resolution: 112x112
34
- - Number of parameters: 11.7M
35
- - Model size (float): 44.6 MB
36
- - Model size (w8a16): 11.5 MB
37
-
38
- | Model | Precision | Device | Chipset | Target Runtime | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit | Target Model
39
- |---|---|---|---|---|---|---|---|---|
40
- | ResNet-Mixed-Convolution | float | QCS8275 (Proxy) | Qualcomm® QCS8275 (Proxy) | TFLITE | 601.115 ms | 0 - 210 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
41
- | ResNet-Mixed-Convolution | float | QCS8275 (Proxy) | Qualcomm® QCS8275 (Proxy) | QNN_DLC | 97.227 ms | 2 - 188 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
42
- | ResNet-Mixed-Convolution | float | QCS8450 (Proxy) | Qualcomm® QCS8450 (Proxy) | TFLITE | 345.7 ms | 0 - 243 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
43
- | ResNet-Mixed-Convolution | float | QCS8450 (Proxy) | Qualcomm® QCS8450 (Proxy) | QNN_DLC | 27.537 ms | 2 - 216 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
44
- | ResNet-Mixed-Convolution | float | QCS8550 (Proxy) | Qualcomm® QCS8550 (Proxy) | TFLITE | 315.577 ms | 0 - 3 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
45
- | ResNet-Mixed-Convolution | float | QCS8550 (Proxy) | Qualcomm® QCS8550 (Proxy) | QNN_DLC | 13.387 ms | 2 - 5 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
46
- | ResNet-Mixed-Convolution | float | QCS8550 (Proxy) | Qualcomm® QCS8550 (Proxy) | ONNX | 13.858 ms | 0 - 29 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.onnx.zip) |
47
- | ResNet-Mixed-Convolution | float | QCS9075 (Proxy) | Qualcomm® QCS9075 (Proxy) | TFLITE | 301.681 ms | 0 - 210 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
48
- | ResNet-Mixed-Convolution | float | QCS9075 (Proxy) | Qualcomm® QCS9075 (Proxy) | QNN_DLC | 115.982 ms | 2 - 188 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
49
- | ResNet-Mixed-Convolution | float | SA7255P ADP | Qualcomm® SA7255P | TFLITE | 601.115 ms | 0 - 210 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
50
- | ResNet-Mixed-Convolution | float | SA7255P ADP | Qualcomm® SA7255P | QNN_DLC | 97.227 ms | 2 - 188 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
51
- | ResNet-Mixed-Convolution | float | SA8295P ADP | Qualcomm® SA8295P | TFLITE | 366.039 ms | 0 - 197 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
52
- | ResNet-Mixed-Convolution | float | SA8295P ADP | Qualcomm® SA8295P | QNN_DLC | 26.694 ms | 0 - 164 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
53
- | ResNet-Mixed-Convolution | float | SA8775P ADP | Qualcomm® SA8775P | TFLITE | 301.681 ms | 0 - 210 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
54
- | ResNet-Mixed-Convolution | float | SA8775P ADP | Qualcomm® SA8775P | QNN_DLC | 115.982 ms | 2 - 188 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
55
- | ResNet-Mixed-Convolution | float | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 Mobile | TFLITE | 238.039 ms | 2 - 271 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
56
- | ResNet-Mixed-Convolution | float | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 Mobile | QNN_DLC | 9.634 ms | 2 - 252 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
57
- | ResNet-Mixed-Convolution | float | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 Mobile | ONNX | 9.981 ms | 2 - 233 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.onnx.zip) |
58
- | ResNet-Mixed-Convolution | float | Samsung Galaxy S25 | Snapdragon® 8 Elite For Galaxy Mobile | TFLITE | 200.143 ms | 0 - 210 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
59
- | ResNet-Mixed-Convolution | float | Samsung Galaxy S25 | Snapdragon® 8 Elite For Galaxy Mobile | QNN_DLC | 7.716 ms | 2 - 191 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
60
- | ResNet-Mixed-Convolution | float | Samsung Galaxy S25 | Snapdragon® 8 Elite For Galaxy Mobile | ONNX | 8.166 ms | 1 - 162 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.onnx.zip) |
61
- | ResNet-Mixed-Convolution | float | Snapdragon 8 Elite Gen 5 QRD | Snapdragon® 8 Elite Gen 5 Mobile | TFLITE | 198.24 ms | 0 - 209 MB | NPU | [ResNet-Mixed-Convolution.tflite](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.tflite) |
62
- | ResNet-Mixed-Convolution | float | Snapdragon 8 Elite Gen 5 QRD | Snapdragon® 8 Elite Gen 5 Mobile | QNN_DLC | 5.668 ms | 2 - 195 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
63
- | ResNet-Mixed-Convolution | float | Snapdragon 8 Elite Gen 5 QRD | Snapdragon® 8 Elite Gen 5 Mobile | ONNX | 5.841 ms | 2 - 166 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.onnx.zip) |
64
- | ResNet-Mixed-Convolution | float | Snapdragon X Elite CRD | Snapdragon® X Elite | QNN_DLC | 14.062 ms | 2 - 2 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.dlc) |
65
- | ResNet-Mixed-Convolution | float | Snapdragon X Elite CRD | Snapdragon® X Elite | ONNX | 14.079 ms | 22 - 22 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution.onnx.zip) |
66
- | ResNet-Mixed-Convolution | w8a16 | Dragonwing Q-6690 MTP | Qualcomm® QCM6690 | QNN_DLC | 168.996 ms | 1 - 183 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
67
- | ResNet-Mixed-Convolution | w8a16 | Dragonwing Q-6690 MTP | Qualcomm® QCM6690 | ONNX | 902.51 ms | 100 - 114 MB | CPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.onnx.zip) |
68
- | ResNet-Mixed-Convolution | w8a16 | Dragonwing RB3 Gen 2 Vision Kit | Qualcomm® QCS6490 | QNN_DLC | 36.025 ms | 1 - 3 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
69
- | ResNet-Mixed-Convolution | w8a16 | Dragonwing RB3 Gen 2 Vision Kit | Qualcomm® QCS6490 | ONNX | 1730.135 ms | 44 - 62 MB | CPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.onnx.zip) |
70
- | ResNet-Mixed-Convolution | w8a16 | QCS8275 (Proxy) | Qualcomm® QCS8275 (Proxy) | QNN_DLC | 29.645 ms | 1 - 161 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
71
- | ResNet-Mixed-Convolution | w8a16 | QCS8450 (Proxy) | Qualcomm® QCS8450 (Proxy) | QNN_DLC | 13.197 ms | 1 - 224 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
72
- | ResNet-Mixed-Convolution | w8a16 | QCS8550 (Proxy) | Qualcomm® QCS8550 (Proxy) | QNN_DLC | 9.355 ms | 1 - 3 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
73
- | ResNet-Mixed-Convolution | w8a16 | QCS8550 (Proxy) | Qualcomm® QCS8550 (Proxy) | ONNX | 8.749 ms | 0 - 15 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.onnx.zip) |
74
- | ResNet-Mixed-Convolution | w8a16 | QCS9075 (Proxy) | Qualcomm® QCS9075 (Proxy) | QNN_DLC | 9.397 ms | 1 - 162 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
75
- | ResNet-Mixed-Convolution | w8a16 | SA7255P ADP | Qualcomm® SA7255P | QNN_DLC | 29.645 ms | 1 - 161 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
76
- | ResNet-Mixed-Convolution | w8a16 | SA8295P ADP | Qualcomm® SA8295P | QNN_DLC | 16.12 ms | 1 - 166 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
77
- | ResNet-Mixed-Convolution | w8a16 | SA8775P ADP | Qualcomm® SA8775P | QNN_DLC | 9.397 ms | 1 - 162 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
78
- | ResNet-Mixed-Convolution | w8a16 | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 Mobile | QNN_DLC | 6.631 ms | 1 - 225 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
79
- | ResNet-Mixed-Convolution | w8a16 | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 Mobile | ONNX | 6.523 ms | 1 - 196 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.onnx.zip) |
80
- | ResNet-Mixed-Convolution | w8a16 | Samsung Galaxy S25 | Snapdragon® 8 Elite For Galaxy Mobile | QNN_DLC | 5.734 ms | 1 - 160 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
81
- | ResNet-Mixed-Convolution | w8a16 | Samsung Galaxy S25 | Snapdragon® 8 Elite For Galaxy Mobile | ONNX | 5.307 ms | 1 - 133 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.onnx.zip) |
82
- | ResNet-Mixed-Convolution | w8a16 | Snapdragon 7 Gen 4 QRD | Snapdragon® 7 Gen 4 Mobile | QNN_DLC | 15.767 ms | 1 - 168 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
83
- | ResNet-Mixed-Convolution | w8a16 | Snapdragon 7 Gen 4 QRD | Snapdragon® 7 Gen 4 Mobile | ONNX | 919.68 ms | 106 - 121 MB | CPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.onnx.zip) |
84
- | ResNet-Mixed-Convolution | w8a16 | Snapdragon 8 Elite Gen 5 QRD | Snapdragon® 8 Elite Gen 5 Mobile | QNN_DLC | 3.817 ms | 1 - 163 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
85
- | ResNet-Mixed-Convolution | w8a16 | Snapdragon 8 Elite Gen 5 QRD | Snapdragon® 8 Elite Gen 5 Mobile | ONNX | 3.912 ms | 0 - 137 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.onnx.zip) |
86
- | ResNet-Mixed-Convolution | w8a16 | Snapdragon X Elite CRD | Snapdragon® X Elite | QNN_DLC | 9.929 ms | 1 - 1 MB | NPU | [ResNet-Mixed-Convolution.dlc](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.dlc) |
87
- | ResNet-Mixed-Convolution | w8a16 | Snapdragon X Elite CRD | Snapdragon® X Elite | ONNX | 9.085 ms | 12 - 12 MB | NPU | [ResNet-Mixed-Convolution.onnx.zip](https://huggingface.co/qualcomm/ResNet-Mixed-Convolution/blob/main/ResNet-Mixed-Convolution_w8a16.onnx.zip) |
88
-
89
-
90
-
91
-
92
- ## Installation
93
-
94
-
95
- Install the package via pip:
96
- ```bash
97
- # NOTE: 3.10 <= PYTHON_VERSION < 3.14 is supported.
98
- pip install "qai-hub-models[resnet-mixed]"
99
- ```
100
-
101
-
102
- ## Configure Qualcomm® AI Hub Workbench to run this model on a cloud-hosted device
103
-
104
- Sign-in to [Qualcomm® AI Hub Workbench](https://workbench.aihub.qualcomm.com/) with your
105
- Qualcomm® ID. Once signed in navigate to `Account -> Settings -> API Token`.
106
-
107
- With this API token, you can configure your client to run models on the cloud
108
- hosted devices.
109
- ```bash
110
- qai-hub configure --api_token API_TOKEN
111
- ```
112
- Navigate to [docs](https://workbench.aihub.qualcomm.com/docs/) for more information.
113
-
114
-
115
-
116
- ## Demo off target
117
-
118
- The package contains a simple end-to-end demo that downloads pre-trained
119
- weights and runs this model on a sample input.
120
-
121
- ```bash
122
- python -m qai_hub_models.models.resnet_mixed.demo
123
- ```
124
-
125
- The above demo runs a reference implementation of pre-processing, model
126
- inference, and post processing.
127
-
128
- **NOTE**: If you want running in a Jupyter Notebook or Google Colab like
129
- environment, please add the following to your cell (instead of the above).
130
- ```
131
- %run -m qai_hub_models.models.resnet_mixed.demo
132
- ```
133
-
134
-
135
- ### Run model on a cloud-hosted device
136
-
137
- In addition to the demo, you can also run the model on a cloud-hosted Qualcomm®
138
- device. This script does the following:
139
- * Performance check on-device on a cloud-hosted device
140
- * Downloads compiled assets that can be deployed on-device for Android.
141
- * Accuracy check between PyTorch and on-device outputs.
142
-
143
- ```bash
144
- python -m qai_hub_models.models.resnet_mixed.export
145
- ```
146
-
147
-
148
-
149
- ## How does this work?
150
-
151
- This [export script](https://aihub.qualcomm.com/models/resnet_mixed/qai_hub_models/models/ResNet-Mixed-Convolution/export.py)
152
- leverages [Qualcomm® AI Hub](https://aihub.qualcomm.com/) to optimize, validate, and deploy this model
153
- on-device. Lets go through each step below in detail:
154
-
155
- Step 1: **Compile model for on-device deployment**
156
-
157
- To compile a PyTorch model for on-device deployment, we first trace the model
158
- in memory using the `jit.trace` and then call the `submit_compile_job` API.
159
-
160
- ```python
161
- import torch
162
-
163
- import qai_hub as hub
164
- from qai_hub_models.models.resnet_mixed import Model
165
-
166
- # Load the model
167
- torch_model = Model.from_pretrained()
168
-
169
- # Device
170
- device = hub.Device("Samsung Galaxy S25")
171
-
172
- # Trace model
173
- input_shape = torch_model.get_input_spec()
174
- sample_inputs = torch_model.sample_inputs()
175
-
176
- pt_model = torch.jit.trace(torch_model, [torch.tensor(data[0]) for _, data in sample_inputs.items()])
177
-
178
- # Compile model on a specific device
179
- compile_job = hub.submit_compile_job(
180
- model=pt_model,
181
- device=device,
182
- input_specs=torch_model.get_input_spec(),
183
- )
184
-
185
- # Get target model to run on-device
186
- target_model = compile_job.get_target_model()
187
-
188
- ```
189
-
190
-
191
- Step 2: **Performance profiling on cloud-hosted device**
192
-
193
- After compiling models from step 1. Models can be profiled model on-device using the
194
- `target_model`. Note that this scripts runs the model on a device automatically
195
- provisioned in the cloud. Once the job is submitted, you can navigate to a
196
- provided job URL to view a variety of on-device performance metrics.
197
- ```python
198
- profile_job = hub.submit_profile_job(
199
- model=target_model,
200
- device=device,
201
- )
202
-
203
- ```
204
-
205
- Step 3: **Verify on-device accuracy**
206
-
207
- To verify the accuracy of the model on-device, you can run on-device inference
208
- on sample input data on the same cloud hosted device.
209
- ```python
210
- input_data = torch_model.sample_inputs()
211
- inference_job = hub.submit_inference_job(
212
- model=target_model,
213
- device=device,
214
- inputs=input_data,
215
- )
216
- on_device_output = inference_job.download_output_data()
217
-
218
- ```
219
- With the output of the model, you can compute like PSNR, relative errors or
220
- spot check the output with expected output.
221
-
222
- **Note**: This on-device profiling and inference requires access to Qualcomm®
223
- AI Hub Workbench. [Sign up for access](https://myaccount.qualcomm.com/signup).
224
-
225
-
226
-
227
-
228
- ## Deploying compiled model to Android
229
-
230
-
231
- The models can be deployed using multiple runtimes:
232
- - TensorFlow Lite (`.tflite` export): [This
233
- tutorial](https://www.tensorflow.org/lite/android/quickstart) provides a
234
- guide to deploy the .tflite model in an Android application.
235
-
236
-
237
- - QNN (`.so` export ): This [sample
238
- app](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/sample_app.html)
239
- provides instructions on how to use the `.so` shared library in an Android application.
240
-
241
-
242
- ## View on Qualcomm® AI Hub
243
- Get more details on ResNet-Mixed-Convolution's performance across various devices [here](https://aihub.qualcomm.com/models/resnet_mixed).
244
- Explore all available models on [Qualcomm® AI Hub](https://aihub.qualcomm.com/)
245
-
246
 
247
  ## License
248
  * The license for the original implementation of ResNet-Mixed-Convolution can be found
249
  [here](https://github.com/pytorch/vision/blob/main/LICENSE).
250
 
251
-
252
-
253
  ## References
254
  * [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://arxiv.org/abs/1711.11248)
255
  * [Source Model Implementation](https://github.com/pytorch/vision/blob/main/torchvision/models/video/resnet.py)
256
 
257
-
258
-
259
  ## Community
260
  * Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
261
  * For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).
262
-
263
-
 
10
 
11
  ![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/resnet_mixed/web-assets/model_demo.png)
12
 
13
+ # ResNet-Mixed-Convolution: Optimized for Qualcomm Devices
 
 
14
 
15
  ResNet Mixed Convolutions is a network with a mixture of 2D and 3D convolutions used for video understanding.
16
 
17
+ This is based on the implementation of ResNet-Mixed-Convolution found [here](https://github.com/pytorch/vision/blob/main/torchvision/models/video/resnet.py).
18
+ This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/quic/ai-hub-models/blob/main/qai_hub_models/models/resnet_mixed) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).
19
+
20
+ Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.
21
+
22
+ ## Getting Started
23
+ There are two ways to deploy this model on your device:
24
+
25
+ ### Option 1: Download Pre-Exported Models
26
+
27
+ Below are pre-exported model assets ready for deployment.
28
+
29
+ | Runtime | Precision | Chipset | SDK Versions | Download |
30
+ |---|---|---|---|---|
31
+ | ONNX | float | Universal | QAIRT 2.37, ONNX Runtime 1.23.0 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/resnet_mixed/releases/v0.46.1/resnet_mixed-onnx-float.zip)
32
+ | ONNX | w8a16 | Universal | QAIRT 2.37, ONNX Runtime 1.23.0 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/resnet_mixed/releases/v0.46.1/resnet_mixed-onnx-w8a16.zip)
33
+ | QNN_DLC | float | Universal | QAIRT 2.42 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/resnet_mixed/releases/v0.46.1/resnet_mixed-qnn_dlc-float.zip)
34
+ | QNN_DLC | w8a16 | Universal | QAIRT 2.42 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/resnet_mixed/releases/v0.46.1/resnet_mixed-qnn_dlc-w8a16.zip)
35
+ | TFLITE | float | Universal | QAIRT 2.42, TFLite 2.17.0 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/resnet_mixed/releases/v0.46.1/resnet_mixed-tflite-float.zip)
36
+
37
+ For more device-specific assets and performance metrics, visit **[ResNet-Mixed-Convolution on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/resnet_mixed)**.
38
+
39
+
40
+ ### Option 2: Export with Custom Configurations
41
+
42
+ Use the [Qualcomm® AI Hub Models](https://github.com/quic/ai-hub-models/blob/main/qai_hub_models/models/resnet_mixed) Python library to compile and export the model with your own:
43
+ - Custom weights (e.g., fine-tuned checkpoints)
44
+ - Custom input shapes
45
+ - Target device and runtime configurations
46
+
47
+ This option is ideal if you need to customize the model beyond the default configuration provided here.
48
+
49
+ See our repository for [ResNet-Mixed-Convolution on GitHub](https://github.com/quic/ai-hub-models/blob/main/qai_hub_models/models/resnet_mixed) for usage instructions.
50
+
51
+ ## Model Details
52
+
53
+ **Model Type:** Model_use_case.video_classification
54
+
55
+ **Model Stats:**
56
+ - Model checkpoint: Kinetics-400
57
+ - Input resolution: 112x112
58
+ - Number of parameters: 11.7M
59
+ - Model size (float): 44.6 MB
60
+ - Model size (w8a16): 11.5 MB
61
+
62
+ ## Performance Summary
63
+ | Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
64
+ |---|---|---|---|---|---|---
65
+ | ResNet-Mixed-Convolution | ONNX | float | Snapdragon® X Elite | 14.062 ms | 22 - 22 MB | NPU
66
+ | ResNet-Mixed-Convolution | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 9.988 ms | 2 - 235 MB | NPU
67
+ | ResNet-Mixed-Convolution | ONNX | float | Qualcomm® QCS8550 (Proxy) | 13.721 ms | 0 - 38 MB | NPU
68
+ | ResNet-Mixed-Convolution | ONNX | float | Qualcomm® QCS9075 | 27.604 ms | 2 - 5 MB | NPU
69
+ | ResNet-Mixed-Convolution | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 8.173 ms | 0 - 167 MB | NPU
70
+ | ResNet-Mixed-Convolution | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 5.86 ms | 2 - 167 MB | NPU
71
+ | ResNet-Mixed-Convolution | ONNX | w8a16 | Snapdragon® X Elite | 9.123 ms | 12 - 12 MB | NPU
72
+ | ResNet-Mixed-Convolution | ONNX | w8a16 | Snapdragon® 8 Gen 3 Mobile | 6.522 ms | 1 - 195 MB | NPU
73
+ | ResNet-Mixed-Convolution | ONNX | w8a16 | Qualcomm® QCS6490 | 1728.182 ms | 45 - 62 MB | CPU
74
+ | ResNet-Mixed-Convolution | ONNX | w8a16 | Qualcomm® QCS8550 (Proxy) | 8.78 ms | 0 - 232 MB | NPU
75
+ | ResNet-Mixed-Convolution | ONNX | w8a16 | Qualcomm® QCS9075 | 9.303 ms | 1 - 4 MB | NPU
76
+ | ResNet-Mixed-Convolution | ONNX | w8a16 | Qualcomm® QCM6690 | 890.317 ms | 106 - 113 MB | CPU
77
+ | ResNet-Mixed-Convolution | ONNX | w8a16 | Snapdragon® 8 Elite For Galaxy Mobile | 5.341 ms | 1 - 135 MB | NPU
78
+ | ResNet-Mixed-Convolution | ONNX | w8a16 | Snapdragon® 7 Gen 4 Mobile | 881.492 ms | 105 - 112 MB | CPU
79
+ | ResNet-Mixed-Convolution | ONNX | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 3.904 ms | 0 - 138 MB | NPU
80
+ | ResNet-Mixed-Convolution | QNN_DLC | float | Snapdragon® X Elite | 14.152 ms | 2 - 2 MB | NPU
81
+ | ResNet-Mixed-Convolution | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 9.536 ms | 0 - 282 MB | NPU
82
+ | ResNet-Mixed-Convolution | QNN_DLC | float | Qualcomm® QCS8550 (Proxy) | 13.367 ms | 2 - 5 MB | NPU
83
+ | ResNet-Mixed-Convolution | QNN_DLC | float | Qualcomm® SA8775P | 25.055 ms | 1 - 223 MB | NPU
84
+ | ResNet-Mixed-Convolution | QNN_DLC | float | Qualcomm® QCS9075 | 27.978 ms | 2 - 6 MB | NPU
85
+ | ResNet-Mixed-Convolution | QNN_DLC | float | Qualcomm® QCS8450 (Proxy) | 27.577 ms | 0 - 240 MB | NPU
86
+ | ResNet-Mixed-Convolution | QNN_DLC | float | Qualcomm® SA8295P | 26.689 ms | 0 - 192 MB | NPU
87
+ | ResNet-Mixed-Convolution | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 7.684 ms | 2 - 227 MB | NPU
88
+ | ResNet-Mixed-Convolution | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 5.76 ms | 2 - 233 MB | NPU
89
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Snapdragon® X Elite | 9.877 ms | 1 - 1 MB | NPU
90
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Snapdragon® 8 Gen 3 Mobile | 6.654 ms | 1 - 255 MB | NPU
91
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Qualcomm® QCS6490 | 37.276 ms | 1 - 4 MB | NPU
92
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Qualcomm® QCS8550 (Proxy) | 9.22 ms | 1 - 3 MB | NPU
93
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Qualcomm® SA8775P | 9.288 ms | 1 - 191 MB | NPU
94
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Qualcomm® QCS9075 | 10.377 ms | 1 - 4 MB | NPU
95
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Qualcomm® QCM6690 | 178.522 ms | 1 - 208 MB | NPU
96
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Qualcomm® QCS8450 (Proxy) | 12.996 ms | 1 - 255 MB | NPU
97
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Qualcomm® SA8295P | 16.239 ms | 1 - 192 MB | NPU
98
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Snapdragon® 8 Elite For Galaxy Mobile | 5.261 ms | 1 - 186 MB | NPU
99
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Snapdragon® 7 Gen 4 Mobile | 15.94 ms | 1 - 197 MB | NPU
100
+ | ResNet-Mixed-Convolution | QNN_DLC | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 3.995 ms | 1 - 190 MB | NPU
101
+ | ResNet-Mixed-Convolution | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 218.735 ms | 0 - 313 MB | NPU
102
+ | ResNet-Mixed-Convolution | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 585.16 ms | 0 - 252 MB | NPU
103
+ | ResNet-Mixed-Convolution | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 313.312 ms | 0 - 2 MB | NPU
104
+ | ResNet-Mixed-Convolution | TFLITE | float | Qualcomm® SA8775P | 296.864 ms | 0 - 252 MB | NPU
105
+ | ResNet-Mixed-Convolution | TFLITE | float | Qualcomm® QCS9075 | 321.486 ms | 0 - 28 MB | NPU
106
+ | ResNet-Mixed-Convolution | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 345.498 ms | 0 - 277 MB | NPU
107
+ | ResNet-Mixed-Convolution | TFLITE | float | Qualcomm® SA7255P | 585.16 ms | 0 - 252 MB | NPU
108
+ | ResNet-Mixed-Convolution | TFLITE | float | Qualcomm® SA8295P | 353.223 ms | 0 - 234 MB | NPU
109
+ | ResNet-Mixed-Convolution | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 203.459 ms | 0 - 257 MB | NPU
110
+ | ResNet-Mixed-Convolution | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 189.334 ms | 0 - 254 MB | NPU
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
111
 
112
  ## License
113
  * The license for the original implementation of ResNet-Mixed-Convolution can be found
114
  [here](https://github.com/pytorch/vision/blob/main/LICENSE).
115
 
 
 
116
  ## References
117
  * [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://arxiv.org/abs/1711.11248)
118
  * [Source Model Implementation](https://github.com/pytorch/vision/blob/main/torchvision/models/video/resnet.py)
119
 
 
 
120
  ## Community
121
  * Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
122
  * For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).
 
 
ResNet-Mixed-Convolution_float.dlc DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:999a0d160f2c1555c21564f773c8048323f920b450b4193a0f72174964aa2269
3
- size 46827372
 
 
 
 
ResNet-Mixed-Convolution_float.onnx.zip DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:3460198a484786b3ab2f0ff61d559ad654ef32db1e5826b85ee02778da6340da
3
- size 43532530
 
 
 
 
ResNet-Mixed-Convolution_float.tflite DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:4d217e70fd6a14f3da1d11bb2df0d5734774db4d50b5c43f618ee844a67085c0
3
- size 46776824
 
 
 
 
ResNet-Mixed-Convolution_w8a16.dlc DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:e4041f9b0f871e9342ba8f9102a5f592ffcd4c9b7b7face9b78bc239a8b2adeb
3
- size 12042956
 
 
 
 
ResNet-Mixed-Convolution_w8a16.onnx.zip DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:da0031df3285d44df42f09f70182e39fb42dd9d2c943ddb64a7afbd1f5e01e10
3
- size 19583000
 
 
 
 
tool-versions.yaml DELETED
@@ -1,4 +0,0 @@
1
- tool_versions:
2
- onnx:
3
- qairt: 2.37.1.250807093845_124904
4
- onnx_runtime: 1.23.0