Hugging Face's logo

qualcomm
/
ControlNet-Canny

Unconditional Image Generation
PyTorch
generative_ai
android
Model card Files
xet
 Community
ControlNet-Canny / README.md
qaihm-bot's picture
qaihm-bot
v0.49.1
9b33c04 verified
preview code
|
raw
history blame contribute delete
18.1 kB
metadata 
library_name: pytorch
license: other
tags:
  - generative_ai
  - android
pipeline_tag: unconditional-image-generation

ControlNet-Canny: Optimized for Qualcomm Devices

On-device, high-resolution image synthesis from text and image prompts. ControlNet guides Stable-diffusion with provided input image to generate accurate images from given input prompt.

This is based on the implementation of ControlNet-Canny found here. This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the Qualcomm® AI Hub Models library to export with custom configurations. More details on model performance across various devices, can be found here.

Qualcomm AI Hub Models uses Qualcomm AI Hub Workbench to compile, profile, and evaluate this model. Sign up to run these models on a hosted Qualcomm® device.

Getting Started

There are two ways to deploy this model on your device:

Option 1: Download Pre-Exported Models

Below are pre-exported model assets ready for deployment.

Runtime Precision Chipset SDK Versions Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile QAIRT 2.42, ONNX Runtime 1.24.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite QAIRT 2.42, ONNX Runtime 1.24.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite QAIRT 2.42, ONNX Runtime 1.24.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile QAIRT 2.42, ONNX Runtime 1.24.1 Download
PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS8550 (Proxy) QAIRT 2.42, ONNX Runtime 1.24.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite For Galaxy Mobile QAIRT 2.42, ONNX Runtime 1.24.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile QAIRT 2.42, ONNX Runtime 1.24.1 Download
PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCM6690 QAIRT 2.42, ONNX Runtime 1.24.1 Download
PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS9075 QAIRT 2.42, ONNX Runtime 1.24.1 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS8550 (Proxy) QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite For Galaxy Mobile QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® QCM6690 QAIRT 2.43 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS9075 QAIRT 2.43 Download

For more device-specific assets and performance metrics, visit ControlNet-Canny on Qualcomm® AI Hub.

Option 2: Export with Custom Configurations

Use the Qualcomm® AI Hub Models Python library to compile and export the model with your own:

  • Custom weights (e.g., fine-tuned checkpoints)
  • Custom input shapes
  • Target device and runtime configurations

This option is ideal if you need to customize the model beyond the default configuration provided here.

See our repository for ControlNet-Canny on GitHub for usage instructions.

Model Details

Model Type: Model_use_case.image_generation

Model Stats:

  • Input: Text prompt and input image as a reference
  • Conditioning Input: Canny-Edge
  • Text Encoder Number of parameters: 340M
  • UNet Number of parameters: 865M
  • VAE Decoder Number of parameters: 83M
  • ControlNet Number of parameters: 361M
  • Model size: 1.4GB

Performance Summary

Model Runtime Precision Chipset Inference Time (ms) Peak Memory Range (MB) Primary Compute Unit
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile 28.336 ms 31 - 41 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite 28.501 ms 351 - 351 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite 76.056 ms 351 - 351 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile 61.637 ms 33 - 46 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS8550 (Proxy) 82.896 ms 30 - 32 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS9075 75.852 ms 30 - 34 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCM6690 1063.912 ms 2 - 13 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite For Galaxy Mobile 45.185 ms 33 - 43 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile 124.515 ms 31 - 42 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile 28.347 ms 2 - 12 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite 29.436 ms 2 - 2 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite 76.012 ms 2 - 2 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile 60.725 ms 2 - 9 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS8275 (Proxy) 141.54 ms 2 - 9 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS8550 (Proxy) 82.486 ms 2 - 16 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P 305.49 ms 2 - 9 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS9075 76.281 ms 0 - 16 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® QCM6690 913.595 ms 2 - 8 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P 141.54 ms 2 - 9 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite For Galaxy Mobile 47.93 ms 2 - 10 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile 122.561 ms 2 - 8 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile 1.494 ms 0 - 10 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite 1.528 ms 156 - 156 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite 3.618 ms 156 - 156 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile 2.298 ms 0 - 10 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS8550 (Proxy) 3.247 ms 0 - 161 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS9075 3.331 ms 0 - 3 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCM6690 22.101 ms 0 - 10 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite For Galaxy Mobile 1.756 ms 0 - 13 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile 3.635 ms 0 - 6 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile 1.493 ms 0 - 9 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite 1.783 ms 0 - 0 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite 3.627 ms 0 - 0 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile 2.323 ms 0 - 7 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS8275 (Proxy) 7.545 ms 0 - 8 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS8550 (Proxy) 3.25 ms 0 - 1 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P 3.524 ms 0 - 9 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS9075 3.364 ms 0 - 2 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® QCM6690 21.98 ms 0 - 6 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P 7.545 ms 0 - 8 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite For Galaxy Mobile 1.745 ms 0 - 13 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile 3.65 ms 0 - 7 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile 45.448 ms 13 - 21 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite 48.612 ms 831 - 831 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite 114.377 ms 829 - 829 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile 80.413 ms 13 - 25 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS8550 (Proxy) 113.125 ms 0 - 882 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS9075 106.688 ms 13 - 29 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCM6690 2162.771 ms 13 - 24 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite For Galaxy Mobile 61.943 ms 6 - 17 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile 180.235 ms 9 - 20 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile 44.689 ms 13 - 22 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite 46.58 ms 13 - 13 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite 113.532 ms 13 - 13 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile 80.842 ms 13 - 20 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS8275 (Proxy) 266.779 ms 0 - 6 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS8550 (Proxy) 112.27 ms 13 - 15 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P 107.75 ms 7 - 14 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS9075 107.147 ms 11 - 25 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® QCM6690 2123.069 ms 13 - 24 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P 266.779 ms 0 - 6 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite For Galaxy Mobile 61.944 ms 6 - 15 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile 181.03 ms 10 - 16 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile 117.124 ms 3 - 13 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite 125.582 ms 62 - 62 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite 266.276 ms 62 - 62 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile 203.046 ms 3 - 10 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS8550 (Proxy) 270.802 ms 1 - 66 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCS9075 250.059 ms 3 - 6 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® QCM6690 6369.029 ms 0 - 6 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite For Galaxy Mobile 178.953 ms 3 - 10 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile 513.926 ms 1 - 8 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile 130.322 ms 0 - 10 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite 133.418 ms 0 - 0 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite 262.733 ms 0 - 0 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile 203.185 ms 0 - 8 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS8275 (Proxy) 718.469 ms 0 - 9 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS8550 (Proxy) 267.018 ms 1 - 3 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P 250.474 ms 0 - 9 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® QCS9075 247.311 ms 0 - 3 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® QCM6690 6429.227 ms 1 - 7 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P 718.469 ms 0 - 9 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite For Galaxy Mobile 177.535 ms 0 - 14 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile 514.686 ms 0 - 6 MB NPU

License

  • The license for the original implementation of ControlNet-Canny can be found here.

References

Community

Usage and Limitations

This model may not be used for or in connection with any of the following applications:

  • Accessing essential private and public services and benefits;
  • Administration of justice and democratic processes;
  • Assessing or recognizing the emotional state of a person;
  • Biometric and biometrics-based systems, including categorization of persons based on sensitive characteristics;
  • Education and vocational training;
  • Employment and workers management;
  • Exploitation of the vulnerabilities of persons resulting in harmful behavior;
  • General purpose social scoring;
  • Law enforcement;
  • Management and operation of critical infrastructure;
  • Migration, asylum and border control management;
  • Predictive policing;
  • Real-time remote biometric identification in public spaces;
  • Recommender systems of social media platforms;
  • Scraping of facial images (from the internet or otherwise); and/or
  • Subliminal manipulation