| | --- |
| | pipeline_tag: image-feature-extraction |
| | license: mit |
| | library_name: diffusion-single-file |
| | --- |
| | |
| | # CleanDIFT Model Card |
| |
|
| |
|
| | Diffusion models learn powerful world representations that have proven valuable for tasks like semantic correspondence detection, |
| | depth estimation, semantic segmentation, and classification. |
| | However, diffusion models require noisy input images, which destroys information and introduces the noise level as a hyperparameter that needs to be tuned for each task. |
| |
|
| |
|
| |
|
| |
|
| | We introduce CleanDIFT, a novel method to extract noise-free, timestep-independent features by enabling diffusion models to work directly with clean input images. |
| | The approach is efficient, training on a single GPU in just 30 minutes. We publish these models alongside our paper ["CleanDIFT: Diffusion Features without Noise"](https://compvis.github.io/cleandift/). |
| |
|
| | We provide checkpoints for Stable Diffusion 1.5 and Stable Diffusion 2.1. |
| |
|
| |
|
| | ## Usage |
| |
|
| | For detailed examples on how to extract features with CleanDIFT and how to use them for downstream tasks, please refer to the notebooks provided [here](https://github.com/CompVis/CleanDIFT/tree/main/notebooks). |
| |
|
| | Our checkpoints are fully compatible with the `diffusers` library. |
| | If you already have a pipeline using SD 1.5 or SD 2.1 from `diffusers`, you can simply replace the U-Net state dict: |
| | ```python |
| | from diffusers import UNet2DConditionModel |
| | from huggingface_hub import hf_hub_download |
| | |
| | unet = UNet2DConditionModel.from_pretrained("stabilityai/stable-diffusion-2-1", subfolder="unet") |
| | ckpt_pth = hf_hub_download(repo_id="CompVis/cleandift", filename="cleandift_sd21_unet.safetensors") |
| | state_dict = load_file(ckpt_pth) |
| | unet.load_state_dict(state_dict, strict=True) |
| | ``` |
| |
|
| | ## Citation |
| |
|
| | ```bibtex |
| | @misc{stracke2024cleandiftdiffusionfeaturesnoise, |
| | title={CleanDIFT: Diffusion Features without Noise}, |
| | author={Nick Stracke and Stefan Andreas Baumann and Kolja Bauer and Frank Fundel and Björn Ommer}, |
| | year={2024}, |
| | eprint={2412.03439}, |
| | archivePrefix={arXiv}, |
| | primaryClass={cs.CV}, |
| | url={https://arxiv.org/abs/2412.03439}, |
| | } |
| | ``` |
