| | --- |
| | license: other |
| | license_name: flux-1-dev-non-commercial-license |
| | license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md |
| |
|
| | language: |
| | - en |
| | library_name: diffusers |
| | pipeline_tag: text-to-image |
| |
|
| | tags: |
| | - Text-to-Image |
| | - ControlNet |
| | - Diffusers |
| | - Flux.1-dev |
| | - image-generation |
| | - Stable Diffusion |
| | base_model: black-forest-labs/FLUX.1-dev |
| | --- |
| | |
| | # FLUX.1-dev-ControlNet-Depth |
| |
|
| | This repository contains a Depth ControlNet for FLUX.1-dev model jointly trained by researchers from [InstantX Team](https://huggingface.co/InstantX) and [Shakker Labs](https://huggingface.co/Shakker-Labs). |
| |
|
| | <div class="container"> |
| | <img src="./assets/poster.png" width="1024"/> |
| | </div> |
| |
|
| | # Model Cards |
| | - The model consists of 4 FluxTransformerBlock and 1 FluxSingleTransformerBlock. |
| | - This checkpoint is trained on both real and generated image datasets, with 16\*A800 for 70K steps. The batch size 16\*4=64 with resolution=1024. The learning rate is set to 5e-6. We use [Depth-Anything-V2](https://github.com/DepthAnything/Depth-Anything-V2) to extract depth maps. |
| | - The recommended controlnet_conditioning_scale is 0.3-0.7. |
| |
|
| | # Showcases |
| |
|
| | <div class="container"> |
| | <img src="./assets/teaser.png" width="1024"/> |
| | </div> |
| |
|
| |
|
| | # Inference |
| | ```python |
| | import torch |
| | from diffusers.utils import load_image |
| | from diffusers import FluxControlNetPipeline, FluxControlNetModel |
| | |
| | base_model = "black-forest-labs/FLUX.1-dev" |
| | controlnet_model = "Shakker-Labs/FLUX.1-dev-ControlNet-Depth" |
| | |
| | controlnet = FluxControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16) |
| | pipe = FluxControlNetPipeline.from_pretrained( |
| | base_model, controlnet=controlnet, torch_dtype=torch.bfloat16 |
| | ) |
| | pipe.to("cuda") |
| | |
| | control_image = load_image("https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Depth/resolve/main/assets/cond1.png") |
| | prompt = "an old man with white hair" |
| | |
| | image = pipe(prompt, |
| | control_image=control_image, |
| | controlnet_conditioning_scale=0.5, |
| | width=control_image.size[0], |
| | height=control_image.size[1], |
| | num_inference_steps=24, |
| | guidance_scale=3.5, |
| | ).images[0] |
| | ``` |
| |
|
| | For multi-ControlNets support, please refer to [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro). |
| |
|
| | # Resources |
| | - [InstantX/FLUX.1-dev-Controlnet-Canny](https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny) |
| | - [Shakker-Labs/FLUX.1-dev-ControlNet-Depth](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Depth) |
| | - [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro) |
| |
|
| | # Acknowledgements |
| | This project is sponsored and released by [Shakker AI](https://www.shakker.ai/). All copyright reserved. |
| |
|
