diffusers/docs/source/en/api/models/controlnet_union.md

ControlNetUnionModel

ControlNetUnionModel is an implementation of ControlNet for Stable Diffusion XL.

The ControlNet model was introduced in ControlNetPlus by xinsir6. It supports multiple conditioning inputs without increasing computation.

We design a new architecture that can support 10+ control types in condition text-to-image generation and can generate high resolution images visually comparable with midjourney. The network is based on the original ControlNet architecture, we propose two new modules to: 1 Extend the original ControlNet to support different image conditions using the same network parameter. 2 Support multiple conditions input without increasing computation offload, which is especially important for designers who want to edit image in detail, different conditions use the same condition encoder, without adding extra computations or parameters.

Loading

By default the [ControlNetUnionModel] should be loaded with [~ModelMixin.from_pretrained].

from diffusers import StableDiffusionXLControlNetUnionPipeline, ControlNetUnionModel

controlnet = ControlNetUnionModel.from_pretrained("xinsir/controlnet-union-sdxl-1.0")
pipe = StableDiffusionXLControlNetUnionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnet)

ControlNetUnionModel

[[autodoc]] ControlNetUnionModel