diffusers/pr_12509/en/api/models/consistency_decoder_vae.md

Consistency Decoder

Consistency decoder can be used to decode the latents from the denoising UNet in the StableDiffusionPipeline. This decoder was introduced in the DALL-E 3 technical report.

The original codebase can be found at openai/consistencydecoder.

Inference is only supported for 2 iterations as of now.

The pipeline could not have been contributed without the help of madebyollin and mrsteyk from this issue.

ConsistencyDecoderVAE[[diffusers.ConsistencyDecoderVAE]]

class diffusers.ConsistencyDecoderVAEdiffusers.ConsistencyDecoderVAEhttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/models/autoencoders/consistency_decoder_vae.py#L52[{"name": "scaling_factor", "val": ": float = 0.18215"}, {"name": "latent_channels", "val": ": int = 4"}, {"name": "sample_size", "val": ": int = 32"}, {"name": "encoder_act_fn", "val": ": str = 'silu'"}, {"name": "encoder_block_out_channels", "val": ": typing.Tuple[int, ...] = (128, 256, 512, 512)"}, {"name": "encoder_double_z", "val": ": bool = True"}, {"name": "encoder_down_block_types", "val": ": typing.Tuple[str, ...] = ('DownEncoderBlock2D', 'DownEncoderBlock2D', 'DownEncoderBlock2D', 'DownEncoderBlock2D')"}, {"name": "encoder_in_channels", "val": ": int = 3"}, {"name": "encoder_layers_per_block", "val": ": int = 2"}, {"name": "encoder_norm_num_groups", "val": ": int = 32"}, {"name": "encoder_out_channels", "val": ": int = 4"}, {"name": "decoder_add_attention", "val": ": bool = False"}, {"name": "decoder_block_out_channels", "val": ": typing.Tuple[int, ...] = (320, 640, 1024, 1024)"}, {"name": "decoder_down_block_types", "val": ": typing.Tuple[str, ...] = ('ResnetDownsampleBlock2D', 'ResnetDownsampleBlock2D', 'ResnetDownsampleBlock2D', 'ResnetDownsampleBlock2D')"}, {"name": "decoder_downsample_padding", "val": ": int = 1"}, {"name": "decoder_in_channels", "val": ": int = 7"}, {"name": "decoder_layers_per_block", "val": ": int = 3"}, {"name": "decoder_norm_eps", "val": ": float = 1e-05"}, {"name": "decoder_norm_num_groups", "val": ": int = 32"}, {"name": "decoder_num_train_timesteps", "val": ": int = 1024"}, {"name": "decoder_out_channels", "val": ": int = 6"}, {"name": "decoder_resnet_time_scale_shift", "val": ": str = 'scale_shift'"}, {"name": "decoder_time_embedding_type", "val": ": str = 'learned'"}, {"name": "decoder_up_block_types", "val": ": typing.Tuple[str, ...] = ('ResnetUpsampleBlock2D', 'ResnetUpsampleBlock2D', 'ResnetUpsampleBlock2D', 'ResnetUpsampleBlock2D')"}]

The consistency decoder used with DALL-E 3.

Examples:

>>> import torch
>>> from diffusers import StableDiffusionPipeline, ConsistencyDecoderVAE

>>> vae = ConsistencyDecoderVAE.from_pretrained("openai/consistency-decoder", torch_dtype=torch.float16)
>>> pipe = StableDiffusionPipeline.from_pretrained(
...     "stable-diffusion-v1-5/stable-diffusion-v1-5", vae=vae, torch_dtype=torch.float16
... ).to("cuda")

>>> image = pipe("horse", generator=torch.manual_seed(0)).images[0]
>>> image

wrapperdiffusers.ConsistencyDecoderVAE.decodehttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/utils/accelerate_utils.py#L43[{"name": "*args", "val": ""}, {"name": "**kwargs", "val": ""}]

disable_slicingdiffusers.ConsistencyDecoderVAE.disable_slicinghttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/models/autoencoders/consistency_decoder_vae.py#L196[]

Disable sliced VAE decoding. If enable_slicing was previously enabled, this method will go back to computing decoding in one step.

disable_tilingdiffusers.ConsistencyDecoderVAE.disable_tilinghttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/models/autoencoders/consistency_decoder_vae.py#L180[]

Disable tiled VAE decoding. If enable_tiling was previously enabled, this method will go back to computing decoding in one step.

enable_slicingdiffusers.ConsistencyDecoderVAE.enable_slicinghttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/models/autoencoders/consistency_decoder_vae.py#L188[]

Enable sliced VAE decoding. When this option is enabled, the VAE will split the input tensor in slices to compute decoding in several steps. This is useful to save some memory and allow larger batch sizes.

enable_tilingdiffusers.ConsistencyDecoderVAE.enable_tilinghttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/models/autoencoders/consistency_decoder_vae.py#L171[{"name": "use_tiling", "val": ": bool = True"}]

Enable tiled VAE decoding. When this option is enabled, the VAE will split the input tensor into tiles to compute decoding and encoding in several steps. This is useful for saving a large amount of memory and to allow processing larger images.

forwarddiffusers.ConsistencyDecoderVAE.forwardhttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/models/autoencoders/consistency_decoder_vae.py#L430[{"name": "sample", "val": ": Tensor"}, {"name": "sample_posterior", "val": ": bool = False"}, {"name": "return_dict", "val": ": bool = True"}, {"name": "generator", "val": ": typing.Optional[torch._C.Generator] = None"}]- sample (torch.Tensor) -- Input sample.

• sample_posterior (bool, optional, defaults to False) -- Whether to sample from the posterior.
• return_dict (bool, optional, defaults to True) -- Whether or not to return a DecoderOutput instead of a plain tuple.
• generator (torch.Generator, optional, defaults to None) -- Generator to use for sampling.0DecoderOutput or tupleIf return_dict is True, a DecoderOutput is returned, otherwise a plain tuple is returned.

set_attn_processordiffusers.ConsistencyDecoderVAE.set_attn_processorhttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/models/autoencoders/consistency_decoder_vae.py#L229[{"name": "processor", "val": ": typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]]"}]- processor (dict of AttentionProcessor or only AttentionProcessor) -- The instantiated processor class or a dictionary of processor classes that will be set as the processor for all Attention layers.

If processor is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors.0

Sets the attention processor to use to compute attention.

set_default_attn_processordiffusers.ConsistencyDecoderVAE.set_default_attn_processorhttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/models/autoencoders/consistency_decoder_vae.py#L264[]

Disables custom attention processors and sets the default attention implementation.

tiled_encodediffusers.ConsistencyDecoderVAE.tiled_encodehttps://github.com/huggingface/diffusers/blob/vr_12509/src/diffusers/models/autoencoders/consistency_decoder_vae.py#L375[{"name": "x", "val": ": Tensor"}, {"name": "return_dict", "val": ": bool = True"}]- x (torch.Tensor) -- Input batch of images.

• return_dict (bool, optional, defaults to True) -- Whether or not to return a ConsistencyDecoderVAEOutput instead of a plain tuple.0ConsistencyDecoderVAEOutput or tupleIf return_dict is True, a ConsistencyDecoderVAEOutput is returned, otherwise a plain tuple is returned. Encode a batch of images using a tiled encoder.

When this option is enabled, the VAE will split the input tensor into tiles to compute encoding in several steps. This is useful to keep memory use constant regardless of image size. The end result of tiled encoding is different from non-tiled encoding because each tile uses a different encoder. To avoid tiling artifacts, the tiles overlap and are blended together to form a smooth output. You may still see tile-sized changes in the output, but they should be much less noticeable.