diffusers/pr_12595/en/api/normalization.md

Normalization layers

Customized normalization layers for supporting various models in 🤗 Diffusers.

AdaLayerNorm[[diffusers.models.normalization.AdaLayerNorm]]

class diffusers.models.normalization.AdaLayerNormdiffusers.models.normalization.AdaLayerNormhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L28[{"name": "embedding_dim", "val": ": int"}, {"name": "num_embeddings", "val": ": typing.Optional[int] = None"}, {"name": "output_dim", "val": ": typing.Optional[int] = None"}, {"name": "norm_elementwise_affine", "val": ": bool = False"}, {"name": "norm_eps", "val": ": float = 1e-05"}, {"name": "chunk_dim", "val": ": int = 0"}]- embedding_dim (int) -- The size of each embedding vector.

• num_embeddings (int, optional) -- The size of the embeddings dictionary.
• output_dim (int, optional) --
• norm_elementwise_affine (bool, defaults to `False) --
• norm_eps (bool, defaults to False) --
• chunk_dim (int, defaults to 0) --0

Norm layer modified to incorporate timestep embeddings.

AdaLayerNormZero[[diffusers.models.normalization.AdaLayerNormZero]]

class diffusers.models.normalization.AdaLayerNormZerodiffusers.models.normalization.AdaLayerNormZerohttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L131[{"name": "embedding_dim", "val": ": int"}, {"name": "num_embeddings", "val": ": typing.Optional[int] = None"}, {"name": "norm_type", "val": " = 'layer_norm'"}, {"name": "bias", "val": " = True"}]- embedding_dim (int) -- The size of each embedding vector.

• num_embeddings (int) -- The size of the embeddings dictionary.0

Norm layer adaptive layer norm zero (adaLN-Zero).

AdaLayerNormSingle[[diffusers.models.normalization.AdaLayerNormSingle]]

class diffusers.models.normalization.AdaLayerNormSinglediffusers.models.normalization.AdaLayerNormSinglehttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L236[{"name": "embedding_dim", "val": ": int"}, {"name": "use_additional_conditions", "val": ": bool = False"}]- embedding_dim (int) -- The size of each embedding vector.

• use_additional_conditions (bool) -- To use additional conditions for normalization or not.0

Norm layer adaptive layer norm single (adaLN-single).

As proposed in PixArt-Alpha (see: https://huggingface.co/papers/2310.00426; Section 2.3).

AdaGroupNorm[[diffusers.models.normalization.AdaGroupNorm]]

class diffusers.models.normalization.AdaGroupNormdiffusers.models.normalization.AdaGroupNormhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L270[{"name": "embedding_dim", "val": ": int"}, {"name": "out_dim", "val": ": int"}, {"name": "num_groups", "val": ": int"}, {"name": "act_fn", "val": ": typing.Optional[str] = None"}, {"name": "eps", "val": ": float = 1e-05"}]- embedding_dim (int) -- The size of each embedding vector.

• num_embeddings (int) -- The size of the embeddings dictionary.
• num_groups (int) -- The number of groups to separate the channels into.
• act_fn (str, optional, defaults to None) -- The activation function to use.
• eps (float, optional, defaults to 1e-5) -- The epsilon value to use for numerical stability.0

GroupNorm layer modified to incorporate timestep embeddings.

AdaLayerNormContinuous[[diffusers.models.normalization.AdaLayerNormContinuous]]

class diffusers.models.normalization.AdaLayerNormContinuousdiffusers.models.normalization.AdaLayerNormContinuoushttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L308[{"name": "embedding_dim", "val": ": int"}, {"name": "conditioning_embedding_dim", "val": ": int"}, {"name": "elementwise_affine", "val": " = True"}, {"name": "eps", "val": " = 1e-05"}, {"name": "bias", "val": " = True"}, {"name": "norm_type", "val": " = 'layer_norm'"}]- embedding_dim (int) -- Embedding dimension to use during projection.

• conditioning_embedding_dim (int) -- Dimension of the input condition.
• elementwise_affine (bool, defaults to True) -- Boolean flag to denote if affine transformation should be applied.
• eps (float, defaults to 1e-5) -- Epsilon factor.
• bias (bias, defaults to True) -- Boolean flag to denote if bias should be use.
• norm_type (str, defaults to "layer_norm") -- Normalization layer to use. Values supported: "layer_norm", "rms_norm".0

Adaptive normalization layer with a norm layer (layer_norm or rms_norm).

RMSNorm[[diffusers.models.normalization.RMSNorm]]

class diffusers.models.normalization.RMSNormdiffusers.models.normalization.RMSNormhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L511[{"name": "dim", "val": ""}, {"name": "eps", "val": ": float"}, {"name": "elementwise_affine", "val": ": bool = True"}, {"name": "bias", "val": ": bool = False"}]- dim (int) -- Number of dimensions to use for weights. Only effective when elementwise_affine is True.

• eps (float) -- Small value to use when calculating the reciprocal of the square-root.
• elementwise_affine (bool, defaults to True) -- Boolean flag to denote if affine transformation should be applied.
• bias (bool, defaults to False) -- If also training the bias param.0

RMS Norm as introduced in https://huggingface.co/papers/1910.07467 by Zhang et al.

GlobalResponseNorm[[diffusers.models.normalization.GlobalResponseNorm]]

class diffusers.models.normalization.GlobalResponseNormdiffusers.models.normalization.GlobalResponseNormhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L601[{"name": "dim", "val": ""}]- dim (int) -- Number of dimensions to use for the gamma and beta.0

Global response normalization as introduced in ConvNeXt-v2 (https://huggingface.co/papers/2301.00808).

LuminaLayerNormContinuous[[diffusers.models.normalization.LuminaLayerNormContinuous]]

class diffusers.models.normalization.LuminaLayerNormContinuousdiffusers.models.normalization.LuminaLayerNormContinuoushttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L355[{"name": "embedding_dim", "val": ": int"}, {"name": "conditioning_embedding_dim", "val": ": int"}, {"name": "elementwise_affine", "val": " = True"}, {"name": "eps", "val": " = 1e-05"}, {"name": "bias", "val": " = True"}, {"name": "norm_type", "val": " = 'layer_norm'"}, {"name": "out_dim", "val": ": typing.Optional[int] = None"}]

SD35AdaLayerNormZeroX[[diffusers.models.normalization.SD35AdaLayerNormZeroX]]

class diffusers.models.normalization.SD35AdaLayerNormZeroXdiffusers.models.normalization.SD35AdaLayerNormZeroXhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L97[{"name": "embedding_dim", "val": ": int"}, {"name": "norm_type", "val": ": str = 'layer_norm'"}, {"name": "bias", "val": ": bool = True"}]- embedding_dim (int) -- The size of each embedding vector.

• num_embeddings (int) -- The size of the embeddings dictionary.0

Norm layer adaptive layer norm zero (AdaLN-Zero).

AdaLayerNormZeroSingle[[diffusers.models.normalization.AdaLayerNormZeroSingle]]

class diffusers.models.normalization.AdaLayerNormZeroSinglediffusers.models.normalization.AdaLayerNormZeroSinglehttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L174[{"name": "embedding_dim", "val": ": int"}, {"name": "norm_type", "val": " = 'layer_norm'"}, {"name": "bias", "val": " = True"}]- embedding_dim (int) -- The size of each embedding vector.

• num_embeddings (int) -- The size of the embeddings dictionary.0

Norm layer adaptive layer norm zero (adaLN-Zero).

LuminaRMSNormZero[[diffusers.models.normalization.LuminaRMSNormZero]]

class diffusers.models.normalization.LuminaRMSNormZerodiffusers.models.normalization.LuminaRMSNormZerohttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L206[{"name": "embedding_dim", "val": ": int"}, {"name": "norm_eps", "val": ": float"}, {"name": "norm_elementwise_affine", "val": ": bool"}]- embedding_dim (int) -- The size of each embedding vector.0

Norm layer adaptive RMS normalization zero.

LpNorm[[diffusers.models.normalization.LpNorm]]

class diffusers.models.normalization.LpNormdiffusers.models.normalization.LpNormhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L621[{"name": "p", "val": ": int = 2"}, {"name": "dim", "val": ": int = -1"}, {"name": "eps", "val": ": float = 1e-12"}]

CogView3PlusAdaLayerNormZeroTextImage[[diffusers.models.normalization.CogView3PlusAdaLayerNormZeroTextImage]]

class diffusers.models.normalization.CogView3PlusAdaLayerNormZeroTextImagediffusers.models.normalization.CogView3PlusAdaLayerNormZeroTextImagehttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L404[{"name": "embedding_dim", "val": ": int"}, {"name": "dim", "val": ": int"}]- embedding_dim (int) -- The size of each embedding vector.

• num_embeddings (int) -- The size of the embeddings dictionary.0

Norm layer adaptive layer norm zero (adaLN-Zero).

CogVideoXLayerNormZero[[diffusers.models.normalization.CogVideoXLayerNormZero]]

class diffusers.models.normalization.CogVideoXLayerNormZerodiffusers.models.normalization.CogVideoXLayerNormZerohttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L449[{"name": "conditioning_dim", "val": ": int"}, {"name": "embedding_dim", "val": ": int"}, {"name": "elementwise_affine", "val": ": bool = True"}, {"name": "eps", "val": ": float = 1e-05"}, {"name": "bias", "val": ": bool = True"}]

MochiRMSNormZero[[diffusers.models.transformers.transformer_mochi.MochiRMSNormZero]]

class diffusers.models.transformers.transformer_mochi.MochiRMSNormZerodiffusers.models.transformers.transformer_mochi.MochiRMSNormZerohttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/transformers/transformer_mochi.py#L88[{"name": "embedding_dim", "val": ": int"}, {"name": "hidden_dim", "val": ": int"}, {"name": "eps", "val": ": float = 1e-05"}, {"name": "elementwise_affine", "val": ": bool = False"}]- embedding_dim (int) -- The size of each embedding vector.0

Adaptive RMS Norm used in Mochi.

MochiRMSNorm[[diffusers.models.normalization.MochiRMSNorm]]

class diffusers.models.normalization.MochiRMSNormdiffusers.models.normalization.MochiRMSNormhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L573[{"name": "dim", "val": ""}, {"name": "eps", "val": ": float"}, {"name": "elementwise_affine", "val": ": bool = True"}]