diffusers/pr_12595/en/api/normalization.md

# Normalization layers

Customized normalization layers for supporting various models in 🤗 Diffusers.

## AdaLayerNorm[[diffusers.models.normalization.AdaLayerNorm]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.AdaLayerNorm</name><anchor>diffusers.models.normalization.AdaLayerNorm</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L28</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "num_embeddings", "val": ": typing.Optional[int] = None"}, {"name": "output_dim", "val": ": typing.Optional[int] = None"}, {"name": "norm_elementwise_affine", "val": ": bool = False"}, {"name": "norm_eps", "val": ": float = 1e-05"}, {"name": "chunk_dim", "val": ": int = 0"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- The size of each embedding vector.
- **num_embeddings** (`int`, *optional*) -- The size of the embeddings dictionary.
- **output_dim** (`int`, *optional*) --
- **norm_elementwise_affine** (`bool`, defaults to `False) --
- **norm_eps** (`bool`, defaults to `False`) --
- **chunk_dim** (`int`, defaults to `0`) --</paramsdesc><paramgroups>0</paramgroups></docstring>

Norm layer modified to incorporate timestep embeddings.




</div>

## AdaLayerNormZero[[diffusers.models.normalization.AdaLayerNormZero]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.AdaLayerNormZero</name><anchor>diffusers.models.normalization.AdaLayerNormZero</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L131</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "num_embeddings", "val": ": typing.Optional[int] = None"}, {"name": "norm_type", "val": " = 'layer_norm'"}, {"name": "bias", "val": " = True"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- The size of each embedding vector.
- **num_embeddings** (`int`) -- The size of the embeddings dictionary.</paramsdesc><paramgroups>0</paramgroups></docstring>

Norm layer adaptive layer norm zero (adaLN-Zero).




</div>

## AdaLayerNormSingle[[diffusers.models.normalization.AdaLayerNormSingle]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.AdaLayerNormSingle</name><anchor>diffusers.models.normalization.AdaLayerNormSingle</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L236</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "use_additional_conditions", "val": ": bool = False"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- The size of each embedding vector.
- **use_additional_conditions** (`bool`) -- To use additional conditions for normalization or not.</paramsdesc><paramgroups>0</paramgroups></docstring>

Norm layer adaptive layer norm single (adaLN-single).

As proposed in PixArt-Alpha (see: https://huggingface.co/papers/2310.00426; Section 2.3).




</div>

## AdaGroupNorm[[diffusers.models.normalization.AdaGroupNorm]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.AdaGroupNorm</name><anchor>diffusers.models.normalization.AdaGroupNorm</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L270</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "out_dim", "val": ": int"}, {"name": "num_groups", "val": ": int"}, {"name": "act_fn", "val": ": typing.Optional[str] = None"}, {"name": "eps", "val": ": float = 1e-05"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- The size of each embedding vector.
- **num_embeddings** (`int`) -- The size of the embeddings dictionary.
- **num_groups** (`int`) -- The number of groups to separate the channels into.
- **act_fn** (`str`, *optional*, defaults to `None`) -- The activation function to use.
- **eps** (`float`, *optional*, defaults to `1e-5`) -- The epsilon value to use for numerical stability.</paramsdesc><paramgroups>0</paramgroups></docstring>

GroupNorm layer modified to incorporate timestep embeddings.




</div>

## AdaLayerNormContinuous[[diffusers.models.normalization.AdaLayerNormContinuous]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.AdaLayerNormContinuous</name><anchor>diffusers.models.normalization.AdaLayerNormContinuous</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L308</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "conditioning_embedding_dim", "val": ": int"}, {"name": "elementwise_affine", "val": " = True"}, {"name": "eps", "val": " = 1e-05"}, {"name": "bias", "val": " = True"}, {"name": "norm_type", "val": " = 'layer_norm'"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- Embedding dimension to use during projection.
- **conditioning_embedding_dim** (`int`) -- Dimension of the input condition.
- **elementwise_affine** (`bool`, defaults to `True`) --
  Boolean flag to denote if affine transformation should be applied.
- **eps** (`float`, defaults to 1e-5) -- Epsilon factor.
- **bias** (`bias`, defaults to `True`) -- Boolean flag to denote if bias should be use.
- **norm_type** (`str`, defaults to `"layer_norm"`) --
  Normalization layer to use. Values supported: "layer_norm", "rms_norm".</paramsdesc><paramgroups>0</paramgroups></docstring>

Adaptive normalization layer with a norm layer (layer_norm or rms_norm).




</div>

## RMSNorm[[diffusers.models.normalization.RMSNorm]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.RMSNorm</name><anchor>diffusers.models.normalization.RMSNorm</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L511</source><parameters>[{"name": "dim", "val": ""}, {"name": "eps", "val": ": float"}, {"name": "elementwise_affine", "val": ": bool = True"}, {"name": "bias", "val": ": bool = False"}]</parameters><paramsdesc>- **dim** (`int`) -- Number of dimensions to use for `weights`. Only effective when `elementwise_affine` is True.
- **eps** (`float`) -- Small value to use when calculating the reciprocal of the square-root.
- **elementwise_affine** (`bool`, defaults to `True`) --
  Boolean flag to denote if affine transformation should be applied.
- **bias** (`bool`, defaults to False) -- If also training the `bias` param.</paramsdesc><paramgroups>0</paramgroups></docstring>

RMS Norm as introduced in https://huggingface.co/papers/1910.07467 by Zhang et al.




</div>

## GlobalResponseNorm[[diffusers.models.normalization.GlobalResponseNorm]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.GlobalResponseNorm</name><anchor>diffusers.models.normalization.GlobalResponseNorm</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L601</source><parameters>[{"name": "dim", "val": ""}]</parameters><paramsdesc>- **dim** (`int`) -- Number of dimensions to use for the `gamma` and `beta`.</paramsdesc><paramgroups>0</paramgroups></docstring>

Global response normalization as introduced in ConvNeXt-v2 (https://huggingface.co/papers/2301.00808).




</div>

## LuminaLayerNormContinuous[[diffusers.models.normalization.LuminaLayerNormContinuous]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.LuminaLayerNormContinuous</name><anchor>diffusers.models.normalization.LuminaLayerNormContinuous</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L355</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "conditioning_embedding_dim", "val": ": int"}, {"name": "elementwise_affine", "val": " = True"}, {"name": "eps", "val": " = 1e-05"}, {"name": "bias", "val": " = True"}, {"name": "norm_type", "val": " = 'layer_norm'"}, {"name": "out_dim", "val": ": typing.Optional[int] = None"}]</parameters></docstring>


</div>

## SD35AdaLayerNormZeroX[[diffusers.models.normalization.SD35AdaLayerNormZeroX]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.SD35AdaLayerNormZeroX</name><anchor>diffusers.models.normalization.SD35AdaLayerNormZeroX</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L97</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "norm_type", "val": ": str = 'layer_norm'"}, {"name": "bias", "val": ": bool = True"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- The size of each embedding vector.
- **num_embeddings** (`int`) -- The size of the embeddings dictionary.</paramsdesc><paramgroups>0</paramgroups></docstring>

Norm layer adaptive layer norm zero (AdaLN-Zero).




</div>

## AdaLayerNormZeroSingle[[diffusers.models.normalization.AdaLayerNormZeroSingle]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.AdaLayerNormZeroSingle</name><anchor>diffusers.models.normalization.AdaLayerNormZeroSingle</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L174</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "norm_type", "val": " = 'layer_norm'"}, {"name": "bias", "val": " = True"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- The size of each embedding vector.
- **num_embeddings** (`int`) -- The size of the embeddings dictionary.</paramsdesc><paramgroups>0</paramgroups></docstring>

Norm layer adaptive layer norm zero (adaLN-Zero).




</div>

## LuminaRMSNormZero[[diffusers.models.normalization.LuminaRMSNormZero]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.LuminaRMSNormZero</name><anchor>diffusers.models.normalization.LuminaRMSNormZero</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L206</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "norm_eps", "val": ": float"}, {"name": "norm_elementwise_affine", "val": ": bool"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- The size of each embedding vector.</paramsdesc><paramgroups>0</paramgroups></docstring>

Norm layer adaptive RMS normalization zero.




</div>

## LpNorm[[diffusers.models.normalization.LpNorm]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.LpNorm</name><anchor>diffusers.models.normalization.LpNorm</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L621</source><parameters>[{"name": "p", "val": ": int = 2"}, {"name": "dim", "val": ": int = -1"}, {"name": "eps", "val": ": float = 1e-12"}]</parameters></docstring>


</div>

## CogView3PlusAdaLayerNormZeroTextImage[[diffusers.models.normalization.CogView3PlusAdaLayerNormZeroTextImage]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.CogView3PlusAdaLayerNormZeroTextImage</name><anchor>diffusers.models.normalization.CogView3PlusAdaLayerNormZeroTextImage</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L404</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "dim", "val": ": int"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- The size of each embedding vector.
- **num_embeddings** (`int`) -- The size of the embeddings dictionary.</paramsdesc><paramgroups>0</paramgroups></docstring>

Norm layer adaptive layer norm zero (adaLN-Zero).




</div>

## CogVideoXLayerNormZero[[diffusers.models.normalization.CogVideoXLayerNormZero]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.CogVideoXLayerNormZero</name><anchor>diffusers.models.normalization.CogVideoXLayerNormZero</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L449</source><parameters>[{"name": "conditioning_dim", "val": ": int"}, {"name": "embedding_dim", "val": ": int"}, {"name": "elementwise_affine", "val": ": bool = True"}, {"name": "eps", "val": ": float = 1e-05"}, {"name": "bias", "val": ": bool = True"}]</parameters></docstring>


</div>

## MochiRMSNormZero[[diffusers.models.transformers.transformer_mochi.MochiRMSNormZero]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.transformers.transformer_mochi.MochiRMSNormZero</name><anchor>diffusers.models.transformers.transformer_mochi.MochiRMSNormZero</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/transformers/transformer_mochi.py#L88</source><parameters>[{"name": "embedding_dim", "val": ": int"}, {"name": "hidden_dim", "val": ": int"}, {"name": "eps", "val": ": float = 1e-05"}, {"name": "elementwise_affine", "val": ": bool = False"}]</parameters><paramsdesc>- **embedding_dim** (`int`) -- The size of each embedding vector.</paramsdesc><paramgroups>0</paramgroups></docstring>

Adaptive RMS Norm used in Mochi.




</div>

## MochiRMSNorm[[diffusers.models.normalization.MochiRMSNorm]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.normalization.MochiRMSNorm</name><anchor>diffusers.models.normalization.MochiRMSNorm</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/normalization.py#L573</source><parameters>[{"name": "dim", "val": ""}, {"name": "eps", "val": ": float"}, {"name": "elementwise_affine", "val": ": bool = True"}]</parameters></docstring>


</div>

<EditOnGithub source="https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/normalization.md" />