diffusers/pr_12595/en/api/activations.md

# Activation functions

Customized activation functions for supporting various models in 🤗 Diffusers.

## GELU[[diffusers.models.activations.GELU]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.activations.GELU</name><anchor>diffusers.models.activations.GELU</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/activations.py#L65</source><parameters>[{"name": "dim_in", "val": ": int"}, {"name": "dim_out", "val": ": int"}, {"name": "approximate", "val": ": str = 'none'"}, {"name": "bias", "val": ": bool = True"}]</parameters><paramsdesc>- **dim_in** (`int`) -- The number of channels in the input.
- **dim_out** (`int`) -- The number of channels in the output.
- **approximate** (`str`, *optional*, defaults to `"none"`) -- If `"tanh"`, use tanh approximation.
- **bias** (`bool`, defaults to True) -- Whether to use a bias in the linear layer.</paramsdesc><paramgroups>0</paramgroups></docstring>

GELU activation function with tanh approximation support with `approximate="tanh"`.




</div>

## GEGLU[[diffusers.models.activations.GEGLU]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.activations.GEGLU</name><anchor>diffusers.models.activations.GEGLU</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/activations.py#L93</source><parameters>[{"name": "dim_in", "val": ": int"}, {"name": "dim_out", "val": ": int"}, {"name": "bias", "val": ": bool = True"}]</parameters><paramsdesc>- **dim_in** (`int`) -- The number of channels in the input.
- **dim_out** (`int`) -- The number of channels in the output.
- **bias** (`bool`, defaults to True) -- Whether to use a bias in the linear layer.</paramsdesc><paramgroups>0</paramgroups></docstring>

A [variant](https://huggingface.co/papers/2002.05202) of the gated linear unit activation function.




</div>

## ApproximateGELU[[diffusers.models.activations.ApproximateGELU]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.activations.ApproximateGELU</name><anchor>diffusers.models.activations.ApproximateGELU</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/activations.py#L149</source><parameters>[{"name": "dim_in", "val": ": int"}, {"name": "dim_out", "val": ": int"}, {"name": "bias", "val": ": bool = True"}]</parameters><paramsdesc>- **dim_in** (`int`) -- The number of channels in the input.
- **dim_out** (`int`) -- The number of channels in the output.
- **bias** (`bool`, defaults to True) -- Whether to use a bias in the linear layer.</paramsdesc><paramgroups>0</paramgroups></docstring>

The approximate form of the Gaussian Error Linear Unit (GELU). For more details, see section 2 of this
[paper](https://huggingface.co/papers/1606.08415).




</div>

## SwiGLU[[diffusers.models.activations.SwiGLU]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.activations.SwiGLU</name><anchor>diffusers.models.activations.SwiGLU</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/activations.py#L126</source><parameters>[{"name": "dim_in", "val": ": int"}, {"name": "dim_out", "val": ": int"}, {"name": "bias", "val": ": bool = True"}]</parameters><paramsdesc>- **dim_in** (`int`) -- The number of channels in the input.
- **dim_out** (`int`) -- The number of channels in the output.
- **bias** (`bool`, defaults to True) -- Whether to use a bias in the linear layer.</paramsdesc><paramgroups>0</paramgroups></docstring>

A [variant](https://huggingface.co/papers/2002.05202) of the gated linear unit activation function. It's similar to
`GEGLU` but uses SiLU / Swish instead of GeLU.




</div>

## FP32SiLU[[diffusers.models.activations.FP32SiLU]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.activations.FP32SiLU</name><anchor>diffusers.models.activations.FP32SiLU</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/activations.py#L53</source><parameters>[]</parameters></docstring>

SiLU activation function with input upcasted to torch.float32.


</div>

## LinearActivation[[diffusers.models.activations.LinearActivation]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.models.activations.LinearActivation</name><anchor>diffusers.models.activations.LinearActivation</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/activations.py#L169</source><parameters>[{"name": "dim_in", "val": ": int"}, {"name": "dim_out", "val": ": int"}, {"name": "bias", "val": ": bool = True"}, {"name": "activation", "val": ": str = 'silu'"}]</parameters></docstring>


</div>

<EditOnGithub source="https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/activations.md" />