diffusers/pr_12595/en/api/parallel.md

# Parallelism

Parallelism strategies help speed up diffusion transformers by distributing computations across multiple devices, allowing for faster inference/training times. Refer to the [Distributed inferece](../training/distributed_inference) guide to learn more.

## ParallelConfig[[diffusers.ParallelConfig]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.ParallelConfig</name><anchor>diffusers.ParallelConfig</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/_modeling_parallel.py#L108</source><parameters>[{"name": "context_parallel_config", "val": ": typing.Optional[diffusers.models._modeling_parallel.ContextParallelConfig] = None"}, {"name": "_rank", "val": ": int = None"}, {"name": "_world_size", "val": ": int = None"}, {"name": "_device", "val": ": device = None"}, {"name": "_cp_mesh", "val": ": DeviceMesh = None"}]</parameters><paramsdesc>- **context_parallel_config** (`ContextParallelConfig`, *optional*) --
  Configuration for context parallelism.</paramsdesc><paramgroups>0</paramgroups></docstring>

Configuration for applying different parallelisms.




</div>

## ContextParallelConfig[[diffusers.ContextParallelConfig]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.ContextParallelConfig</name><anchor>diffusers.ContextParallelConfig</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/_modeling_parallel.py#L41</source><parameters>[{"name": "ring_degree", "val": ": typing.Optional[int] = None"}, {"name": "ulysses_degree", "val": ": typing.Optional[int] = None"}, {"name": "convert_to_fp32", "val": ": bool = True"}, {"name": "rotate_method", "val": ": typing.Literal['allgather', 'alltoall'] = 'allgather'"}, {"name": "_rank", "val": ": int = None"}, {"name": "_world_size", "val": ": int = None"}, {"name": "_device", "val": ": device = None"}, {"name": "_mesh", "val": ": DeviceMesh = None"}, {"name": "_flattened_mesh", "val": ": DeviceMesh = None"}, {"name": "_ring_mesh", "val": ": DeviceMesh = None"}, {"name": "_ulysses_mesh", "val": ": DeviceMesh = None"}, {"name": "_ring_local_rank", "val": ": int = None"}, {"name": "_ulysses_local_rank", "val": ": int = None"}]</parameters><paramsdesc>- **ring_degree** (`int`, *optional*, defaults to `1`) --
  Number of devices to use for ring attention within a context parallel region. Must be a divisor of the
  total number of devices in the context parallel mesh.
- **ulysses_degree** (`int`, *optional*, defaults to `1`) --
  Number of devices to use for ulysses attention within a context parallel region. Must be a divisor of the
  total number of devices in the context parallel mesh.
- **convert_to_fp32** (`bool`, *optional*, defaults to `True`) --
  Whether to convert output and LSE to float32 for ring attention numerical stability.
- **rotate_method** (`str`, *optional*, defaults to `"allgather"`) --
  Method to use for rotating key/value states across devices in ring attention. Currently, only `"allgather"`
  is supported.</paramsdesc><paramgroups>0</paramgroups></docstring>

Configuration for context parallelism.




</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>diffusers.hooks.apply_context_parallel</name><anchor>diffusers.hooks.apply_context_parallel</anchor><source>https://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/hooks/context_parallel.py#L78</source><parameters>[{"name": "module", "val": ": Module"}, {"name": "parallel_config", "val": ": ContextParallelConfig"}, {"name": "plan", "val": ": typing.Dict[str, typing.Dict[str, typing.Union[typing.Dict[typing.Union[str, int], typing.Union[diffusers.models._modeling_parallel.ContextParallelInput, typing.List[diffusers.models._modeling_parallel.ContextParallelInput], typing.Tuple[diffusers.models._modeling_parallel.ContextParallelInput, ...]]], diffusers.models._modeling_parallel.ContextParallelOutput, typing.List[diffusers.models._modeling_parallel.ContextParallelOutput], typing.Tuple[diffusers.models._modeling_parallel.ContextParallelOutput, ...]]]]"}]</parameters></docstring>
Apply context parallel on a model.

</div>

<EditOnGithub source="https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/parallel.md" />