diffusers/pr_12595/en/api/parallel.md

Parallelism

Parallelism strategies help speed up diffusion transformers by distributing computations across multiple devices, allowing for faster inference/training times. Refer to the Distributed inferece guide to learn more.

ParallelConfig[[diffusers.ParallelConfig]]

class diffusers.ParallelConfigdiffusers.ParallelConfighttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/_modeling_parallel.py#L108[{"name": "context_parallel_config", "val": ": typing.Optional[diffusers.models._modeling_parallel.ContextParallelConfig] = None"}, {"name": "_rank", "val": ": int = None"}, {"name": "_world_size", "val": ": int = None"}, {"name": "_device", "val": ": device = None"}, {"name": "_cp_mesh", "val": ": DeviceMesh = None"}]- context_parallel_config (ContextParallelConfig, optional) -- Configuration for context parallelism.0

Configuration for applying different parallelisms.

ContextParallelConfig[[diffusers.ContextParallelConfig]]

class diffusers.ContextParallelConfigdiffusers.ContextParallelConfighttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/_modeling_parallel.py#L41[{"name": "ring_degree", "val": ": typing.Optional[int] = None"}, {"name": "ulysses_degree", "val": ": typing.Optional[int] = None"}, {"name": "convert_to_fp32", "val": ": bool = True"}, {"name": "rotate_method", "val": ": typing.Literal['allgather', 'alltoall'] = 'allgather'"}, {"name": "_rank", "val": ": int = None"}, {"name": "_world_size", "val": ": int = None"}, {"name": "_device", "val": ": device = None"}, {"name": "_mesh", "val": ": DeviceMesh = None"}, {"name": "_flattened_mesh", "val": ": DeviceMesh = None"}, {"name": "_ring_mesh", "val": ": DeviceMesh = None"}, {"name": "_ulysses_mesh", "val": ": DeviceMesh = None"}, {"name": "_ring_local_rank", "val": ": int = None"}, {"name": "_ulysses_local_rank", "val": ": int = None"}]- ring_degree (int, optional, defaults to 1) -- Number of devices to use for ring attention within a context parallel region. Must be a divisor of the total number of devices in the context parallel mesh.

• ulysses_degree (int, optional, defaults to 1) -- Number of devices to use for ulysses attention within a context parallel region. Must be a divisor of the total number of devices in the context parallel mesh.
• convert_to_fp32 (bool, optional, defaults to True) -- Whether to convert output and LSE to float32 for ring attention numerical stability.
• rotate_method (str, optional, defaults to "allgather") -- Method to use for rotating key/value states across devices in ring attention. Currently, only "allgather" is supported.0

Configuration for context parallelism.

diffusers.hooks.apply_context_paralleldiffusers.hooks.apply_context_parallelhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/hooks/context_parallel.py#L78[{"name": "module", "val": ": Module"}, {"name": "parallel_config", "val": ": ContextParallelConfig"}, {"name": "plan", "val": ": typing.Dict[str, typing.Dict[str, typing.Union[typing.Dict[typing.Union[str, int], typing.Union[diffusers.models._modeling_parallel.ContextParallelInput, typing.List[diffusers.models._modeling_parallel.ContextParallelInput], typing.Tuple[diffusers.models._modeling_parallel.ContextParallelInput, ...]]], diffusers.models._modeling_parallel.ContextParallelOutput, typing.List[diffusers.models._modeling_parallel.ContextParallelOutput], typing.Tuple[diffusers.models._modeling_parallel.ContextParallelOutput, ...]]]]"}] Apply context parallel on a model.