diffusers/pr_14539/en/api/pipelines/stable_audio_3.md

Stable Audio 3

Stable Audio 3 (SA3) is a text-to-audio model from Stability AI that generates high-quality stereo audio at 44.1 kHz. It uses a rectified-flow DiT conditioned on two signals:

• Text — encoded by a frozen T5Gemma encoder and injected via cross-attention.
• Duration — a float (seconds) embedded by StableAudio3DurationEmbedder and used as a global conditioning vector for adaptive layer normalisation.

Audio is decoded by the SAME (Semantically-Aligned Music Encoder) autoencoder, AutoencoderSAME.

Both checkpoints use FlowMatchEulerDiscreteScheduler with a log-SNR-uniform sigma schedule, differing only in stochastic_sampling and the default step count:

Checkpoint	diffusion_objective	stochastic_sampling	num_inference_steps
stable-audio-3-medium-base	rectified_flow	False (deterministic Euler)	100 (not distilled)
stable-audio-3-medium (distilled)	rf_denoiser	True (ping-pong re-noise)	8 (distilled for 8 steps)

The correct scheduler config is baked into each converted checkpoint, so num_inference_steps defaults to the right value when you leave it unset. Only pass it to override.

Original codebase: Stability-AI/stable-audio-3.

Converting original checkpoints

The Stability AI checkpoints are not published in diffusers format, so convert them locally. The script downloads the checkpoint's model_config.json and selects the scheduler from its diffusion_objective:

python scripts/convert_stable_audio_3_to_diffusers.py \
  --checkpoint_path stabilityai/stable-audio-3-medium-base \
  --text_encoder_repo google/t5gemma-b-b-ul2 \
  --output_dir /tmp/sa3-diffusers-euler \
  --dtype float32

stable-audio-3-medium-base is a gated repo. Run hf auth login with an account that has access before converting, otherwise the download fails with a 401.

Usage example

Load the converted checkpoint from its local output directory (install soundfile with pip install soundfile):

import torch
import soundfile as sf
from diffusers import StableAudio3Pipeline

pipe = StableAudio3Pipeline.from_pretrained("/tmp/sa3-diffusers-euler", torch_dtype=torch.float32)
pipe = pipe.to("cuda")

generator = torch.Generator("cuda").manual_seed(0)
audio = pipe(
    "A gentle piano melody with soft strings in a concert hall",
    duration=10.0,  # seconds; latent length is computed automatically
    generator=generator,
).audios

sf.write("sa3_output.wav", audio[0].T.cpu().float().numpy(), samplerate=44100)

The pipeline is also registered with AutoPipelineForText2Audio, which resolves the checkpoint to StableAudio3Pipeline automatically:

from diffusers import AutoPipelineForText2Audio

pipe = AutoPipelineForText2Audio.from_pretrained("/tmp/sa3-diffusers-euler", torch_dtype=torch.float32)

The examples use a local path because stabilityai/stable-audio-3-medium and stable-audio-3-medium-base are not yet published in diffusers format (loading by repo id returns a 404). Once published, the repo id works in place of the local path.

Tips

• Use torch.float32 on CPU or MPS (Apple Silicon) — torch.float16 on MPS produces noise.
• The distilled model (stable-audio-3-medium) is adversarially distilled — guidance is baked into the weights. Leave guidance_scale=1.0 (the default) and don't pass a negative_prompt for that checkpoint; both only do something useful for the non-distilled stable-audio-3-medium-base checkpoint.
• silence_padding_duration (default 0.0) adds silent headroom at the end of the latent sequence. Leave it at 0.0 unless the model is trained to mask that padding — otherwise the extra frames drain output energy and the result gets quiet.
• Set num_waveforms_per_prompt > 1 to generate multiple clips per prompt.

StableAudio3Pipeline[[diffusers.StableAudio3Pipeline]]

diffusers.StableAudio3Pipeline[[diffusers.StableAudio3Pipeline]]

diffusers.StableAudio3Pipeline(vae: AutoencoderSAME, text_encoder: T5GemmaEncoderModel, tokenizer: GemmaTokenizer, duration_embedder: StableAudio3DurationEmbedder, transformer: StableAudio3DiTModel, scheduler: FlowMatchEulerDiscreteScheduler)

Source

Parameters:

vae ([AutoencoderSAME]) : SAME autoencoder used to encode and decode audio latents.

text_encoder ([~transformers.T5GemmaEncoderModel]) : Frozen T5Gemma text encoder (google/t5gemma-b-b-ul2).

tokenizer ([~transformers.GemmaTokenizerFast]) : Tokenizer for the text encoder.

duration_embedder ([StableAudio3DurationEmbedder]) : Maps duration in seconds to a global conditioning vector for AdaLN in each DiT block.

transformer ([StableAudio3DiTModel]) : The rectified-flow velocity-prediction DiT.

scheduler ([FlowMatchEulerDiscreteScheduler]) : Scheduler for the iterative denoising loop. The production (distilled) SA3 Medium checkpoint uses stochastic_sampling=True for exactly 8 ping-pong steps; the non-distilled base checkpoint uses stochastic_sampling=False for ~100 deterministic Euler steps.

Pipeline for text-to-audio generation using Stable Audio 3.

SA3 uses a distilled rectified-flow DiT with ping-pong sampling. Classifier-free guidance (guidance_scale / negative_prompt) is unnecessary for the distilled checkpoint (leave guidance_scale=1.0, the default) but is meaningful for the non-distilled stable-audio-3-medium-base checkpoint.

This model inherits from [DiffusionPipeline]. Check the superclass documentation for the generic methods implemented for all pipelines.

call[[diffusers.StableAudio3Pipeline.call]]

__call__(prompt: typing.Union[str, typing.List[str], NoneType] = None, duration: float = 10.0, num_inference_steps: typing.Optional[int] = None, logsnr_min: float = -6.2, logsnr_max: float = 2.0, silence_padding_duration: float = 0.0, guidance_scale: float = 1.0, negative_prompt: typing.Union[str, typing.List[str], NoneType] = None, num_waveforms_per_prompt: int = 1, generator: typing.Union[torch.Generator, typing.List[torch.Generator], NoneType] = None, latents: typing.Optional[torch.Tensor] = None, prompt_embeds: typing.Optional[torch.Tensor] = None, encoder_attention_mask: typing.Optional[torch.LongTensor] = None, negative_prompt_embeds: typing.Optional[torch.Tensor] = None, negative_encoder_attention_mask: typing.Optional[torch.LongTensor] = None, return_dict: bool = True, callback_on_step_end: typing.Optional[typing.Callable[[int, int, dict], dict]] = None, callback_on_step_end_tensor_inputs: typing.List[str] = ['latents'], output_type: str = 'pt')

Source

Parameters:

prompt (str or list[str], optional) : Text prompt(s). Pass prompt_embeds instead to skip tokenization and encoding.

duration (float, defaults to 10.0) : Requested output duration in seconds.

num_inference_steps (int, optional) : Number of denoising steps. When None (default), the step count is chosen from the scheduler's stochastic_sampling config: 8 when True (the distilled ping-pong-style checkpoint) and 100 when False (the non-distilled base checkpoint). Pass an explicit value to override.

logsnr_min (float, defaults to -6.2) : Minimum log-SNR value for the noise schedule — maps to the high-noise start of the schedule.

logsnr_max (float, defaults to 2.0) : Maximum log-SNR value for the noise schedule — maps to the low-noise end of the schedule.

silence_padding_duration (float, defaults to 0.0) : Extra seconds of latent context generated beyond the target content, giving the model headroom at the boundary; the output is trimmed back to duration. Defaults to 0.0 (disabled). Increase only if the model is trained/distilled to mask this padding — otherwise the extra frames drain output energy.

guidance_scale (float, defaults to 1.0) : Classifier-free guidance scale. 1.0 disables guidance (the default, and the only sensible value for the distilled SA3 Medium checkpoint, whose CFG is baked into the weights). Values > 1.0 are meaningful for the non-distilled stable-audio-3-medium-base checkpoint; higher values follow the prompt more closely at the cost of diversity.

negative_prompt (str or list[str], optional) : Prompt(s) describing what to steer away from when guidance_scale > 1.0. Defaults to an empty string (unconditional) when guidance_scale > 1.0 and neither this nor negative_prompt_embeds is given. Ignored when guidance_scale <= 1.0.

num_waveforms_per_prompt (int, defaults to 1) : Number of waveforms to generate per prompt.

generator (torch.Generator or list[torch.Generator], optional) : For deterministic generation and reproducible re-noise in the ping-pong loop.

latents (torch.Tensor, optional) : Pre-generated starting latents. If None a fresh Gaussian tensor is sampled.

prompt_embeds (torch.Tensor, optional) : Pre-computed text embeddings (batch, seq_len, 768).

encoder_attention_mask (torch.LongTensor, optional) : Boolean mask for pre-computed embeddings.

negative_prompt_embeds (torch.Tensor, optional) : Pre-computed negative text embeddings, as an alternative to negative_prompt.

negative_encoder_attention_mask (torch.LongTensor, optional) : Boolean mask for pre-computed negative embeddings.

return_dict (bool, defaults to True) : Return an AudioPipelineOutput or a plain tuple.

callback_on_step_end (Callable, optional) : Called at the end of each denoising step with (self, step_idx, timestep, callback_kwargs), where callback_kwargs contains the tensors listed in callback_on_step_end_tensor_inputs. Must return a dict with the (optionally modified) tensors to use for the rest of the loop.

callback_on_step_end_tensor_inputs (list[str], defaults to ["latents"]) : The tensors passed to callback_on_step_end. Must be a subset of self._callback_tensor_inputs.

output_type (str, defaults to "pt") : "pt" for a PyTorch tensor, "np" for a NumPy array, or "latent" to skip decoding and return the raw latents.

Returns: [~pipelines.AudioPipelineOutput] or tuple

.audios is a tensor / array of shape (batch * num_waveforms_per_prompt, audio_channels, samples).

Generate audio from a text prompt.

Examples:

>>> import torch
>>> import soundfile as sf
>>> from diffusers import StableAudio3Pipeline

>>> pipe = StableAudio3Pipeline.from_pretrained("stabilityai/stable-audio-3-medium", torch_dtype=torch.float16)
>>> pipe = pipe.to("cuda")

>>> generator = torch.Generator("cuda").manual_seed(0)
>>> audio = pipe(
...     "A gentle piano melody with soft strings in a concert hall",
...     duration=10.0,
...     generator=generator,
... ).audios

>>> sf.write("output.wav", audio[0].T.cpu().float().numpy(), samplerate=pipe.vae.config.sampling_rate)

encode_duration[[diffusers.StableAudio3Pipeline.encode_duration]]

encode_duration(duration: float, device: device, num_waveforms_per_prompt: int, batch_size: int)

Source

Parameters:

duration : Duration in seconds, applied to every sample in the batch.

device : Target device.

num_waveforms_per_prompt : Tile factor.

batch_size : Number of prompts.

Returns:

(batch * num_waveforms_per_prompt, output_dim) tensor.

Embed the duration value into the global conditioning vector.

encode_prompt[[diffusers.StableAudio3Pipeline.encode_prompt]]

encode_prompt(prompt: typing.Union[str, typing.List[str], NoneType], device: device, num_waveforms_per_prompt: int, prompt_embeds: typing.Optional[torch.Tensor] = None, encoder_attention_mask: typing.Optional[torch.LongTensor] = None)

Source

Parameters:

prompt : Text prompt or list of prompts. Ignored when prompt_embeds is provided.

device : Target device.

num_waveforms_per_prompt : How many output waveforms to generate per prompt; conditioning tensors are tiled accordingly.

prompt_embeds : Pre-computed text embeddings (batch, seq_len, hidden_size).

encoder_attention_mask : Boolean mask (batch, seq_len) for pre-computed embeddings; 1 = real token, 0 = pad.

Returns:

(prompt_embeds, encoder_attention_mask) both tiled to batch * num_waveforms_per_prompt.

Encode text prompt(s) into cross-attention conditioning tensors.

prepare_cross_attention[[diffusers.StableAudio3Pipeline.prepare_cross_attention]]

prepare_cross_attention(prompt_embeds: Tensor, encoder_attention_mask: Tensor, global_hidden_states: Tensor)

Source

Returns:

(context, context_mask) of shapes (batch, T_text + 1, dim) and (batch, T_text + 1).

Build the cross-attention context by appending the duration embedding as an extra token.

SA3 routes the seconds_total conditioner to both the global (AdaLN) input and the cross-attention context (cross_attention_cond_ids = ["prompt", "seconds_total"]). The duration embedding is concatenated after the text tokens, and the attention mask is extended with one valid entry.

StableAudio3InpaintPipeline[[diffusers.StableAudio3InpaintPipeline]]

diffusers.StableAudio3InpaintPipeline[[diffusers.StableAudio3InpaintPipeline]]

diffusers.StableAudio3InpaintPipeline(vae: AutoencoderSAME, text_encoder: T5GemmaEncoderModel, tokenizer: GemmaTokenizer, duration_embedder: StableAudio3DurationEmbedder, transformer: StableAudio3DiTModel, scheduler: FlowMatchEulerDiscreteScheduler)

Source

Parameters:

vae ([AutoencoderSAME]) : SAME autoencoder used to encode and decode audio latents.

text_encoder ([~transformers.T5GemmaEncoderModel]) : Frozen T5Gemma text encoder (google/t5gemma-b-b-ul2).

tokenizer ([~transformers.GemmaTokenizerFast]) : Tokenizer for the text encoder.

duration_embedder ([StableAudio3DurationEmbedder]) : Maps duration in seconds to a global conditioning vector for AdaLN in each DiT block.

transformer ([StableAudio3DiTModel]) : The rectified-flow velocity-prediction DiT.

scheduler ([FlowMatchEulerDiscreteScheduler]) : Scheduler for the iterative denoising loop. The production (distilled) SA3 Medium checkpoint uses stochastic_sampling=True for exactly 8 ping-pong steps; the non-distilled base checkpoint uses stochastic_sampling=False for ~100 deterministic Euler steps.

Audio inpainting pipeline for Stable Audio 3.

Shares its text-to-audio logic with [StableAudio3Pipeline] (kept in sync via # Copied from). When audio and mask are provided, encodes the reference audio with the frozen SAME encoder and injects masked_latent ∥ mask as local-additive conditioning into each DiT block via the transformer's local_add_cond pathway (to_local_embed).

Call signature extension (see __call__): audio (torch.Tensor of shape (batch, channels, samples)): Reference audio waveform at vae.config.sampling_rate Hz. mask (torch.Tensor of shape (batch, 1, latent_length)): Per-frame binary mask in latent space. 1 = preserve original audio; 0 = region to be inpainted.

call[[diffusers.StableAudio3InpaintPipeline.call]]

__call__(prompt: typing.Union[str, typing.List[str], NoneType] = None, duration: float = 10.0, audio: typing.Optional[torch.Tensor] = None, mask: typing.Optional[torch.Tensor] = None, mask_start_seconds: typing.Union[float, typing.List[float], NoneType] = None, mask_end_seconds: typing.Union[float, typing.List[float], NoneType] = None, num_inference_steps: typing.Optional[int] = None, logsnr_min: float = -6.2, logsnr_max: float = 2.0, silence_padding_duration: float = 0.0, num_waveforms_per_prompt: int = 1, generator: typing.Union[torch.Generator, typing.List[torch.Generator], NoneType] = None, latents: typing.Optional[torch.Tensor] = None, prompt_embeds: typing.Optional[torch.Tensor] = None, encoder_attention_mask: typing.Optional[torch.LongTensor] = None, return_dict: bool = True, callback_on_step_end: typing.Optional[typing.Callable[[int, int, dict], dict]] = None, callback_on_step_end_tensor_inputs: typing.List[str] = ['latents'], output_type: str = 'pt')

Source

Parameters:

prompt (str or list[str], optional) : Text prompt(s).

duration (float, defaults to 10.0) : Output duration in seconds. Should match the reference audio.

audio (torch.Tensor, optional) : Reference waveform (batch, channels, samples) at vae.config.sampling_rate Hz. Required for inpainting.

mask (torch.Tensor, optional) : Per-frame latent-space mask (batch, 1, L) with 0 = inpaint region, 1 = preserve. Either mask or mask_start_seconds / mask_end_seconds must be provided.

mask_start_seconds (float or list[float], optional) : Start time(s) of the inpaint region in seconds.

mask_end_seconds (float or list[float], optional) : End time(s) of the inpaint region (must pair with mask_start_seconds).

num_inference_steps (int, optional) : Number of denoising steps. When None (default), the step count is chosen from the scheduler's stochastic_sampling config, matching [StableAudio3Pipeline].

logsnr_min (float, defaults to -6.2) : Minimum log-SNR value for the noise schedule — maps to the high-noise start of the schedule.

logsnr_max (float, defaults to 2.0) : Maximum log-SNR value for the noise schedule — maps to the low-noise end of the schedule.

silence_padding_duration (float, defaults to 0.0) : Extra latent headroom after the target content.

num_waveforms_per_prompt (int, defaults to 1) : Waveforms per prompt.

generator : RNG for reproducibility.

latents : Pre-generated starting noise (None → sample fresh).

prompt_embeds : Pre-computed text embeddings.

encoder_attention_mask : Mask for pre-computed embeddings.

return_dict (bool, defaults to True) : Return AudioPipelineOutput or tuple.

callback_on_step_end (Callable, optional) : Called at the end of each denoising step with (self, step_idx, timestep, callback_kwargs). Must return a dict with the (optionally modified) tensors to use for the rest of the loop.

callback_on_step_end_tensor_inputs (list[str], defaults to ["latents"]) : The tensors passed to callback_on_step_end. Must be a subset of self._callback_tensor_inputs.

output_type (str, defaults to "pt") : "pt" / "np" / "latent".

Returns:

[~pipelines.AudioPipelineOutput] with .audios.

Generate inpainted audio conditioned on a text prompt and reference.

Examples:

>>> import torch
>>> import soundfile as sf
>>> import torchaudio
>>> from diffusers import StableAudio3InpaintPipeline

>>> pipe = StableAudio3InpaintPipeline.from_pretrained(
...     "stabilityai/stable-audio-3-medium", torch_dtype=torch.float16
... )
>>> pipe = pipe.to("cuda")

>>> audio, sr = torchaudio.load("reference.wav")
>>> audio = torchaudio.functional.resample(audio, sr, pipe.vae.config.sampling_rate).unsqueeze(0).to("cuda")

>>> generator = torch.Generator("cuda").manual_seed(0)
>>> audio = pipe(
...     "A gentle piano melody with soft strings in a concert hall",
...     duration=10.0,
...     audio=audio,
...     mask_start_seconds=4.0,
...     mask_end_seconds=6.0,
...     generator=generator,
... ).audios

>>> sf.write("output.wav", audio[0].T.cpu().float().numpy(), samplerate=pipe.vae.config.sampling_rate)

encode_duration[[diffusers.StableAudio3InpaintPipeline.encode_duration]]

encode_duration(duration: float, device: device, num_waveforms_per_prompt: int, batch_size: int)

Source

Parameters:

duration : Duration in seconds, applied to every sample in the batch.

device : Target device.

num_waveforms_per_prompt : Tile factor.

batch_size : Number of prompts.

Returns:

(batch * num_waveforms_per_prompt, output_dim) tensor.

Embed the duration value into the global conditioning vector.

encode_prompt[[diffusers.StableAudio3InpaintPipeline.encode_prompt]]

encode_prompt(prompt: typing.Union[str, typing.List[str], NoneType], device: device, num_waveforms_per_prompt: int, prompt_embeds: typing.Optional[torch.Tensor] = None, encoder_attention_mask: typing.Optional[torch.LongTensor] = None)

Source

Parameters:

prompt : Text prompt or list of prompts. Ignored when prompt_embeds is provided.

device : Target device.

num_waveforms_per_prompt : How many output waveforms to generate per prompt; conditioning tensors are tiled accordingly.

prompt_embeds : Pre-computed text embeddings (batch, seq_len, hidden_size).

encoder_attention_mask : Boolean mask (batch, seq_len) for pre-computed embeddings; 1 = real token, 0 = pad.

Returns:

(prompt_embeds, encoder_attention_mask) both tiled to batch * num_waveforms_per_prompt.

Encode text prompt(s) into cross-attention conditioning tensors.

prepare_cross_attention[[diffusers.StableAudio3InpaintPipeline.prepare_cross_attention]]

prepare_cross_attention(prompt_embeds: Tensor, encoder_attention_mask: Tensor, global_hidden_states: Tensor)

Source

Returns:

(context, context_mask) of shapes (batch, T_text + 1, dim) and (batch, T_text + 1).

Build the cross-attention context by appending the duration embedding as an extra token.

SA3 routes the seconds_total conditioner to both the global (AdaLN) input and the cross-attention context (cross_attention_cond_ids = ["prompt", "seconds_total"]). The duration embedding is concatenated after the text tokens, and the attention mask is extended with one valid entry.

StableAudio3AudioToAudioPipeline[[diffusers.StableAudio3AudioToAudioPipeline]]

Generates a variation of a reference audio clip: the whole reference is noised to init_noise_level and denoised from there, unlike StableAudio3InpaintPipeline's per-frame local-additive conditioning which preserves specific frames exactly.

diffusers.StableAudio3AudioToAudioPipeline[[diffusers.StableAudio3AudioToAudioPipeline]]

diffusers.StableAudio3AudioToAudioPipeline(vae: AutoencoderSAME, text_encoder: T5GemmaEncoderModel, tokenizer: GemmaTokenizer, duration_embedder: StableAudio3DurationEmbedder, transformer: StableAudio3DiTModel, scheduler: FlowMatchEulerDiscreteScheduler)

Source

Parameters:

vae ([AutoencoderSAME]) : SAME autoencoder used to encode and decode audio latents.

text_encoder ([~transformers.T5GemmaEncoderModel]) : Frozen T5Gemma text encoder (google/t5gemma-b-b-ul2).

tokenizer ([~transformers.GemmaTokenizerFast]) : Tokenizer for the text encoder.

duration_embedder ([StableAudio3DurationEmbedder]) : Maps duration in seconds to a global conditioning vector for AdaLN in each DiT block.

transformer ([StableAudio3DiTModel]) : The rectified-flow velocity-prediction DiT.

scheduler ([FlowMatchEulerDiscreteScheduler]) : Scheduler for the iterative denoising loop. The production (distilled) SA3 Medium checkpoint uses stochastic_sampling=True for exactly 8 ping-pong steps; the non-distilled base checkpoint uses stochastic_sampling=False for ~100 deterministic Euler steps.

Audio-to-audio variation pipeline for Stable Audio 3.

Shares its text-to-audio logic with [StableAudio3Pipeline] (kept in sync via # Copied from). Encodes the reference audio with the frozen SAME encoder, mixes it with fresh noise according to init_noise_level, and denoises from there — the whole signal is noised/denoised globally, unlike [StableAudio3InpaintPipeline]'s per-frame local-additive conditioning.

Call signature extension (see __call__): audio (torch.Tensor of shape (batch, channels, samples)): Reference audio waveform at vae.config.sampling_rate Hz. init_noise_level (float): How much noise to mix into the reference before denoising. 1.0 = full noise (equivalent to text-to-audio); lower values retain more of the reference.

call[[diffusers.StableAudio3AudioToAudioPipeline.call]]

__call__(prompt: typing.Union[str, typing.List[str], NoneType] = None, duration: float = 10.0, audio: typing.Optional[torch.Tensor] = None, init_noise_level: float = 1.0, num_inference_steps: typing.Optional[int] = None, logsnr_min: float = -6.2, logsnr_max: float = 2.0, silence_padding_duration: float = 0.0, num_waveforms_per_prompt: int = 1, generator: typing.Union[torch.Generator, typing.List[torch.Generator], NoneType] = None, latents: typing.Optional[torch.Tensor] = None, prompt_embeds: typing.Optional[torch.Tensor] = None, encoder_attention_mask: typing.Optional[torch.LongTensor] = None, return_dict: bool = True, callback_on_step_end: typing.Optional[typing.Callable[[int, int, dict], dict]] = None, callback_on_step_end_tensor_inputs: typing.List[str] = ['latents'], output_type: str = 'pt')

Source

Parameters:

prompt (str or list[str], optional) : Text prompt(s).

duration (float, defaults to 10.0) : Output duration in seconds. Should match the reference audio.

audio (torch.Tensor, optional) : Reference waveform (batch, channels, samples) at vae.config.sampling_rate Hz. Required.

init_noise_level (float, defaults to 1.0) : Noise level (in (0, 1]) mixed into the reference before denoising: x_start = (1 - init_noise_level) * reference_latents + init_noise_level * noise. 1.0 discards the reference entirely (equivalent to [StableAudio3Pipeline]); lower values retain progressively more of the reference's structure while still running the full step count.

num_inference_steps (int, optional) : Number of denoising steps. When None (default), the step count is chosen from the scheduler's stochastic_sampling config, matching [StableAudio3Pipeline].

logsnr_min (float, defaults to -6.2) : Minimum log-SNR value for the noise schedule — maps to the high-noise start of the schedule.

logsnr_max (float, defaults to 2.0) : Maximum log-SNR value for the noise schedule — maps to the low-noise end of the schedule.

silence_padding_duration (float, defaults to 0.0) : Extra latent headroom after the target content.

num_waveforms_per_prompt (int, defaults to 1) : Waveforms per prompt.

generator : RNG for reproducibility.

latents : Pre-generated starting noise (None → sample fresh).

prompt_embeds : Pre-computed text embeddings.

encoder_attention_mask : Mask for pre-computed embeddings.

return_dict (bool, defaults to True) : Return AudioPipelineOutput or tuple.

callback_on_step_end (Callable, optional) : Called at the end of each denoising step with (self, step_idx, timestep, callback_kwargs). Must return a dict with the (optionally modified) tensors to use for the rest of the loop.

callback_on_step_end_tensor_inputs (list[str], defaults to ["latents"]) : The tensors passed to callback_on_step_end. Must be a subset of self._callback_tensor_inputs.

output_type (str, defaults to "pt") : "pt" / "np" / "latent".

Returns:

[~pipelines.AudioPipelineOutput] with .audios.

Generate an audio variation conditioned on a text prompt and a reference waveform.

Examples:

>>> import torch
>>> import soundfile as sf
>>> import torchaudio
>>> from diffusers import StableAudio3AudioToAudioPipeline

>>> pipe = StableAudio3AudioToAudioPipeline.from_pretrained(
...     "stabilityai/stable-audio-3-medium", torch_dtype=torch.float16
... )
>>> pipe = pipe.to("cuda")

>>> audio, sr = torchaudio.load("reference.wav")
>>> audio = torchaudio.functional.resample(audio, sr, pipe.vae.config.sampling_rate).unsqueeze(0).to("cuda")

>>> generator = torch.Generator("cuda").manual_seed(0)
>>> audio = pipe(
...     "A gentle piano melody with soft strings in a concert hall",
...     duration=10.0,
...     audio=audio,
...     init_noise_level=0.6,
...     generator=generator,
... ).audios

>>> sf.write("output.wav", audio[0].T.cpu().float().numpy(), samplerate=pipe.vae.config.sampling_rate)

encode_duration[[diffusers.StableAudio3AudioToAudioPipeline.encode_duration]]

encode_duration(duration: float, device: device, num_waveforms_per_prompt: int, batch_size: int)

Source

Parameters:

duration : Duration in seconds, applied to every sample in the batch.

device : Target device.

num_waveforms_per_prompt : Tile factor.

batch_size : Number of prompts.

Returns:

(batch * num_waveforms_per_prompt, output_dim) tensor.

Embed the duration value into the global conditioning vector.

encode_prompt[[diffusers.StableAudio3AudioToAudioPipeline.encode_prompt]]

encode_prompt(prompt: typing.Union[str, typing.List[str], NoneType], device: device, num_waveforms_per_prompt: int, prompt_embeds: typing.Optional[torch.Tensor] = None, encoder_attention_mask: typing.Optional[torch.LongTensor] = None)

Source

Parameters:

prompt : Text prompt or list of prompts. Ignored when prompt_embeds is provided.

device : Target device.

num_waveforms_per_prompt : How many output waveforms to generate per prompt; conditioning tensors are tiled accordingly.

prompt_embeds : Pre-computed text embeddings (batch, seq_len, hidden_size).

encoder_attention_mask : Boolean mask (batch, seq_len) for pre-computed embeddings; 1 = real token, 0 = pad.

Returns:

(prompt_embeds, encoder_attention_mask) both tiled to batch * num_waveforms_per_prompt.

Encode text prompt(s) into cross-attention conditioning tensors.

prepare_cross_attention[[diffusers.StableAudio3AudioToAudioPipeline.prepare_cross_attention]]

prepare_cross_attention(prompt_embeds: Tensor, encoder_attention_mask: Tensor, global_hidden_states: Tensor)

Source

Returns:

(context, context_mask) of shapes (batch, T_text + 1, dim) and (batch, T_text + 1).

Build the cross-attention context by appending the duration embedding as an extra token.

SA3 routes the seconds_total conditioner to both the global (AdaLN) input and the cross-attention context (cross_attention_cond_ids = ["prompt", "seconds_total"]). The duration embedding is concatenated after the text tokens, and the attention mask is extended with one valid entry.

StableAudio3DurationEmbedder[[diffusers.StableAudio3DurationEmbedder]]

diffusers.StableAudio3DurationEmbedder[[diffusers.StableAudio3DurationEmbedder]]

diffusers.StableAudio3DurationEmbedder(output_dim: int = 768, fourier_dim: int = 256, min_val: float = 0.0, max_val: float = 384.0, min_freq: float = 0.5, max_freq: float = 10000.0)

Source

Parameters:

output_dim (int, defaults to 768) : Dimension of the output embedding. Must match the DiT's global_cond_dim.

fourier_dim (int, defaults to 256) : Internal Fourier feature dimension (must be even).

min_val (float, defaults to 0.0) : Minimum duration value for normalization clamping.

max_val (float, defaults to 384.0) : Maximum duration value for normalization clamping. Values above this are clamped. 384 seconds is the production SA3 Medium upper bound for the seconds_total conditioner.

min_freq (float, defaults to 0.5) : Minimum frequency for the exponential Fourier basis.

max_freq (float, defaults to 10000.0) : Maximum frequency for the exponential Fourier basis.

Embeds a duration value (in seconds) into a global conditioning vector for the Stable Audio 3 DiT (used as the global_hidden_states AdaLN input).

Replicates NumberConditioner(fourier_features_type="expo") from the SA3 reference implementation.

forward[[diffusers.StableAudio3DurationEmbedder.forward]]

forward(seconds: Tensor)

Source

Parameters:

seconds (torch.Tensor of shape (batch,)) : Duration values in seconds.

Returns:

torch.Tensor of shape (batch, output_dim) — duration embeddings.