Hugging Face's logo

Diffusers documentation

Stable Audio 3

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Stable Audio 3

Stable Audio 3 (SA3) is a text-to-audio model from Stability AI that generates high-quality stereo audio at 44.1 kHz. It uses a rectified-flow DiT conditioned on two signals:

  • Text — encoded by a frozen T5Gemma encoder and injected via cross-attention.
  • Duration — a float (seconds) embedded by StableAudio3DurationEmbedder and used as a global conditioning vector for adaptive layer normalisation.

Audio is decoded by the SAME (Semantically-Aligned Music Encoder) autoencoder, AutoencoderSAME.

Both checkpoints use FlowMatchEulerDiscreteScheduler with a log-SNR-uniform sigma schedule, differing only in stochastic_sampling and the default step count:

Checkpointdiffusion_objectivestochastic_samplingnum_inference_steps
stable-audio-3-medium-baserectified_flowFalse (deterministic Euler)100 (not distilled)
stable-audio-3-medium (distilled)rf_denoiserTrue (ping-pong re-noise)8 (distilled for 8 steps)

The correct scheduler config is baked into each converted checkpoint, so num_inference_steps defaults to the right value when you leave it unset. Only pass it to override.

Original codebase: Stability-AI/stable-audio-3.

Converting original checkpoints

The Stability AI checkpoints are not published in diffusers format, so convert them locally. The script downloads the checkpoint’s model_config.json and selects the scheduler from its diffusion_objective:

python scripts/convert_stable_audio_3_to_diffusers.py \
  --checkpoint_path stabilityai/stable-audio-3-medium-base \
  --text_encoder_repo google/t5gemma-b-b-ul2 \
  --output_dir /tmp/sa3-diffusers-euler \
  --dtype float32

stable-audio-3-medium-base is a gated repo. Run hf auth login with an account that has access before converting, otherwise the download fails with a 401.

Usage example

Load the converted checkpoint from its local output directory (install soundfile with pip install soundfile):

import torch
import soundfile as sf
from diffusers import StableAudio3Pipeline

pipe = StableAudio3Pipeline.from_pretrained("/tmp/sa3-diffusers-euler", torch_dtype=torch.float32)
pipe = pipe.to("cuda")

generator = torch.Generator("cuda").manual_seed(0)
audio = pipe(
    "A gentle piano melody with soft strings in a concert hall",
    duration=10.0,  # seconds; latent length is computed automatically
    generator=generator,
).audios

sf.write("sa3_output.wav", audio[0].T.cpu().float().numpy(), samplerate=44100)

The pipeline is also registered with AutoPipelineForText2Audio, which resolves the checkpoint to StableAudio3Pipeline automatically:

from diffusers import AutoPipelineForText2Audio

pipe = AutoPipelineForText2Audio.from_pretrained("/tmp/sa3-diffusers-euler", torch_dtype=torch.float32)

The examples use a local path because stabilityai/stable-audio-3-medium and stable-audio-3-medium-base are not yet published in diffusers format (loading by repo id returns a 404). Once published, the repo id works in place of the local path.

Tips

  • Use torch.float32 on CPU or MPS (Apple Silicon) — torch.float16 on MPS produces noise.
  • The distilled model (stable-audio-3-medium) is adversarially distilled — guidance is baked into the weights. Leave guidance_scale=1.0 (the default) and don’t pass a negative_prompt for that checkpoint; both only do something useful for the non-distilled stable-audio-3-medium-base checkpoint.
  • silence_padding_duration (default 0.0) adds silent headroom at the end of the latent sequence. Leave it at 0.0 unless the model is trained to mask that padding — otherwise the extra frames drain output energy and the result gets quiet.
  • Set num_waveforms_per_prompt > 1 to generate multiple clips per prompt.

StableAudio3Pipeline

class diffusers.StableAudio3Pipeline

< >

( vae: AutoencoderSAMEtext_encoder: T5GemmaEncoderModeltokenizer: GemmaTokenizerduration_embedder: StableAudio3DurationEmbeddertransformer: StableAudio3DiTModelscheduler: FlowMatchEulerDiscreteScheduler )

Parameters

  • vae ([AutoencoderSAME]) — SAME autoencoder used to encode and decode audio latents.
  • text_encoder ([~transformers.T5GemmaEncoderModel]) — Frozen T5Gemma text encoder (google/t5gemma-b-b-ul2).
  • tokenizer ([~transformers.GemmaTokenizerFast]) — Tokenizer for the text encoder.
  • duration_embedder ([StableAudio3DurationEmbedder]) — Maps duration in seconds to a global conditioning vector for AdaLN in each DiT block.
  • transformer ([StableAudio3DiTModel]) — The rectified-flow velocity-prediction DiT.
  • scheduler ([FlowMatchEulerDiscreteScheduler]) — Scheduler for the iterative denoising loop. The production (distilled) SA3 Medium checkpoint uses stochastic_sampling=True for exactly 8 ping-pong steps; the non-distilled base checkpoint uses stochastic_sampling=False for ~100 deterministic Euler steps.

Pipeline for text-to-audio generation using Stable Audio 3.

SA3 uses a distilled rectified-flow DiT with ping-pong sampling. Classifier-free guidance (guidance_scale / negative_prompt) is unnecessary for the distilled checkpoint (leave guidance_scale=1.0, the default) but is meaningful for the non-distilled stable-audio-3-medium-base checkpoint.

This model inherits from [DiffusionPipeline]. Check the superclass documentation for the generic methods implemented for all pipelines.

__call__

< >

( prompt: typing.Union[str, typing.List[str], NoneType] = Noneduration: float = 10.0num_inference_steps: typing.Optional[int] = Nonelogsnr_min: float = -6.2logsnr_max: float = 2.0silence_padding_duration: float = 0.0guidance_scale: float = 1.0negative_prompt: typing.Union[str, typing.List[str], NoneType] = Nonenum_waveforms_per_prompt: int = 1generator: typing.Union[torch.Generator, typing.List[torch.Generator], NoneType] = Nonelatents: typing.Optional[torch.Tensor] = Noneprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = Nonenegative_prompt_embeds: typing.Optional[torch.Tensor] = Nonenegative_encoder_attention_mask: typing.Optional[torch.LongTensor] = Nonereturn_dict: bool = Truecallback_on_step_end: typing.Optional[typing.Callable[[int, int, dict], dict]] = Nonecallback_on_step_end_tensor_inputs: typing.List[str] = ['latents']output_type: str = 'pt' ) → [~pipelines.AudioPipelineOutput] or tuple

Parameters

  • prompt (str or list[str], optional) — Text prompt(s). Pass prompt_embeds instead to skip tokenization and encoding.
  • duration (float, defaults to 10.0) — Requested output duration in seconds.
  • num_inference_steps (int, optional) — Number of denoising steps. When None (default), the step count is chosen from the scheduler’s stochastic_sampling config: 8 when True (the distilled ping-pong-style checkpoint) and 100 when False (the non-distilled base checkpoint). Pass an explicit value to override.
  • logsnr_min (float, defaults to -6.2) — Minimum log-SNR value for the noise schedule — maps to the high-noise start of the schedule.
  • logsnr_max (float, defaults to 2.0) — Maximum log-SNR value for the noise schedule — maps to the low-noise end of the schedule.
  • silence_padding_duration (float, defaults to 0.0) — Extra seconds of latent context generated beyond the target content, giving the model headroom at the boundary; the output is trimmed back to duration. Defaults to 0.0 (disabled). Increase only if the model is trained/distilled to mask this padding — otherwise the extra frames drain output energy.
  • guidance_scale (float, defaults to 1.0) — Classifier-free guidance scale. 1.0 disables guidance (the default, and the only sensible value for the distilled SA3 Medium checkpoint, whose CFG is baked into the weights). Values > 1.0 are meaningful for the non-distilled stable-audio-3-medium-base checkpoint; higher values follow the prompt more closely at the cost of diversity.
  • negative_prompt (str or list[str], optional) — Prompt(s) describing what to steer away from when guidance_scale > 1.0. Defaults to an empty string (unconditional) when guidance_scale > 1.0 and neither this nor negative_prompt_embeds is given. Ignored when guidance_scale <= 1.0.
  • num_waveforms_per_prompt (int, defaults to 1) — Number of waveforms to generate per prompt.
  • generator (torch.Generator or list[torch.Generator], optional) — For deterministic generation and reproducible re-noise in the ping-pong loop.
  • latents (torch.Tensor, optional) — Pre-generated starting latents. If None a fresh Gaussian tensor is sampled.
  • prompt_embeds (torch.Tensor, optional) — Pre-computed text embeddings (batch, seq_len, 768).
  • encoder_attention_mask (torch.LongTensor, optional) — Boolean mask for pre-computed embeddings.
  • negative_prompt_embeds (torch.Tensor, optional) — Pre-computed negative text embeddings, as an alternative to negative_prompt.
  • negative_encoder_attention_mask (torch.LongTensor, optional) — Boolean mask for pre-computed negative embeddings.
  • return_dict (bool, defaults to True) — Return an AudioPipelineOutput or a plain tuple.
  • callback_on_step_end (Callable, optional) — Called at the end of each denoising step with (self, step_idx, timestep, callback_kwargs), where callback_kwargs contains the tensors listed in callback_on_step_end_tensor_inputs. Must return a dict with the (optionally modified) tensors to use for the rest of the loop.
  • callback_on_step_end_tensor_inputs (list[str], defaults to [“latents”]) — The tensors passed to callback_on_step_end. Must be a subset of self._callback_tensor_inputs.
  • output_type (str, defaults to "pt") — "pt" for a PyTorch tensor, "np" for a NumPy array, or "latent" to skip decoding and return the raw latents.

Returns

[~pipelines.AudioPipelineOutput] or tuple

.audios is a tensor / array of shape (batch * num_waveforms_per_prompt, audio_channels, samples).

Generate audio from a text prompt.

Examples:

>>> import torch
>>> import soundfile as sf
>>> from diffusers import StableAudio3Pipeline

>>> pipe = StableAudio3Pipeline.from_pretrained("stabilityai/stable-audio-3-medium", torch_dtype=torch.float16)
>>> pipe = pipe.to("cuda")

>>> generator = torch.Generator("cuda").manual_seed(0)
>>> audio = pipe(
...     "A gentle piano melody with soft strings in a concert hall",
...     duration=10.0,
...     generator=generator,
... ).audios

>>> sf.write("output.wav", audio[0].T.cpu().float().numpy(), samplerate=pipe.vae.config.sampling_rate)

encode_duration

< >

( duration: floatdevice: devicenum_waveforms_per_prompt: intbatch_size: int )

Parameters

  • duration — Duration in seconds, applied to every sample in the batch.
  • device — Target device.
  • num_waveforms_per_prompt — Tile factor.
  • batch_size — Number of prompts.

Embed the duration value into the global conditioning vector.

encode_prompt

< >

( prompt: typing.Union[str, typing.List[str], NoneType]device: devicenum_waveforms_per_prompt: intprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = None )

Parameters

  • prompt — Text prompt or list of prompts. Ignored when prompt_embeds is provided.
  • device — Target device.
  • num_waveforms_per_prompt — How many output waveforms to generate per prompt; conditioning tensors are tiled accordingly.
  • prompt_embeds — Pre-computed text embeddings (batch, seq_len, hidden_size).
  • encoder_attention_mask — Boolean mask (batch, seq_len) for pre-computed embeddings; 1 = real token, 0 = pad.

Encode text prompt(s) into cross-attention conditioning tensors.

prepare_cross_attention

< >

( prompt_embeds: Tensorencoder_attention_mask: Tensorglobal_hidden_states: Tensor )

Build the cross-attention context by appending the duration embedding as an extra token.

SA3 routes the seconds_total conditioner to both the global (AdaLN) input and the cross-attention context (cross_attention_cond_ids = ["prompt", "seconds_total"]). The duration embedding is concatenated after the text tokens, and the attention mask is extended with one valid entry.

StableAudio3InpaintPipeline

class diffusers.StableAudio3InpaintPipeline

< >

( vae: AutoencoderSAMEtext_encoder: T5GemmaEncoderModeltokenizer: GemmaTokenizerduration_embedder: StableAudio3DurationEmbeddertransformer: StableAudio3DiTModelscheduler: FlowMatchEulerDiscreteScheduler )

Parameters

  • vae ([AutoencoderSAME]) — SAME autoencoder used to encode and decode audio latents.
  • text_encoder ([~transformers.T5GemmaEncoderModel]) — Frozen T5Gemma text encoder (google/t5gemma-b-b-ul2).
  • tokenizer ([~transformers.GemmaTokenizerFast]) — Tokenizer for the text encoder.
  • duration_embedder ([StableAudio3DurationEmbedder]) — Maps duration in seconds to a global conditioning vector for AdaLN in each DiT block.
  • transformer ([StableAudio3DiTModel]) — The rectified-flow velocity-prediction DiT.
  • scheduler ([FlowMatchEulerDiscreteScheduler]) — Scheduler for the iterative denoising loop. The production (distilled) SA3 Medium checkpoint uses stochastic_sampling=True for exactly 8 ping-pong steps; the non-distilled base checkpoint uses stochastic_sampling=False for ~100 deterministic Euler steps.

Audio inpainting pipeline for Stable Audio 3.

Shares its text-to-audio logic with [StableAudio3Pipeline] (kept in sync via # Copied from). When audio and mask are provided, encodes the reference audio with the frozen SAME encoder and injects masked_latent ∥ mask as local-additive conditioning into each DiT block via the transformer’s local_add_cond pathway (to_local_embed).

Call signature extension (see __call__): audio (torch.Tensor of shape (batch, channels, samples)): Reference audio waveform at vae.config.sampling_rate Hz. mask (torch.Tensor of shape (batch, 1, latent_length)): Per-frame binary mask in latent space. 1 = preserve original audio; 0 = region to be inpainted.

__call__

< >

( prompt: typing.Union[str, typing.List[str], NoneType] = Noneduration: float = 10.0audio: typing.Optional[torch.Tensor] = Nonemask: typing.Optional[torch.Tensor] = Nonemask_start_seconds: typing.Union[float, typing.List[float], NoneType] = Nonemask_end_seconds: typing.Union[float, typing.List[float], NoneType] = Nonenum_inference_steps: typing.Optional[int] = Nonelogsnr_min: float = -6.2logsnr_max: float = 2.0silence_padding_duration: float = 0.0num_waveforms_per_prompt: int = 1generator: typing.Union[torch.Generator, typing.List[torch.Generator], NoneType] = Nonelatents: typing.Optional[torch.Tensor] = Noneprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = Nonereturn_dict: bool = Truecallback_on_step_end: typing.Optional[typing.Callable[[int, int, dict], dict]] = Nonecallback_on_step_end_tensor_inputs: typing.List[str] = ['latents']output_type: str = 'pt' )

Parameters

  • prompt (str or list[str], optional) — Text prompt(s).
  • duration (float, defaults to 10.0) — Output duration in seconds. Should match the reference audio.
  • audio (torch.Tensor, optional) — Reference waveform (batch, channels, samples) at vae.config.sampling_rate Hz. Required for inpainting.
  • mask (torch.Tensor, optional) — Per-frame latent-space mask (batch, 1, L) with 0 = inpaint region, 1 = preserve. Either mask or mask_start_seconds / mask_end_seconds must be provided.
  • mask_start_seconds (float or list[float], optional) — Start time(s) of the inpaint region in seconds.
  • mask_end_seconds (float or list[float], optional) — End time(s) of the inpaint region (must pair with mask_start_seconds).
  • num_inference_steps (int, optional) — Number of denoising steps. When None (default), the step count is chosen from the scheduler’s stochastic_sampling config, matching [StableAudio3Pipeline].
  • logsnr_min (float, defaults to -6.2) — Minimum log-SNR value for the noise schedule — maps to the high-noise start of the schedule.
  • logsnr_max (float, defaults to 2.0) — Maximum log-SNR value for the noise schedule — maps to the low-noise end of the schedule.
  • silence_padding_duration (float, defaults to 0.0) — Extra latent headroom after the target content.
  • num_waveforms_per_prompt (int, defaults to 1) — Waveforms per prompt.
  • generator — RNG for reproducibility.
  • latents — Pre-generated starting noise (None → sample fresh).
  • prompt_embeds — Pre-computed text embeddings.
  • encoder_attention_mask — Mask for pre-computed embeddings.
  • return_dict (bool, defaults to True) — Return AudioPipelineOutput or tuple.
  • callback_on_step_end (Callable, optional) — Called at the end of each denoising step with (self, step_idx, timestep, callback_kwargs). Must return a dict with the (optionally modified) tensors to use for the rest of the loop.
  • callback_on_step_end_tensor_inputs (list[str], defaults to [“latents”]) — The tensors passed to callback_on_step_end. Must be a subset of self._callback_tensor_inputs.
  • output_type (str, defaults to "pt") — "pt" / "np" / "latent".

Generate inpainted audio conditioned on a text prompt and reference.

Examples:

>>> import torch
>>> import soundfile as sf
>>> import torchaudio
>>> from diffusers import StableAudio3InpaintPipeline

>>> pipe = StableAudio3InpaintPipeline.from_pretrained(
...     "stabilityai/stable-audio-3-medium", torch_dtype=torch.float16
... )
>>> pipe = pipe.to("cuda")

>>> audio, sr = torchaudio.load("reference.wav")
>>> audio = torchaudio.functional.resample(audio, sr, pipe.vae.config.sampling_rate).unsqueeze(0).to("cuda")

>>> generator = torch.Generator("cuda").manual_seed(0)
>>> audio = pipe(
...     "A gentle piano melody with soft strings in a concert hall",
...     duration=10.0,
...     audio=audio,
...     mask_start_seconds=4.0,
...     mask_end_seconds=6.0,
...     generator=generator,
... ).audios

>>> sf.write("output.wav", audio[0].T.cpu().float().numpy(), samplerate=pipe.vae.config.sampling_rate)

encode_duration

< >

( duration: floatdevice: devicenum_waveforms_per_prompt: intbatch_size: int )

Parameters

  • duration — Duration in seconds, applied to every sample in the batch.
  • device — Target device.
  • num_waveforms_per_prompt — Tile factor.
  • batch_size — Number of prompts.

Embed the duration value into the global conditioning vector.

encode_prompt

< >

( prompt: typing.Union[str, typing.List[str], NoneType]device: devicenum_waveforms_per_prompt: intprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = None )

Parameters

  • prompt — Text prompt or list of prompts. Ignored when prompt_embeds is provided.
  • device — Target device.
  • num_waveforms_per_prompt — How many output waveforms to generate per prompt; conditioning tensors are tiled accordingly.
  • prompt_embeds — Pre-computed text embeddings (batch, seq_len, hidden_size).
  • encoder_attention_mask — Boolean mask (batch, seq_len) for pre-computed embeddings; 1 = real token, 0 = pad.

Encode text prompt(s) into cross-attention conditioning tensors.

prepare_cross_attention

< >

( prompt_embeds: Tensorencoder_attention_mask: Tensorglobal_hidden_states: Tensor )

Build the cross-attention context by appending the duration embedding as an extra token.

SA3 routes the seconds_total conditioner to both the global (AdaLN) input and the cross-attention context (cross_attention_cond_ids = ["prompt", "seconds_total"]). The duration embedding is concatenated after the text tokens, and the attention mask is extended with one valid entry.

StableAudio3AudioToAudioPipeline

Generates a variation of a reference audio clip: the whole reference is noised to init_noise_level and denoised from there, unlike StableAudio3InpaintPipeline’s per-frame local-additive conditioning which preserves specific frames exactly.

class diffusers.StableAudio3AudioToAudioPipeline

< >

( vae: AutoencoderSAMEtext_encoder: T5GemmaEncoderModeltokenizer: GemmaTokenizerduration_embedder: StableAudio3DurationEmbeddertransformer: StableAudio3DiTModelscheduler: FlowMatchEulerDiscreteScheduler )

Parameters

  • vae ([AutoencoderSAME]) — SAME autoencoder used to encode and decode audio latents.
  • text_encoder ([~transformers.T5GemmaEncoderModel]) — Frozen T5Gemma text encoder (google/t5gemma-b-b-ul2).
  • tokenizer ([~transformers.GemmaTokenizerFast]) — Tokenizer for the text encoder.
  • duration_embedder ([StableAudio3DurationEmbedder]) — Maps duration in seconds to a global conditioning vector for AdaLN in each DiT block.
  • transformer ([StableAudio3DiTModel]) — The rectified-flow velocity-prediction DiT.
  • scheduler ([FlowMatchEulerDiscreteScheduler]) — Scheduler for the iterative denoising loop. The production (distilled) SA3 Medium checkpoint uses stochastic_sampling=True for exactly 8 ping-pong steps; the non-distilled base checkpoint uses stochastic_sampling=False for ~100 deterministic Euler steps.

Audio-to-audio variation pipeline for Stable Audio 3.

Shares its text-to-audio logic with [StableAudio3Pipeline] (kept in sync via # Copied from). Encodes the reference audio with the frozen SAME encoder, mixes it with fresh noise according to init_noise_level, and denoises from there — the whole signal is noised/denoised globally, unlike [StableAudio3InpaintPipeline]’s per-frame local-additive conditioning.

Call signature extension (see __call__): audio (torch.Tensor of shape (batch, channels, samples)): Reference audio waveform at vae.config.sampling_rate Hz. init_noise_level (float): How much noise to mix into the reference before denoising. 1.0 = full noise (equivalent to text-to-audio); lower values retain more of the reference.

__call__

< >

( prompt: typing.Union[str, typing.List[str], NoneType] = Noneduration: float = 10.0audio: typing.Optional[torch.Tensor] = Noneinit_noise_level: float = 1.0num_inference_steps: typing.Optional[int] = Nonelogsnr_min: float = -6.2logsnr_max: float = 2.0silence_padding_duration: float = 0.0num_waveforms_per_prompt: int = 1generator: typing.Union[torch.Generator, typing.List[torch.Generator], NoneType] = Nonelatents: typing.Optional[torch.Tensor] = Noneprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = Nonereturn_dict: bool = Truecallback_on_step_end: typing.Optional[typing.Callable[[int, int, dict], dict]] = Nonecallback_on_step_end_tensor_inputs: typing.List[str] = ['latents']output_type: str = 'pt' )

Parameters

  • prompt (str or list[str], optional) — Text prompt(s).
  • duration (float, defaults to 10.0) — Output duration in seconds. Should match the reference audio.
  • audio (torch.Tensor, optional) — Reference waveform (batch, channels, samples) at vae.config.sampling_rate Hz. Required.
  • init_noise_level (float, defaults to 1.0) — Noise level (in (0, 1]) mixed into the reference before denoising: x_start = (1 - init_noise_level) * reference_latents + init_noise_level * noise. 1.0 discards the reference entirely (equivalent to [StableAudio3Pipeline]); lower values retain progressively more of the reference’s structure while still running the full step count.
  • num_inference_steps (int, optional) — Number of denoising steps. When None (default), the step count is chosen from the scheduler’s stochastic_sampling config, matching [StableAudio3Pipeline].
  • logsnr_min (float, defaults to -6.2) — Minimum log-SNR value for the noise schedule — maps to the high-noise start of the schedule.
  • logsnr_max (float, defaults to 2.0) — Maximum log-SNR value for the noise schedule — maps to the low-noise end of the schedule.
  • silence_padding_duration (float, defaults to 0.0) — Extra latent headroom after the target content.
  • num_waveforms_per_prompt (int, defaults to 1) — Waveforms per prompt.
  • generator — RNG for reproducibility.
  • latents — Pre-generated starting noise (None → sample fresh).
  • prompt_embeds — Pre-computed text embeddings.
  • encoder_attention_mask — Mask for pre-computed embeddings.
  • return_dict (bool, defaults to True) — Return AudioPipelineOutput or tuple.
  • callback_on_step_end (Callable, optional) — Called at the end of each denoising step with (self, step_idx, timestep, callback_kwargs). Must return a dict with the (optionally modified) tensors to use for the rest of the loop.
  • callback_on_step_end_tensor_inputs (list[str], defaults to [“latents”]) — The tensors passed to callback_on_step_end. Must be a subset of self._callback_tensor_inputs.
  • output_type (str, defaults to "pt") — "pt" / "np" / "latent".

Generate an audio variation conditioned on a text prompt and a reference waveform.

Examples:

>>> import torch
>>> import soundfile as sf
>>> import torchaudio
>>> from diffusers import StableAudio3AudioToAudioPipeline

>>> pipe = StableAudio3AudioToAudioPipeline.from_pretrained(
...     "stabilityai/stable-audio-3-medium", torch_dtype=torch.float16
... )
>>> pipe = pipe.to("cuda")

>>> audio, sr = torchaudio.load("reference.wav")
>>> audio = torchaudio.functional.resample(audio, sr, pipe.vae.config.sampling_rate).unsqueeze(0).to("cuda")

>>> generator = torch.Generator("cuda").manual_seed(0)
>>> audio = pipe(
...     "A gentle piano melody with soft strings in a concert hall",
...     duration=10.0,
...     audio=audio,
...     init_noise_level=0.6,
...     generator=generator,
... ).audios

>>> sf.write("output.wav", audio[0].T.cpu().float().numpy(), samplerate=pipe.vae.config.sampling_rate)

encode_duration

< >

( duration: floatdevice: devicenum_waveforms_per_prompt: intbatch_size: int )

Parameters

  • duration — Duration in seconds, applied to every sample in the batch.
  • device — Target device.
  • num_waveforms_per_prompt — Tile factor.
  • batch_size — Number of prompts.

Embed the duration value into the global conditioning vector.

encode_prompt

< >

( prompt: typing.Union[str, typing.List[str], NoneType]device: devicenum_waveforms_per_prompt: intprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = None )

Parameters

  • prompt — Text prompt or list of prompts. Ignored when prompt_embeds is provided.
  • device — Target device.
  • num_waveforms_per_prompt — How many output waveforms to generate per prompt; conditioning tensors are tiled accordingly.
  • prompt_embeds — Pre-computed text embeddings (batch, seq_len, hidden_size).
  • encoder_attention_mask — Boolean mask (batch, seq_len) for pre-computed embeddings; 1 = real token, 0 = pad.

Encode text prompt(s) into cross-attention conditioning tensors.

prepare_cross_attention

< >

( prompt_embeds: Tensorencoder_attention_mask: Tensorglobal_hidden_states: Tensor )

Build the cross-attention context by appending the duration embedding as an extra token.

SA3 routes the seconds_total conditioner to both the global (AdaLN) input and the cross-attention context (cross_attention_cond_ids = ["prompt", "seconds_total"]). The duration embedding is concatenated after the text tokens, and the attention mask is extended with one valid entry.

StableAudio3DurationEmbedder

class diffusers.StableAudio3DurationEmbedder

< >

( output_dim: int = 768fourier_dim: int = 256min_val: float = 0.0max_val: float = 384.0min_freq: float = 0.5max_freq: float = 10000.0 )

Parameters

  • output_dim (int, defaults to 768) — Dimension of the output embedding. Must match the DiT’s global_cond_dim.
  • fourier_dim (int, defaults to 256) — Internal Fourier feature dimension (must be even).
  • min_val (float, defaults to 0.0) — Minimum duration value for normalization clamping.
  • max_val (float, defaults to 384.0) — Maximum duration value for normalization clamping. Values above this are clamped. 384 seconds is the production SA3 Medium upper bound for the seconds_total conditioner.
  • min_freq (float, defaults to 0.5) — Minimum frequency for the exponential Fourier basis.
  • max_freq (float, defaults to 10000.0) — Maximum frequency for the exponential Fourier basis.

Embeds a duration value (in seconds) into a global conditioning vector for the Stable Audio 3 DiT (used as the global_hidden_states AdaLN input).

Replicates NumberConditioner(fourier_features_type="expo") from the SA3 reference implementation.

forward

< >

( seconds: Tensor )

Parameters

  • seconds (torch.Tensor of shape (batch,)) — Duration values in seconds.
Update on GitHub

←Stable Audio Anima→