Hugging Face's logo

Diffusers documentation

Stable Cascade

You are viewing v0.29.2 version. A newer version v0.31.0 is available.
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Stable Cascade

This model is built upon the Würstchen architecture and its main difference to other models like Stable Diffusion is that it is working at a much smaller latent space. Why is this important? The smaller the latent space, the faster you can run inference and the cheaper the training becomes. How small is the latent space? Stable Diffusion uses a compression factor of 8, resulting in a 1024x1024 image being encoded to 128x128. Stable Cascade achieves a compression factor of 42, meaning that it is possible to encode a 1024x1024 image to 24x24, while maintaining crisp reconstructions. The text-conditional model is then trained in the highly compressed latent space. Previous versions of this architecture, achieved a 16x cost reduction over Stable Diffusion 1.5.

Therefore, this kind of model is well suited for usages where efficiency is important. Furthermore, all known extensions like finetuning, LoRA, ControlNet, IP-Adapter, LCM etc. are possible with this method as well.

The original codebase can be found at Stability-AI/StableCascade.

Model Overview

Stable Cascade consists of three models: Stage A, Stage B and Stage C, representing a cascade to generate images, hence the name “Stable Cascade”.

Stage A & B are used to compress images, similar to what the job of the VAE is in Stable Diffusion. However, with this setup, a much higher compression of images can be achieved. While the Stable Diffusion models use a spatial compression factor of 8, encoding an image with resolution of 1024 x 1024 to 128 x 128, Stable Cascade achieves a compression factor of 42. This encodes a 1024 x 1024 image to 24 x 24, while being able to accurately decode the image. This comes with the great benefit of cheaper training and inference. Furthermore, Stage C is responsible for generating the small 24 x 24 latents given a text prompt.

The Stage C model operates on the small 24 x 24 latents and denoises the latents conditioned on text prompts. The model is also the largest component in the Cascade pipeline and is meant to be used with the StableCascadePriorPipeline

The Stage B and Stage A models are used with the StableCascadeDecoderPipeline and are responsible for generating the final image given the small 24 x 24 latents.

There are some restrictions on data types that can be used with the Stable Cascade models. The official checkpoints for the StableCascadePriorPipeline do not support the torch.float16 data type. Please use torch.bfloat16 instead.

In order to use the torch.bfloat16 data type with the StableCascadeDecoderPipeline you need to have PyTorch 2.2.0 or higher installed. This also means that using the StableCascadeCombinedPipeline with torch.bfloat16 requires PyTorch 2.2.0 or higher, since it calls the StableCascadeDecoderPipeline internally.

If it is not possible to install PyTorch 2.2.0 or higher in your environment, the StableCascadeDecoderPipeline can be used on its own with the torch.float16 data type. You can download the full precision or bf16 variant weights for the pipeline and cast the weights to torch.float16.

Usage example

import torch
from diffusers import StableCascadeDecoderPipeline, StableCascadePriorPipeline

prompt = "an image of a shiba inu, donning a spacesuit and helmet"
negative_prompt = ""

prior = StableCascadePriorPipeline.from_pretrained("stabilityai/stable-cascade-prior", variant="bf16", torch_dtype=torch.bfloat16)
decoder = StableCascadeDecoderPipeline.from_pretrained("stabilityai/stable-cascade", variant="bf16", torch_dtype=torch.float16)

prior.enable_model_cpu_offload()
prior_output = prior(
    prompt=prompt,
    height=1024,
    width=1024,
    negative_prompt=negative_prompt,
    guidance_scale=4.0,
    num_images_per_prompt=1,
    num_inference_steps=20
)

decoder.enable_model_cpu_offload()
decoder_output = decoder(
    image_embeddings=prior_output.image_embeddings.to(torch.float16),
    prompt=prompt,
    negative_prompt=negative_prompt,
    guidance_scale=0.0,
    output_type="pil",
    num_inference_steps=10
).images[0]
decoder_output.save("cascade.png")

Using the Lite Versions of the Stage B and Stage C models

import torch
from diffusers import (
    StableCascadeDecoderPipeline,
    StableCascadePriorPipeline,
    StableCascadeUNet,
)

prompt = "an image of a shiba inu, donning a spacesuit and helmet"
negative_prompt = ""

prior_unet = StableCascadeUNet.from_pretrained("stabilityai/stable-cascade-prior", subfolder="prior_lite")
decoder_unet = StableCascadeUNet.from_pretrained("stabilityai/stable-cascade", subfolder="decoder_lite")

prior = StableCascadePriorPipeline.from_pretrained("stabilityai/stable-cascade-prior", prior=prior_unet)
decoder = StableCascadeDecoderPipeline.from_pretrained("stabilityai/stable-cascade", decoder=decoder_unet)

prior.enable_model_cpu_offload()
prior_output = prior(
    prompt=prompt,
    height=1024,
    width=1024,
    negative_prompt=negative_prompt,
    guidance_scale=4.0,
    num_images_per_prompt=1,
    num_inference_steps=20
)

decoder.enable_model_cpu_offload()
decoder_output = decoder(
    image_embeddings=prior_output.image_embeddings,
    prompt=prompt,
    negative_prompt=negative_prompt,
    guidance_scale=0.0,
    output_type="pil",
    num_inference_steps=10
).images[0]
decoder_output.save("cascade.png")

Loading original checkpoints with from_single_file

Loading the original format checkpoints is supported via from_single_file method in the StableCascadeUNet.

import torch
from diffusers import (
    StableCascadeDecoderPipeline,
    StableCascadePriorPipeline,
    StableCascadeUNet,
)

prompt = "an image of a shiba inu, donning a spacesuit and helmet"
negative_prompt = ""

prior_unet = StableCascadeUNet.from_single_file(
    "https://huggingface.co/stabilityai/stable-cascade/resolve/main/stage_c_bf16.safetensors",
    torch_dtype=torch.bfloat16
)
decoder_unet = StableCascadeUNet.from_single_file(
    "https://huggingface.co/stabilityai/stable-cascade/blob/main/stage_b_bf16.safetensors",
    torch_dtype=torch.bfloat16
)

prior = StableCascadePriorPipeline.from_pretrained("stabilityai/stable-cascade-prior", prior=prior_unet, torch_dtype=torch.bfloat16)
decoder = StableCascadeDecoderPipeline.from_pretrained("stabilityai/stable-cascade", decoder=decoder_unet, torch_dtype=torch.bfloat16)

prior.enable_model_cpu_offload()
prior_output = prior(
    prompt=prompt,
    height=1024,
    width=1024,
    negative_prompt=negative_prompt,
    guidance_scale=4.0,
    num_images_per_prompt=1,
    num_inference_steps=20
)

decoder.enable_model_cpu_offload()
decoder_output = decoder(
    image_embeddings=prior_output.image_embeddings,
    prompt=prompt,
    negative_prompt=negative_prompt,
    guidance_scale=0.0,
    output_type="pil",
    num_inference_steps=10
).images[0]
decoder_output.save("cascade-single-file.png")

Uses

Direct Use

The model is intended for research purposes for now. Possible research areas and tasks include

  • Research on generative models.
  • Safe deployment of models which have the potential to generate harmful content.
  • Probing and understanding the limitations and biases of generative models.
  • Generation of artworks and use in design and other artistic processes.
  • Applications in educational or creative tools.

Excluded uses are described below.

Out-of-Scope Use

The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model. The model should not be used in any way that violates Stability AI’s Acceptable Use Policy.

Limitations and Bias

Limitations

  • Faces and people in general may not be generated properly.
  • The autoencoding part of the model is lossy.

StableCascadeCombinedPipeline

class diffusers.StableCascadeCombinedPipeline

< >

( tokenizer: CLIPTokenizer text_encoder: CLIPTextModel decoder: StableCascadeUNet scheduler: DDPMWuerstchenScheduler vqgan: PaellaVQModel prior_prior: StableCascadeUNet prior_text_encoder: CLIPTextModel prior_tokenizer: CLIPTokenizer prior_scheduler: DDPMWuerstchenScheduler prior_feature_extractor: Optional = None prior_image_encoder: Optional = None )

Parameters

  • tokenizer (CLIPTokenizer) — The decoder tokenizer to be used for text inputs.
  • text_encoder (CLIPTextModel) — The decoder text encoder to be used for text inputs.
  • decoder (StableCascadeUNet) — The decoder model to be used for decoder image generation pipeline.
  • scheduler (DDPMWuerstchenScheduler) — The scheduler to be used for decoder image generation pipeline.
  • vqgan (PaellaVQModel) — The VQGAN model to be used for decoder image generation pipeline.
  • feature_extractor (CLIPImageProcessor) — Model that extracts features from generated images to be used as inputs for the image_encoder.
  • image_encoder (CLIPVisionModelWithProjection) — Frozen CLIP image-encoder (clip-vit-large-patch14).
  • prior_prior (StableCascadeUNet) — The prior model to be used for prior pipeline.
  • prior_scheduler (DDPMWuerstchenScheduler) — The scheduler to be used for prior pipeline.

Combined Pipeline for text-to-image generation using Stable Cascade.

This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

__call__

< >

( prompt: Union = None images: Union = None height: int = 512 width: int = 512 prior_num_inference_steps: int = 60 prior_guidance_scale: float = 4.0 num_inference_steps: int = 12 decoder_guidance_scale: float = 0.0 negative_prompt: Union = None prompt_embeds: Optional = None prompt_embeds_pooled: Optional = None negative_prompt_embeds: Optional = None negative_prompt_embeds_pooled: Optional = None num_images_per_prompt: int = 1 generator: Union = None latents: Optional = None output_type: Optional = 'pil' return_dict: bool = True prior_callback_on_step_end: Optional = None prior_callback_on_step_end_tensor_inputs: List = ['latents'] callback_on_step_end: Optional = None callback_on_step_end_tensor_inputs: List = ['latents'] )

Parameters

  • prompt (str or List[str]) — The prompt or prompts to guide the image generation for the prior and decoder.
  • images (torch.Tensor, PIL.Image.Image, List[torch.Tensor], List[PIL.Image.Image], optional) — The images to guide the image generation for the prior.
  • negative_prompt (str or List[str], optional) — The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if guidance_scale is less than 1).
  • prompt_embeds (torch.Tensor, optional) — Pre-generated text embeddings for the prior. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, text embeddings will be generated from prompt input argument.
  • prompt_embeds_pooled (torch.Tensor, optional) — Pre-generated text embeddings for the prior. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, text embeddings will be generated from prompt input argument.
  • negative_prompt_embeds (torch.Tensor, optional) — Pre-generated negative text embeddings for the prior. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, negative_prompt_embeds will be generated from negative_prompt input argument.
  • negative_prompt_embeds_pooled (torch.Tensor, optional) — Pre-generated negative text embeddings for the prior. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, negative_prompt_embeds will be generated from negative_prompt input argument.
  • num_images_per_prompt (int, optional, defaults to 1) — The number of images to generate per prompt.
  • height (int, optional, defaults to 512) — The height in pixels of the generated image.
  • width (int, optional, defaults to 512) — The width in pixels of the generated image.
  • prior_guidance_scale (float, optional, defaults to 4.0) — Guidance scale as defined in Classifier-Free Diffusion Guidance. prior_guidance_scale is defined as w of equation 2. of Imagen Paper. Guidance scale is enabled by setting prior_guidance_scale > 1. Higher guidance scale encourages to generate images that are closely linked to the text prompt, usually at the expense of lower image quality.
  • prior_num_inference_steps (Union[int, Dict[float, int]], optional, defaults to 60) — The number of prior denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. For more specific timestep spacing, you can pass customized prior_timesteps
  • num_inference_steps (int, optional, defaults to 12) — The number of decoder denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. For more specific timestep spacing, you can pass customized timesteps
  • decoder_guidance_scale (float, optional, defaults to 0.0) — Guidance scale as defined in Classifier-Free Diffusion Guidance. guidance_scale is defined as w of equation 2. of Imagen Paper. Guidance scale is enabled by setting guidance_scale > 1. Higher guidance scale encourages to generate images that are closely linked to the text prompt, usually at the expense of lower image quality.
  • generator (torch.Generator or List[torch.Generator], optional) — One or a list of torch generator(s) to make generation deterministic.
  • latents (torch.Tensor, optional) — Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random generator.
  • output_type (str, optional, defaults to "pil") — The output format of the generate image. Choose between: "pil" (PIL.Image.Image), "np" (np.array) or "pt" (torch.Tensor).
  • return_dict (bool, optional, defaults to True) — Whether or not to return a ImagePipelineOutput instead of a plain tuple.
  • prior_callback_on_step_end (Callable, optional) — A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: prior_callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict).
  • prior_callback_on_step_end_tensor_inputs (List, optional) — The list of tensor inputs for the prior_callback_on_step_end function. The tensors specified in the list will be passed as callback_kwargs argument. You will only be able to include variables listed in the ._callback_tensor_inputs attribute of your pipeline class.
  • callback_on_step_end (Callable, optional) — A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict). callback_kwargs will include a list of all tensors as specified by callback_on_step_end_tensor_inputs.
  • callback_on_step_end_tensor_inputs (List, optional) — The list of tensor inputs for the callback_on_step_end function. The tensors specified in the list will be passed as callback_kwargs argument. You will only be able to include variables listed in the ._callback_tensor_inputs attribute of your pipeline class.

Function invoked when calling the pipeline for generation.

Examples:

>>> import torch
>>> from diffusers import StableCascadeCombinedPipeline

>>> pipe = StableCascadeCombinedPipeline.from_pretrained(
...     "stabilityai/stable-cascade", variant="bf16", torch_dtype=torch.bfloat16
... )
>>> pipe.enable_model_cpu_offload()
>>> prompt = "an image of a shiba inu, donning a spacesuit and helmet"
>>> images = pipe(prompt=prompt)

enable_model_cpu_offload

< >

( gpu_id: Optional = None device: Union = 'cuda' )

Offloads all models to CPU using accelerate, reducing memory usage with a low impact on performance. Compared to enable_sequential_cpu_offload, this method moves one whole model at a time to the GPU when its forward method is called, and the model remains in GPU until the next model runs. Memory savings are lower than with enable_sequential_cpu_offload, but performance is much better due to the iterative execution of the unet.

enable_sequential_cpu_offload

< >

( gpu_id: Optional = None device: Union = 'cuda' )

Offloads all models (unet, text_encoder, vae, and safety checker state dicts) to CPU using 🤗 Accelerate, significantly reducing memory usage. Models are moved to a torch.device('meta') and loaded on a GPU only when their specific submodule’s forward method is called. Offloading happens on a submodule basis. Memory savings are higher than using enable_model_cpu_offload, but performance is lower.

StableCascadePriorPipeline

class diffusers.StableCascadePriorPipeline

< >

( tokenizer: CLIPTokenizer text_encoder: CLIPTextModelWithProjection prior: StableCascadeUNet scheduler: DDPMWuerstchenScheduler resolution_multiple: float = 42.67 feature_extractor: Optional = None image_encoder: Optional = None )

Parameters

  • prior (StableCascadeUNet) — The Stable Cascade prior to approximate the image embedding from the text and/or image embedding.
  • text_encoder (CLIPTextModelWithProjection) — Frozen text-encoder (laion/CLIP-ViT-bigG-14-laion2B-39B-b160k).
  • feature_extractor (CLIPImageProcessor) — Model that extracts features from generated images to be used as inputs for the image_encoder.
  • image_encoder (CLIPVisionModelWithProjection) — Frozen CLIP image-encoder (clip-vit-large-patch14).
  • tokenizer (CLIPTokenizer) — Tokenizer of class CLIPTokenizer.
  • scheduler (DDPMWuerstchenScheduler) — A scheduler to be used in combination with prior to generate image embedding.
  • resolution_multiple (‘float’, optional, defaults to 42.67) — Default resolution for multiple images generated.

Pipeline for generating image prior for Stable Cascade.

This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

__call__

< >

( prompt: Union = None images: Union = None height: int = 1024 width: int = 1024 num_inference_steps: int = 20 timesteps: List = None guidance_scale: float = 4.0 negative_prompt: Union = None prompt_embeds: Optional = None prompt_embeds_pooled: Optional = None negative_prompt_embeds: Optional = None negative_prompt_embeds_pooled: Optional = None image_embeds: Optional = None num_images_per_prompt: Optional = 1 generator: Union = None latents: Optional = None output_type: Optional = 'pt' return_dict: bool = True callback_on_step_end: Optional = None callback_on_step_end_tensor_inputs: List = ['latents'] )

Parameters

  • prompt (str or List[str]) — The prompt or prompts to guide the image generation.
  • height (int, optional, defaults to 1024) — The height in pixels of the generated image.
  • width (int, optional, defaults to 1024) — The width in pixels of the generated image.
  • num_inference_steps (int, optional, defaults to 60) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.
  • guidance_scale (float, optional, defaults to 8.0) — Guidance scale as defined in Classifier-Free Diffusion Guidance. decoder_guidance_scale is defined as w of equation 2. of Imagen Paper. Guidance scale is enabled by setting decoder_guidance_scale > 1. Higher guidance scale encourages to generate images that are closely linked to the text prompt, usually at the expense of lower image quality.
  • negative_prompt (str or List[str], optional) — The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if decoder_guidance_scale is less than 1).
  • prompt_embeds (torch.Tensor, optional) — Pre-generated text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, text embeddings will be generated from prompt input argument.
  • prompt_embeds_pooled (torch.Tensor, optional) — Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, pooled text embeddings will be generated from prompt input argument.
  • negative_prompt_embeds (torch.Tensor, optional) — Pre-generated negative text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, negative_prompt_embeds will be generated from negative_prompt input argument.
  • negative_prompt_embeds_pooled (torch.Tensor, optional) — Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, negative_prompt_embeds_pooled will be generated from negative_prompt input argument.
  • image_embeds (torch.Tensor, optional) — Pre-generated image embeddings. Can be used to easily tweak image inputs, e.g. prompt weighting. If not provided, image embeddings will be generated from image input argument if existing.
  • num_images_per_prompt (int, optional, defaults to 1) — The number of images to generate per prompt.
  • generator (torch.Generator or List[torch.Generator], optional) — One or a list of torch generator(s) to make generation deterministic.
  • latents (torch.Tensor, optional) — Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random generator.
  • output_type (str, optional, defaults to "pil") — The output format of the generate image. Choose between: "pil" (PIL.Image.Image), "np" (np.array) or "pt" (torch.Tensor).
  • return_dict (bool, optional, defaults to True) — Whether or not to return a ImagePipelineOutput instead of a plain tuple.
  • callback_on_step_end (Callable, optional) — A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict). callback_kwargs will include a list of all tensors as specified by callback_on_step_end_tensor_inputs.
  • callback_on_step_end_tensor_inputs (List, optional) — The list of tensor inputs for the callback_on_step_end function. The tensors specified in the list will be passed as callback_kwargs argument. You will only be able to include variables listed in the ._callback_tensor_inputs attribute of your pipeline class.

Function invoked when calling the pipeline for generation.

Examples:

>>> import torch
>>> from diffusers import StableCascadePriorPipeline

>>> prior_pipe = StableCascadePriorPipeline.from_pretrained(
...     "stabilityai/stable-cascade-prior", torch_dtype=torch.bfloat16
... ).to("cuda")

>>> prompt = "an image of a shiba inu, donning a spacesuit and helmet"
>>> prior_output = pipe(prompt)

StableCascadePriorPipelineOutput

class diffusers.pipelines.stable_cascade.pipeline_stable_cascade_prior.StableCascadePriorPipelineOutput

< >

( image_embeddings: Union prompt_embeds: Union prompt_embeds_pooled: Union negative_prompt_embeds: Union negative_prompt_embeds_pooled: Union )

Parameters

  • image_embeddings (torch.Tensor or np.ndarray) — Prior image embeddings for text prompt
  • prompt_embeds (torch.Tensor) — Text embeddings for the prompt.
  • negative_prompt_embeds (torch.Tensor) — Text embeddings for the negative prompt.

Output class for WuerstchenPriorPipeline.

StableCascadeDecoderPipeline

class diffusers.StableCascadeDecoderPipeline

< >

( decoder: StableCascadeUNet tokenizer: CLIPTokenizer text_encoder: CLIPTextModel scheduler: DDPMWuerstchenScheduler vqgan: PaellaVQModel latent_dim_scale: float = 10.67 )

Parameters

  • tokenizer (CLIPTokenizer) — The CLIP tokenizer.
  • text_encoder (CLIPTextModel) — The CLIP text encoder.
  • decoder (StableCascadeUNet) — The Stable Cascade decoder unet.
  • vqgan (PaellaVQModel) — The VQGAN model.
  • scheduler (DDPMWuerstchenScheduler) — A scheduler to be used in combination with prior to generate image embedding.
  • latent_dim_scale (float, optional, defaults to 10.67) — Multiplier to determine the VQ latent space size from the image embeddings. If the image embeddings are height=24 and width=24, the VQ latent shape needs to be height=int(2410.67)=256 and width=int(2410.67)=256 in order to match the training conditions.

Pipeline for generating images from the Stable Cascade model.

This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

__call__

< >

( image_embeddings: Union prompt: Union = None num_inference_steps: int = 10 guidance_scale: float = 0.0 negative_prompt: Union = None prompt_embeds: Optional = None prompt_embeds_pooled: Optional = None negative_prompt_embeds: Optional = None negative_prompt_embeds_pooled: Optional = None num_images_per_prompt: int = 1 generator: Union = None latents: Optional = None output_type: Optional = 'pil' return_dict: bool = True callback_on_step_end: Optional = None callback_on_step_end_tensor_inputs: List = ['latents'] )

Parameters

  • image_embedding (torch.Tensor or List[torch.Tensor]) — Image Embeddings either extracted from an image or generated by a Prior Model.
  • prompt (str or List[str]) — The prompt or prompts to guide the image generation.
  • num_inference_steps (int, optional, defaults to 12) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.
  • guidance_scale (float, optional, defaults to 0.0) — Guidance scale as defined in Classifier-Free Diffusion Guidance. decoder_guidance_scale is defined as w of equation 2. of Imagen Paper. Guidance scale is enabled by setting decoder_guidance_scale > 1. Higher guidance scale encourages to generate images that are closely linked to the text prompt, usually at the expense of lower image quality.
  • negative_prompt (str or List[str], optional) — The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if decoder_guidance_scale is less than 1).
  • prompt_embeds (torch.Tensor, optional) — Pre-generated text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, text embeddings will be generated from prompt input argument.
  • prompt_embeds_pooled (torch.Tensor, optional) — Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, pooled text embeddings will be generated from prompt input argument.
  • negative_prompt_embeds (torch.Tensor, optional) — Pre-generated negative text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, negative_prompt_embeds will be generated from negative_prompt input argument.
  • negative_prompt_embeds_pooled (torch.Tensor, optional) — Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, negative_prompt_embeds_pooled will be generated from negative_prompt input argument.
  • num_images_per_prompt (int, optional, defaults to 1) — The number of images to generate per prompt.
  • generator (torch.Generator or List[torch.Generator], optional) — One or a list of torch generator(s) to make generation deterministic.
  • latents (torch.Tensor, optional) — Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random generator.
  • output_type (str, optional, defaults to "pil") — The output format of the generate image. Choose between: "pil" (PIL.Image.Image), "np" (np.array) or "pt" (torch.Tensor).
  • return_dict (bool, optional, defaults to True) — Whether or not to return a ImagePipelineOutput instead of a plain tuple.
  • callback_on_step_end (Callable, optional) — A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict). callback_kwargs will include a list of all tensors as specified by callback_on_step_end_tensor_inputs.
  • callback_on_step_end_tensor_inputs (List, optional) — The list of tensor inputs for the callback_on_step_end function. The tensors specified in the list will be passed as callback_kwargs argument. You will only be able to include variables listed in the ._callback_tensor_inputs attribute of your pipeline class.

Function invoked when calling the pipeline for generation.

Examples:

>>> import torch
>>> from diffusers import StableCascadePriorPipeline, StableCascadeDecoderPipeline

>>> prior_pipe = StableCascadePriorPipeline.from_pretrained(
...     "stabilityai/stable-cascade-prior", torch_dtype=torch.bfloat16
... ).to("cuda")
>>> gen_pipe = StableCascadeDecoderPipeline.from_pretrain(
...     "stabilityai/stable-cascade", torch_dtype=torch.float16
... ).to("cuda")

>>> prompt = "an image of a shiba inu, donning a spacesuit and helmet"
>>> prior_output = pipe(prompt)
>>> images = gen_pipe(prior_output.image_embeddings, prompt=prompt)
< > Update on GitHub

←Shap-E Overview→