🧨 Diffusers

🤗 Diffusers provides pretrained vision and audio diffusion models, and serves as a modular toolbox for inference and training.

More precisely, 🤗 Diffusers offers:

• State-of-the-art diffusion pipelines that can be run in inference with just a couple of lines of code (see Using Diffusers) or have a look at Pipelines to get an overview of all supported pipelines and their corresponding papers.
• Various noise schedulers that can be used interchangeably for the preferred speed vs. quality trade-off in inference. For more information see Schedulers.
• Multiple types of models, such as UNet, can be used as building blocks in an end-to-end diffusion system. See Models for more details
• Training examples to show how to train the most popular diffusion model tasks. For more information see Training.

🧨 Diffusers Pipelines

The following table summarizes all officially supported pipelines, their corresponding paper, and if available a colab notebook to directly try them out.

Pipeline	Paper	Tasks	Colab
alt_diffusion	AltDiffusion	Image-to-Image Text-Guided Generation
audio_diffusion	Audio Diffusion	Unconditional Audio Generation
controlnet	ControlNet with Stable Diffusion	Image-to-Image Text-Guided Generation	[
cycle_diffusion	Cycle Diffusion	Image-to-Image Text-Guided Generation
dance_diffusion	Dance Diffusion	Unconditional Audio Generation
ddpm	Denoising Diffusion Probabilistic Models	Unconditional Image Generation
ddim	Denoising Diffusion Implicit Models	Unconditional Image Generation
latent_diffusion	High-Resolution Image Synthesis with Latent Diffusion Models	Text-to-Image Generation
latent_diffusion	High-Resolution Image Synthesis with Latent Diffusion Models	Super Resolution Image-to-Image
latent_diffusion_uncond	High-Resolution Image Synthesis with Latent Diffusion Models	Unconditional Image Generation
paint_by_example	Paint by Example: Exemplar-based Image Editing with Diffusion Models	Image-Guided Image Inpainting
pndm	Pseudo Numerical Methods for Diffusion Models on Manifolds	Unconditional Image Generation
score_sde_ve	Score-Based Generative Modeling through Stochastic Differential Equations	Unconditional Image Generation
score_sde_vp	Score-Based Generative Modeling through Stochastic Differential Equations	Unconditional Image Generation
semantic_stable_diffusion	Semantic Guidance	Text-Guided Generation
stable_diffusion_text2img	Stable Diffusion	Text-to-Image Generation
stable_diffusion_img2img	Stable Diffusion	Image-to-Image Text-Guided Generation
stable_diffusion_inpaint	Stable Diffusion	Text-Guided Image Inpainting
stable_diffusion_panorama	MultiDiffusion	Text-to-Panorama Generation
stable_diffusion_pix2pix	InstructPix2Pix	Text-Guided Image Editing
stable_diffusion_pix2pix_zero	Zero-shot Image-to-Image Translation	Text-Guided Image Editing
stable_diffusion_attend_and_excite	Attend and Excite for Stable Diffusion	Text-to-Image Generation
stable_diffusion_self_attention_guidance	Self-Attention Guidance	Text-to-Image Generation
stable_diffusion_image_variation	Stable Diffusion Image Variations	Image-to-Image Generation
stable_diffusion_latent_upscale	Stable Diffusion Latent Upscaler	Text-Guided Super Resolution Image-to-Image
stable_diffusion_2	Stable Diffusion 2	Text-to-Image Generation
stable_diffusion_2	Stable Diffusion 2	Text-Guided Image Inpainting
stable_diffusion_2	Depth-Conditional Stable Diffusion	Depth-to-Image Generation
stable_diffusion_2	Stable Diffusion 2	Text-Guided Super Resolution Image-to-Image
stable_diffusion_safe	Safe Stable Diffusion	Text-Guided Generation
stable_unclip	Stable unCLIP	Text-to-Image Generation
stable_unclip	Stable unCLIP	Image-to-Image Text-Guided Generation
stochastic_karras_ve	Elucidating the Design Space of Diffusion-Based Generative Models	Unconditional Image Generation
unclip	Hierarchical Text-Conditional Image Generation with CLIP Latents	Text-to-Image Generation
versatile_diffusion	Versatile Diffusion: Text, Images and Variations All in One Diffusion Model	Text-to-Image Generation
versatile_diffusion	Versatile Diffusion: Text, Images and Variations All in One Diffusion Model	Image Variations Generation
versatile_diffusion	Versatile Diffusion: Text, Images and Variations All in One Diffusion Model	Dual Image and Text Guided Generation
vq_diffusion	Vector Quantized Diffusion Model for Text-to-Image Synthesis	Text-to-Image Generation

Note: Pipelines are simple examples of how to play around with the diffusion systems as described in the corresponding papers.