Diffusers documentation

Using Diffusers with other modalities

Diffusers

Get started

🧨 Diffusers Quicktour Effective and efficient diffusion Installation

Tutorials

Overview Understanding pipelines, models and schedulers AutoPipeline Train a diffusion model Inference with PEFT

Using Diffusers

Loading & Hub

Overview Load pipelines, models, and schedulers Load and compare different schedulers Load community pipelines and components Load safetensors Load different Stable Diffusion formats Load adapters Push files to the Hub

Tasks

Overview Unconditional image generation Text-to-image Image-to-image Inpainting Depth-to-image

Techniques

Textual inversion Distributed inference with multiple GPUs Improve image quality with deterministic generation Control image brightness Prompt weighting Improve generation quality with FreeU

Specific pipeline examples

Overview Stable Diffusion XL SDXL Turbo Kandinsky ControlNet Shap-E DiffEdit Distilled Stable Diffusion inference Pipeline callbacks Create reproducible pipelines Community pipelines Contribute a community pipeline Latent Consistency Model-LoRA Latent Consistency Model Stable Video Diffusion

Training

Overview Create a dataset for training Adapt a model to a new task

Models

Unconditional image generation Text-to-image Stable Diffusion XL Kandinsky 2.2 Wuerstchen ControlNet T2I-Adapters InstructPix2Pix

Methods

Textual Inversion DreamBooth LoRA Custom Diffusion Latent Consistency Distillation Reinforcement learning training with DDPO

Taking Diffusers Beyond Images

Other Modalities

Optimization

Overview

General optimizations

Speed up inference Reduce memory usage PyTorch 2.0 xFormers Token merging

Optimized model types

JAX/Flax ONNX OpenVINO Core ML

Optimized hardware

Metal Performance Shaders (MPS) Habana Gaudi

Conceptual Guides

Philosophy Controlled generation How to contribute? Diffusers' Ethical Guidelines Evaluating Diffusion Models

API

Main Classes

Configuration Logging Outputs

Loaders

LoRA Single files Textual Inversion UNet

Models

Overview UNet1DModel UNet2DModel UNet2DConditionModel UNet3DConditionModel UNetMotionModel VQModel AutoencoderKL AsymmetricAutoencoderKL Tiny AutoEncoder ConsistencyDecoderVAE Transformer2D Transformer Temporal Prior Transformer ControlNet

Pipelines

Overview AltDiffusion AnimateDiff Attend-and-Excite Audio Diffusion AudioLDM AudioLDM 2 AutoPipeline BLIP-Diffusion Consistency Models ControlNet ControlNet with Stable Diffusion XL Cycle Diffusion Dance Diffusion DDIM DDPM DeepFloyd IF DiffEdit DiT InstructPix2Pix Kandinsky 2.1 Kandinsky 2.2 Kandinsky 3 Latent Consistency Models Latent Diffusion MultiDiffusion MusicLDM Paint by Example Parallel Sampling of Diffusion Models Pix2Pix Zero PixArt-α PNDM RePaint Score SDE VE Self-Attention Guidance Semantic Guidance Shap-E Spectrogram Diffusion

Stable Diffusion

Overview Text-to-image Image-to-image Inpainting Depth-to-image Image variation Safe Stable Diffusion Stable Diffusion 2 Stable Diffusion XL SDXL Turbo Latent upscaler Super-resolution LDM3D Text-to-(RGB, Depth), Text-to-(RGB-pano, Depth-pano), LDM3D Upscaler Stable Diffusion T2I-Adapter GLIGEN (Grounded Language-to-Image Generation)

Stable unCLIP Stochastic Karras VE Text-to-image model editing Text-to-video Text2Video-Zero unCLIP Unconditional Latent Diffusion UniDiffuser Value-guided sampling Versatile Diffusion VQ Diffusion Wuerstchen

Schedulers

Overview CMStochasticIterativeScheduler ConsistencyDecoderScheduler DDIMInverseScheduler DDIMScheduler DDPMScheduler DEISMultistepScheduler DPMSolverMultistepInverse DPMSolverMultistepScheduler DPMSolverSDEScheduler DPMSolverSinglestepScheduler EulerAncestralDiscreteScheduler EulerDiscreteScheduler HeunDiscreteScheduler IPNDMScheduler KarrasVeScheduler KDPM2AncestralDiscreteScheduler KDPM2DiscreteScheduler LCMScheduler LMSDiscreteScheduler PNDMScheduler RePaintScheduler ScoreSdeVeScheduler ScoreSdeVpScheduler UniPCMultistepScheduler VQDiffusionScheduler

Internal classes

Overview Attention Processor Custom activation functions Custom normalization layers Utilities VAE Image Processor

You are viewing v0.24.0 version. A newer version v0.31.0 is available.

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Using Diffusers with other modalities

Diffusers is in the process of expanding to modalities other than images.

Example type	Colab	Pipeline
Molecule conformation generation		❌

More coming soon!

←Reinforcement learning training with DDPO Overview→

Using Diffusers with other modalities