Diffusers documentation

Using Diffusers with other modalities

Diffusers

Get started

🧨 Diffusers Quicktour Effective and efficient diffusion Installation

Tutorials

Overview Understanding pipelines, models and schedulers AutoPipeline Train a diffusion model Inference with PEFT Accelerate inference of text-to-image diffusion models

Using Diffusers

Loading & Hub

Overview Load pipelines, models, and schedulers Load and compare different schedulers Load community pipelines and components Load safetensors Load different Stable Diffusion formats Load adapters Push files to the Hub

Tasks

Overview Unconditional image generation Text-to-image Image-to-image Inpainting Depth-to-image

Techniques

Textual inversion Distributed inference with multiple GPUs Improve image quality with deterministic generation Control image brightness Prompt weighting Improve generation quality with FreeU

Specific pipeline examples

Overview Stable Diffusion XL SDXL Turbo Kandinsky ControlNet Shap-E DiffEdit Distilled Stable Diffusion inference Pipeline callbacks Create reproducible pipelines Community pipelines Contribute a community pipeline Latent Consistency Model-LoRA Latent Consistency Model Stable Video Diffusion

Training

Overview Create a dataset for training Adapt a model to a new task

Models

Unconditional image generation Text-to-image Stable Diffusion XL Kandinsky 2.2 Wuerstchen ControlNet T2I-Adapters InstructPix2Pix

Methods

Textual Inversion DreamBooth LoRA Custom Diffusion Latent Consistency Distillation Reinforcement learning training with DDPO

Taking Diffusers Beyond Images

Other Modalities

Optimization

Overview

General optimizations

Speed up inference Reduce memory usage PyTorch 2.0 xFormers Token merging DeepCache

Optimized model types

JAX/Flax ONNX OpenVINO Core ML

Optimized hardware

Metal Performance Shaders (MPS) Habana Gaudi

Conceptual Guides

Philosophy Controlled generation How to contribute? Diffusers' Ethical Guidelines Evaluating Diffusion Models

API

Main Classes

Configuration Logging Outputs

Loaders

IP-Adapter LoRA Single files Textual Inversion UNet PEFT

Models

Overview UNet1DModel UNet2DModel UNet2DConditionModel UNet3DConditionModel UNetMotionModel UViT2DModel VQModel AutoencoderKL AsymmetricAutoencoderKL Tiny AutoEncoder ConsistencyDecoderVAE Transformer2D Transformer Temporal Prior Transformer ControlNet

Pipelines

Overview aMUSEd AnimateDiff Attend-and-Excite AudioLDM AudioLDM 2 AutoPipeline BLIP-Diffusion Consistency Models ControlNet ControlNet with Stable Diffusion XL Dance Diffusion DDIM DDPM DeepFloyd IF DiffEdit DiT I2VGen-XL InstructPix2Pix Kandinsky 2.1 Kandinsky 2.2 Kandinsky 3 Latent Consistency Models Latent Diffusion MultiDiffusion MusicLDM Paint by Example Personalized Image Animator (PIA) PixArt-α Self-Attention Guidance Semantic Guidance Shap-E

Stable Diffusion

Overview Text-to-image Image-to-image Inpainting Depth-to-image Image variation Safe Stable Diffusion Stable Diffusion 2 Stable Diffusion XL SDXL Turbo Latent upscaler Super-resolution K-Diffusion LDM3D Text-to-(RGB, Depth), Text-to-(RGB-pano, Depth-pano), LDM3D Upscaler Stable Diffusion T2I-Adapter GLIGEN (Grounded Language-to-Image Generation)

Stable unCLIP Text-to-video Text2Video-Zero unCLIP UniDiffuser Value-guided sampling Wuerstchen

Schedulers

Overview CMStochasticIterativeScheduler ConsistencyDecoderScheduler DDIMInverseScheduler DDIMScheduler DDPMScheduler DEISMultistepScheduler DPMSolverMultistepInverse DPMSolverMultistepScheduler DPMSolverSDEScheduler DPMSolverSinglestepScheduler EulerAncestralDiscreteScheduler EulerDiscreteScheduler HeunDiscreteScheduler IPNDMScheduler KarrasVeScheduler KDPM2AncestralDiscreteScheduler KDPM2DiscreteScheduler LCMScheduler LMSDiscreteScheduler PNDMScheduler RePaintScheduler ScoreSdeVeScheduler ScoreSdeVpScheduler UniPCMultistepScheduler VQDiffusionScheduler

Internal classes

Overview Attention Processor Custom activation functions Custom normalization layers Utilities VAE Image Processor

You are viewing v0.26.3 version. A newer version v0.31.0 is available.

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Using Diffusers with other modalities

Diffusers is in the process of expanding to modalities other than images.

Example type	Colab	Pipeline
Molecule conformation generation		❌

More coming soon!

←Reinforcement learning training with DDPO Overview→

Using Diffusers with other modalities