Schedulers

Diffusers contains multiple pre-built schedule functions for the diffusion process.

What is a scheduler?

The schedule functions, denoted Schedulers in the library take in the output of a trained model, a sample which the diffusion process is iterating on, and a timestep to return a denoised sample. That’s why schedulers may also be called Samplers in other diffusion models implementations.

• Schedulers define the methodology for iteratively adding noise to an image or for updating a sample based on model outputs.
  • adding noise in different manners represent the algorithmic processes to train a diffusion model by adding noise to images.
  • for inference, the scheduler defines how to update a sample based on an output from a pretrained model.
• Schedulers are often defined by a noise schedule and an update rule to solve the differential equation solution.

Discrete versus continuous schedulers

All schedulers take in a timestep to predict the updated version of the sample being diffused. The timesteps dictate where in the diffusion process the step is, where data is generated by iterating forward in time and inference is executed by propagating backwards through timesteps. Different algorithms use timesteps that can be discrete (accepting int inputs), such as the DDPMScheduler or PNDMScheduler, or continuous (accepting float inputs), such as the score-based schedulers ScoreSdeVeScheduler or ScoreSdeVpScheduler.

Designing Re-usable schedulers

The core design principle between the schedule functions is to be model, system, and framework independent. This allows for rapid experimentation and cleaner abstractions in the code, where the model prediction is separated from the sample update. To this end, the design of schedulers is such that:

• Schedulers can be used interchangeably between diffusion models in inference to find the preferred trade-off between speed and generation quality.
• Schedulers are currently by default in PyTorch, but are designed to be framework independent (partial Jax support currently exists).
• Many diffusion pipelines, such as StableDiffusionPipeline and DiTPipeline can use any of KarrasDiffusionSchedulers

Schedulers Summary

The following table summarizes all officially supported schedulers, their corresponding paper

Scheduler	Paper
ddim	Denoising Diffusion Implicit Models
ddim_inverse	Denoising Diffusion Implicit Models
ddpm	Denoising Diffusion Probabilistic Models
deis	DEISMultistepScheduler
singlestep_dpm_solver	Singlestep DPM-Solver
multistep_dpm_solver	Multistep DPM-Solver
heun	Heun scheduler inspired by Karras et. al paper
dpm_discrete	DPM Discrete Scheduler inspired by Karras et. al paper
dpm_discrete_ancestral	DPM Discrete Scheduler with ancestral sampling inspired by Karras et. al paper
stochastic_karras_ve	Variance exploding, stochastic sampling from Karras et. al
lms_discrete	Linear multistep scheduler for discrete beta schedules
pndm	Pseudo numerical methods for diffusion models (PNDM)
score_sde_ve	variance exploding stochastic differential equation (VE-SDE) scheduler
ipndm	improved pseudo numerical methods for diffusion models (iPNDM)
score_sde_vp	Variance preserving stochastic differential equation (VP-SDE) scheduler
euler	Euler scheduler
euler_ancestral	Euler Ancestral scheduler
vq_diffusion	VQDiffusionScheduler
unipc	UniPCMultistepScheduler
repaint	RePaint scheduler

API

The core API for any new scheduler must follow a limited structure.

• Schedulers should provide one or more def step(...) functions that should be called to update the generated sample iteratively.
• Schedulers should provide a set_timesteps(...) method that configures the parameters of a schedule function for a specific inference task.
• Schedulers should be framework-specific.

The base class SchedulerMixin implements low level utilities used by multiple schedulers.

SchedulerMixin

SchedulerOutput

The class `SchedulerOutput` contains the outputs from any schedulers `step(...)` call.

KarrasDiffusionSchedulers

KarrasDiffusionSchedulers encompasses the main generalization of schedulers in Diffusers. The schedulers in this class are distinguished, at a high level, by their noise sampling strategy; the type of network and scaling; and finally the training strategy or how the loss is weighed.

The different schedulers, depending on the type of ODE solver, fall into the above taxonomy and provide a good abstraction for the design of the main schedulers implemented in Diffusers. The schedulers in this class are given below: