Diffusers documentation

Schedulers

You are viewing v0.14.0 version. A newer version v0.31.0 is available.
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Schedulers

Diffusers contains multiple pre-built schedule functions for the diffusion process.

What is a scheduler?

The schedule functions, denoted Schedulers in the library take in the output of a trained model, a sample which the diffusion process is iterating on, and a timestep to return a denoised sample. That’s why schedulers may also be called Samplers in other diffusion models implementations.

  • Schedulers define the methodology for iteratively adding noise to an image or for updating a sample based on model outputs.
    • adding noise in different manners represent the algorithmic processes to train a diffusion model by adding noise to images.
    • for inference, the scheduler defines how to update a sample based on an output from a pretrained model.
  • Schedulers are often defined by a noise schedule and an update rule to solve the differential equation solution.

Discrete versus continuous schedulers

All schedulers take in a timestep to predict the updated version of the sample being diffused. The timesteps dictate where in the diffusion process the step is, where data is generated by iterating forward in time and inference is executed by propagating backwards through timesteps. Different algorithms use timesteps that can be discrete (accepting int inputs), such as the DDPMScheduler or PNDMScheduler, or continuous (accepting float inputs), such as the score-based schedulers ScoreSdeVeScheduler or ScoreSdeVpScheduler.

Designing Re-usable schedulers

The core design principle between the schedule functions is to be model, system, and framework independent. This allows for rapid experimentation and cleaner abstractions in the code, where the model prediction is separated from the sample update. To this end, the design of schedulers is such that:

  • Schedulers can be used interchangeably between diffusion models in inference to find the preferred trade-off between speed and generation quality.
  • Schedulers are currently by default in PyTorch, but are designed to be framework independent (partial Jax support currently exists).
  • Many diffusion pipelines, such as StableDiffusionPipeline and DiTPipeline can use any of KarrasDiffusionSchedulers

Schedulers Summary

The following table summarizes all officially supported schedulers, their corresponding paper

Scheduler Paper
ddim Denoising Diffusion Implicit Models
ddim_inverse Denoising Diffusion Implicit Models
ddpm Denoising Diffusion Probabilistic Models
deis DEISMultistepScheduler
singlestep_dpm_solver Singlestep DPM-Solver
multistep_dpm_solver Multistep DPM-Solver
heun Heun scheduler inspired by Karras et. al paper
dpm_discrete DPM Discrete Scheduler inspired by Karras et. al paper
dpm_discrete_ancestral DPM Discrete Scheduler with ancestral sampling inspired by Karras et. al paper
stochastic_karras_ve Variance exploding, stochastic sampling from Karras et. al
lms_discrete Linear multistep scheduler for discrete beta schedules
pndm Pseudo numerical methods for diffusion models (PNDM)
score_sde_ve variance exploding stochastic differential equation (VE-SDE) scheduler
ipndm improved pseudo numerical methods for diffusion models (iPNDM)
score_sde_vp Variance preserving stochastic differential equation (VP-SDE) scheduler
euler Euler scheduler
euler_ancestral Euler Ancestral scheduler
vq_diffusion VQDiffusionScheduler
unipc UniPCMultistepScheduler
repaint RePaint scheduler

API

The core API for any new scheduler must follow a limited structure.

  • Schedulers should provide one or more def step(...) functions that should be called to update the generated sample iteratively.
  • Schedulers should provide a set_timesteps(...) method that configures the parameters of a schedule function for a specific inference task.
  • Schedulers should be framework-specific.

The base class SchedulerMixin implements low level utilities used by multiple schedulers.

SchedulerMixin

class diffusers.SchedulerMixin

< >

( )

Mixin containing common functions for the schedulers.

Class attributes:

  • _compatibles (List[str]) — A list of classes that are compatible with the parent class, so that from_config can be used from a class different than the one used to save the config (should be overridden by parent class).

from_pretrained

< >

( pretrained_model_name_or_path: typing.Dict[str, typing.Any] = None subfolder: typing.Optional[str] = None return_unused_kwargs = False **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike, optional) — Can be either:

    • A string, the model id of a model repo on huggingface.co. Valid model ids should have an organization name, like google/ddpm-celebahq-256.
    • A path to a directory containing the schedluer configurations saved using save_pretrained(), e.g., ./my_model_directory/.
  • subfolder (str, optional) — In case the relevant files are located inside a subfolder of the model repo (either remote in huggingface.co or downloaded locally), you can specify the folder name here.
  • return_unused_kwargs (bool, optional, defaults to False) — Whether kwargs that are not consumed by the Python class should be returned or not.
  • cache_dir (Union[str, os.PathLike], optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (i.e., do not try to download the model).
  • use_auth_token (str or bool, optional) — The token to use as HTTP bearer authorization for remote files. If True, will use the token generated when running transformers-cli login (stored in ~/.huggingface).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.

Instantiate a Scheduler class from a pre-defined JSON configuration file inside a directory or Hub repo.

It is required to be logged in (huggingface-cli login) when you want to use private or gated models.

Activate the special “offline-mode” to use this method in a firewalled environment.

save_pretrained

< >

( save_directory: typing.Union[str, os.PathLike] push_to_hub: bool = False **kwargs )

Parameters

  • save_directory (str or os.PathLike) — Directory where the configuration JSON file will be saved (will be created if it does not exist).

Save a scheduler configuration object to the directory save_directory, so that it can be re-loaded using the from_pretrained() class method.

SchedulerOutput

The class `SchedulerOutput` contains the outputs from any schedulers `step(...)` call.

class diffusers.schedulers.scheduling_utils.SchedulerOutput

< >

( prev_sample: FloatTensor )

Parameters

  • prev_sample (torch.FloatTensor of shape (batch_size, num_channels, height, width) for images) — Computed sample (x_{t-1}) of previous timestep. prev_sample should be used as next model input in the denoising loop.

Base class for the scheduler’s step function output.

KarrasDiffusionSchedulers

KarrasDiffusionSchedulers encompasses the main generalization of schedulers in Diffusers. The schedulers in this class are distinguished, at a high level, by their noise sampling strategy; the type of network and scaling; and finally the training strategy or how the loss is weighed.

The different schedulers, depending on the type of ODE solver, fall into the above taxonomy and provide a good abstraction for the design of the main schedulers implemented in Diffusers. The schedulers in this class are given below:

class diffusers.schedulers.KarrasDiffusionSchedulers

< >

( value names = None module = None qualname = None type = None start = 1 )

An enumeration.