Reproducibility

Before reading about reproducibility for Diffusers, it is strongly recommended to take a look at PyTorch’s statement about reproducibility.

PyTorch states that

completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. While one can never expect the same results across platforms, one can expect results to be reproducible across releases, platforms, etc… within a certain tolerance. However, this tolerance strongly varies depending on the diffusion pipeline and checkpoint.

In the following, we show how to best control sources of randomness for diffusion models.

Inference

During inference, diffusion pipelines heavily rely on random sampling operations, such as the creating the gaussian noise tensors to be denoised and adding noise to the scheduling step.

Let’s have a look at an example. We run the DDIM pipeline for just two inference steps and return a numpy tensor to look into the numerical values of the output.

from diffusers import DDIMPipeline
import numpy as np

model_id = "google/ddpm-cifar10-32"

# load model and scheduler
ddim = DDIMPipeline.from_pretrained(model_id)

# run pipeline for just two steps and return numpy tensor
image = ddim(num_inference_steps=2, output_type="np").images
print(np.abs(image).sum())

Running the above prints a value of 1464.2076, but running it again prints a different value of 1495.1768. What is going on here? Every time the pipeline is run, gaussian noise is created and step-wise denoised. To create the gaussian noise with torch.randn, a different random seed is taken every time, thus leading to a different result. This is a desired property of diffusion pipelines, as it means that the pipeline can create a different random image every time it is run. In many cases, one would like to generate the exact same image of a certain run, for which case an instance of a PyTorch generator has to be passed:

import torch
from diffusers import DDIMPipeline
import numpy as np

model_id = "google/ddpm-cifar10-32"

# load model and scheduler
ddim = DDIMPipeline.from_pretrained(model_id)

# create a generator for reproducibility
generator = torch.Generator(device="cpu").manual_seed(0)

# run pipeline for just two steps and return numpy tensor
image = ddim(num_inference_steps=2, output_type="np", generator=generator).images
print(np.abs(image).sum())

Running the above always prints a value of 1491.1711 - also upon running it again because we define the generator object to be passed to all random functions of the pipeline.

If you run this code snippet on your specific hardware and version, you should get a similar, if not the same, result.

Great! Now, we know how to write reproducible pipelines, but it gets a bit trickier since the above example only runs on the CPU. How do we also achieve reproducibility on GPU? In short, one should not expect full reproducibility across different hardware when running pipelines on GPU as matrix multiplications are less deterministic on GPU than on CPU and diffusion pipelines tend to require a lot of matrix multiplications. Let’s see what we can do to keep the randomness within limits across different GPU hardware.

To achieve maximum speed performance, it is recommended to create the generator directly on GPU when running the pipeline on GPU:

import torch
from diffusers import DDIMPipeline
import numpy as np

model_id = "google/ddpm-cifar10-32"

# load model and scheduler
ddim = DDIMPipeline.from_pretrained(model_id)
ddim.to("cuda")

# create a generator for reproducibility
generator = torch.Generator(device="cuda").manual_seed(0)

# run pipeline for just two steps and return numpy tensor
image = ddim(num_inference_steps=2, output_type="np", generator=generator).images
print(np.abs(image).sum())

Running the above now prints a value of 1389.8634 - even though we’re using the exact same seed! This is unfortunate as it means we cannot reproduce the results we achieved on GPU, also on CPU. Nevertheless, it should be expected since the GPU uses a different random number generator than the CPU.

To circumvent this problem, we created a randn_tensor function, which can create random noise on the CPU and then move the tensor to GPU if necessary. The function is used everywhere inside the pipelines allowing the user to always pass a CPU generator even if the pipeline is run on GPU:

import torch
from diffusers import DDIMPipeline
import numpy as np

model_id = "google/ddpm-cifar10-32"

# load model and scheduler
ddim = DDIMPipeline.from_pretrained(model_id)
ddim.to("cuda")

# create a generator for reproducibility
generator = torch.manual_seed(0)

# run pipeline for just two steps and return numpy tensor
image = ddim(num_inference_steps=2, output_type="np", generator=generator).images
print(np.abs(image).sum())

Running the above now prints a value of 1491.1713, much closer to the value of 1491.1711 when the pipeline is fully run on the CPU.

Finally, we noticed that more complex pipelines, such as UnCLIPPipeline are often extremely susceptible to precision error propagation and thus one cannot expect even similar results across different GPU hardware or PyTorch versions. In such cases, one has to make sure to run exactly the same hardware and PyTorch version for full Reproducibility.

Randomness utilities

randn_tensor