Re-using seeds for fast prompt engineering

A common use case when generating images is to generate a batch of images, select one image and improve it with a better, more detailed prompt in a second run. To do this, one needs to make each generated image of the batch deterministic. Images are generated by denoising gaussian random noise which can be instantiated by passing a torch generator.

Now, for batched generation, we need to make sure that every single generated image in the batch is tied exactly to one seed. In 🧨 Diffusers, this can be achieved by not passing one generator, but a list of generators to the pipeline.

Let’s go through an example using runwayml/stable-diffusion-v1-5. We want to generate several versions of the prompt:

prompt = "Labrador in the style of Vermeer"

Let’s load the pipeline

>>> from diffusers import DiffusionPipeline

>>> pipe = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
>>> pipe = pipe.to("cuda")

Now, let’s define 4 different generators, since we would like to reproduce a certain image. We’ll use seeds 0 to 3 to create our generators.

>>> import torch

>>> generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(4)]

Let’s generate 4 images:

>>> images = pipe(prompt, generator=generator, num_images_per_prompt=4).images
>>> images

Ok, the last images has some double eyes, but the first image looks good! Let’s try to make the prompt a bit better while keeping the first seed so that the images are similar to the first image.

prompt = [prompt + t for t in [", highly realistic", ", artsy", ", trending", ", colorful"]]
generator = [torch.Generator(device="cuda").manual_seed(0) for i in range(4)]

We create 4 generators with seed 0, which is the first seed we used before.

Let’s run the pipeline again.

>>> images = pipe(prompt, generator=generator).images
>>> images