Text-guided diffusion models generate images based on a given text prompt. The text prompt can include multiple concepts that the model should generate and it’s often desirable to weight certain parts of the prompt more or less.
Diffusion models work by conditioning the cross attention layers of the diffusion model with contextualized text embeddings (see the Stable Diffusion Guide for more information). Thus a simple way to emphasize (or de-emphasize) certain parts of the prompt is by increasing or reducing the scale of the text embedding vector that corresponds to the relevant part of the prompt. This is called “prompt-weighting” and has been a highly demanded feature by the community (see issue here).
How to do prompt-weighting in Diffusers
We believe the role of
diffusers is to be a toolbox that provides essential features that enable other projects, such as InvokeAI or diffuzers, to build powerful UIs. In order to support arbitrary methods to manipulate prompts,
diffusers exposes a
prompt_embeds function argument to many pipelines such as StableDiffusionPipeline, allowing to directly pass the “prompt-weighted”/scaled text embeddings to the pipeline.
The compel library provides an easy way to emphasize or de-emphasize portions of the prompt for you. We strongly recommend it instead of preparing the embeddings yourself.
Let’s look at a simple example. Imagine you want to generate an image of
"a red cat playing with a ball" as
from diffusers import StableDiffusionPipeline, UniPCMultistepScheduler pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config) prompt = "a red cat playing with a ball" generator = torch.Generator(device="cpu").manual_seed(33) image = pipe(prompt, generator=generator, num_inference_steps=20).images image
This gives you:
As you can see, there is no “ball” in the image. Let’s emphasize this part!
For this we should install the
pip install compel
and then create a
from compel import Compel compel_proc = Compel(tokenizer=pipe.tokenizer, text_encoder=pipe.text_encoder)
Now we emphasize the part “ball” with the
prompt = "a red cat playing with a ball++"
and instead of passing this to the pipeline directly, we have to process it using
prompt_embeds = compel_proc(prompt)
Now we can pass
prompt_embeds directly to the pipeline:
generator = torch.Generator(device="cpu").manual_seed(33) images = pipe(prompt_embeds=prompt_embeds, generator=generator, num_inference_steps=20).images image
We now get the following image which has a “ball”!
Similarly, we de-emphasize parts of the sentence by using the
-- suffix for words, feel free to give it
If your favorite pipeline does not have a
prompt_embeds input, please make sure to open an issue, the
diffusers team tries to be as responsive as possible.
Also, please check out the documentation of the compel library for more information.