Paper: Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models (ICLR-2024)

Reviewer comment (Experiments):
I'm not totally convinced that semantics in padding tokens have so much impact. My own empirical experience is that the padding tokens usually have very small attention scores (=> close to 0 attention probabilities) compared to meaningful tokens, and thus their semantics, if any, add little to the image features. Though, due to the large number of padding tokens, it might accumulate to somewhat significant impact, especially when the prompt is short. This needs more systematic experiments to confirm, e.g. a diagram of the padding token impact with respect to the prompt length, where the prompts are randomly drawn from a pool.

Author response:
**1. More systematic experiments for the prompt with various length**

**1.1. Replacing meaningful tokens with padding tokens.**

We agree that the padding token contains less semantic information compared to meaningful tokens.
However, we observe that the padding token contains small yet useful semantic information, as demonstrated in Fig.2c in our main paper.
As shown in Fig.2c, we randomly select one padding token to replace all meaningful tokens. The generated images following this replacement have similar semantic information (see Fig.2c).
To further evaluate whether the padding token contains useful semantic information in text prompts of various lengths, we replace the meaningful tokens with not just one padding token, but multiple.
We use part of the padding token when its length exceeds that of the meaningful token (short sentence), and we copy multiple copies of the whole padding token when its length is shorter than meaningful token (long sentence).

In more detail, we first randomly chose 50 prompts from the prompt sets as mentioned in Sec.4 in our main paper. These text prompts include various syntactical structures, such as "A living area with a television and a table", "A black and white cat relaxing inside a laptop" and "There is a homemade pizza on a cutting board".
We add description words with lengths 8, 16, 32 and 56 following the initial text prompt $\mathbf{p}^{src}$ to obtain a long sentence, dubbed as $\mathbf{p}^{src+8ws}$, $\mathbf{p}^{src+16ws}$, $\mathbf{p}^{src+32ws}$, and $\mathbf{p}^{src+56ws}$, respectively.
For instance, when $\mathbf{p}^{src}$ is "A living area with a television and a table", $\mathbf{p}^{src+8ws}$ would be extended to "A living area with a television and a table, highly detailed and precision with extreme detail description".

We use Clipscore to evaluate that the generated images match the given prompt. In this case, we test our model under various length prompts ($\mathbf{p}^{src}$,$\mathbf{p}^{src+8ws}$, $\mathbf{p}^{src+16ws}$, $\mathbf{p}^{src+32ws}$, and $\mathbf{p}^{src+56ws}$) (see Tabel 1).
As shown in Tabel 1, the generated images corresponding padding replacement prompts also contain similar semantic information compared to the initial prompt.
The degeneration of the Clipscore is small (less than 0.11), indicating that the padding token also contains semantic information. Fig.21 (see in Appendix D. Fig.21) shows some more qualitative results.

Table 1: Comparison results with original tokens and their replacement version. We evaluate it with Clipscore.
| Method                    | $\mathbf{p}^{src}$ | $\mathbf{p}^{src+8ws}$ | $\mathbf{p}^{src+16ws}$ | $\mathbf{p}^{src+32ws}$ | $\mathbf{p}^{src+56ws}$ |
|---------------------------|---------------------|-------------------------|--------------------------|--------------------------|---------------------------|
| SD                        | 0.8208              | 0.8173                  | 0.8162                   | 0.8102                   | 0.8058                    |
| SD w/ replacement         | 0.7674              | 0.7505                  | 0.7479                   | 0.7264                   | 0.7035                    |



Related work also consider the padding embedding for other tasks. For example,  P2P  manipulates the padding  attention injection when conducing image-to-image translation.  P2P [1] swaps whole tokens attention,  including both the tokens and padding token attentions.

**References**

[1] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022

Reviewer comment (Evaluation):
CLIPscores in Table 1 are a bit confusing. Are they the similarity between the images and negative prompts?

Author response:
**4. The CLIPscores in Table 1 are a bit confusing, do they represent the similarity between the images and negative prompts?**

In our paper, Clipscore is a metric that evaluates the quality of a pair of a negative prompt
and an edited image. We have updated it in our updated paper.

Reviewer comment (Novelty):
The diffusion model is a hot topic in the machine learning and computer vision community, and the differences should be further highlighted.

Author response:
**1. The diffusion model is a hot topic in the machine learning and computer vision community, and further highlighting its differences is essential**

Indeed, diffusion models are a hot topic, and one of the most impactful applications is their usage for text-guided image editing. The ability to create realistic images that faithfully follow user-provided prompts is of great importance for many applications. However, existing latent-space diffusion models struggle to effectively suppress the generation of undesired content, which is explicitly requested to be omitted from the generated image in the prompt. Therefore, in our method, we aim to address this problem and propose an improved method for negative target suppression.

Reviewer comment (Experiments):
With this methodology, we can remove subjects from an image or add subjects to it. Is it possible to change one subject to another in one go? For example, can we change the “toothbrush” in “Girl holding toothbrush” image to a “pen”?

Author response:
**2. Is it possible to change one subject to another in one go?**

Subject replacement is a common task in various image editing methods [1-4].
We can edit an image by replacing subject with another using only the prompt (see in Appendix F. Fig.31 (the second, fourth, and sixth columns)).
We replace the text of the edited subject in the source prompt with the desired one to create the target prompt. Subsequently, we translate the real image using the target prompt into latent code. We then apply the same strategy as in the subsection on Appendix.F (**Attend-and-Excite similar results**) to obtain the edited image.
For example, we can replace the "toothbrush" in the "Girl holding toothbrush" image with the "pen".
The DetScore with "toothbrush" of the source image is 0.79, and the Clipscore with "pen" of the target image is 0.728.
We find that the brace is also replaced with the pen, as the cross attention map for "toothbrush" couples the toothbrush and the brace.
We will add subject replacement results to our paper in the final version.

**References**

[1] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.

[2] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. arXiv preprint arXiv:2302.03027, 2023.

[3] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794, 2022.

[4] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921–1930, 2023

Reviewer comment (Novelty):
From the algorithmic perspective, both improvement points are existing methods, and thus lack a certain level of novelty.

Author response:
$\textbf{1. From the algorithmic perspective, both improvement points are existing methods and thus lack a certain level of novelty}$

We would like to stress that for efficient text-guided image generation, the usage of negative lexemes is very important. They are known to be essential for humans to precisely communicate the desired image content. Addressing this problem for text-guided image generation is therefore of great importance to the community and the wide application of these methods.

Recently, both the pixel-space diffusion models (i.e.
DeepFloyd-IF) and the latent-space diffusion models struggle to effectively suppress the generation of undesired content, which is explicitly requested to be omitted
from the generated image in the prompt.  To achieve this goal, our work makes the following contributions: (I) Our analysis shows that the [EOT] embeddings contain significant, redundant and duplicated semantic information of the whole input
prompt (the whole embeddings). This needs to be taken into account when removing negative target
information. Therefore, we propose soft-weighted regularization to eliminate the negative target
information from the [EOT] embeddings.  Although SVD is a well-known approach, to our best knowledge, we are the first to use it to address this suppression problem. (II) To further suppress the negative target generation,
and encourage positive target content, we propose inference-time text embedding optimization. While attention mechanisms in our approach are known to the community for other tasks, it requires careful algorithmic design and a significant effort to incorporate the ideas of text embedding suppression into a single generative architecture.

Reviewer comment (Evaluation):
In the comparative experiments, it would be beneficial to specifically list the time and memory consumption ratios of this method compared to other methods, as this is necessary for a more application-oriented task.

Author response:
$\textbf{3. The time and memory consumption ratios of this method compared to other methods}$

In the following table, we report  the time and memory consumption ratios.   We randomly select 100 prompts and feed them into the SD model.  Compared to the baselines, we need additional time and memory consumption.  Note that, we aim to suppress the negative target generation in diffusion models. In  the Table 1 in our main paper,   For real-image negative target suppression,  we achieve the best score in both Clipscore and DetScore.

Table 1: Computational cost evaluation in baselines and ours.
| Method            | SD     | Negative Prompt | P2P     | ESD     | Concept-ablation | Forget-Me-Not | Inst-Inpaint | SEGA    | Ours    |
|-------------------|--------|-----------------|---------|---------|-------------------|---------------|--------------|---------|---------|
| Inference time (s/image)    | 7.35   | 7.42            | 15.434  | 14.407  | 5.240             | 13.231        | 1.093        | 20.157  | 24.617  |
| GPU Memory demand (GB)  | 12.3   | 12.3            | 12.4    | 9.9     | 9.7               | 21.8          | 9.8          | 11.8    | 14.3    |
| Tuning-free       | ✔      | ✔               | ✔       | ✘      | ✘                 | ✔             | ✘           | ✔       | ✔       |
| Image resolution  | 512x512| 512x512         | 512x512 | 512x512 | 512x512           | 512x512       | 256x256      | 512x512 | 512x512 |

Reviewer comment (Experiments):
In the first phase, this method uses coefficients to adjust the size of the negative information matrix to suppress the expression of negative information. If the singular value decomposition method is not employed, but instead, the entire matrix is multiplied by an attenuation factor, how would that affect the image editing results?

Author response:
**4. If the singular value decomposition method is not employed but instead the entire matrix is multiplied by an attenuation factor, how would that affect the image editing results?**

We evaluate the advised method involving an attenuation factor. We experimentally observed that employing an attenuation factor (e.g., 0.1) for the negative target embedding matrix would impact the positive target (see in Appendix D. Fig.20). Hence, using an attenuation factor leads to unexpected subject changes as well as changes to the target subject. This is due to the fact that the [EOT] embeddings contain significant information about the input prompt, including both the negative target and the positive target (see Sec 3.2). Furthermore, the selection of factors needs to be carefully performed for each image to achieve satisfactory suppression results.

Reviewer comment (Evaluation):
This work introduces some new matrix computations, such as the SVD in soft-weighted regularization and the attention map alignment in ITO. However, the authors do not discuss the additional computational overhead of these computations.

Author response:
$\textbf{1. The additional computational cost of SWR and ITO}$

We randomly select 100 prompts and feed them into the SD model.  As shown in the following Table, we report the average values for inference time (s/image) and GPU memory demand (GB), respectively. For a comparison of time and memory consumption ratios with other methods, also see the third question ( REVIEWER2 69SC).


Table: Computational cost evaluation for Inference time (s/image).
| Method | SWR | ITO | GPU Memory demand (GB) |
|----------|----------|----------|----------|
| Ours  | 0.134  | 24.483  | 14.3  |