Paper: MoLE: Human-centric Text-to-image Diffusion with Mixture of Low-rank Experts (ICLR-2024)

Reviewer comment (Experiments):
The chosen SD v1.5 is somewhat outdated, since the stable diffusion team continually updates their model to better generate human content. In SD v2.1, released about 10 months ago, they claim that "the new release delivers improved anatomy and hands" by reducing the number of filtered-out people in the dataset. It would be better to evaluate whether the proposed method could still boost human content generation quality upon this newer release.

Author response:
**Q2:** The chosen SD v1.5 is somewhat an outdated baseline, since the stable diffusion team is also continually updating their model toward better generation of human content. In SD v2.1, they claim that "the new release delivers improved anatomy and hands" by reducing the number of the filtered-out people in the dataset. It would be better to see whether the proposed method could still boost the human content generation quality upon this release.

**A2:** We thank the reviewer and follow this kind suggestion to perform an experiment on SD v2.1. We evaluate MoLE and SD v2.1 using COCO Human prompt. The experiment shows that MoLE produces $20.45 \pm 0.09$ for hps, outperforming SD v2.1 ($20.17 \pm 0.08$). This result further verifies the effectiveness of our methods.

Reviewer comment (Reproducibility):
There are concerns about reproducibility — specifically, whether the good results presented in the paper are carefully selected few examples.

Author response:
**Q3:** I have some concerns about reproducibility: whether the good results presented in the paper are carefully selected few ones.

**A3:** For generated images presented in the paper, we repeat 3 times for each prompt and pick the best one for better visualization. And we also follow this process to generate images for other methods to ensure a fair comparison of visualizations in Figure 8 and Figure 9 and qualitatively verify the effectiveness of our method. Besides qualitative results, the quantitative results also demonstrate the effectiveness of our method compared to others. Additionally, we release our code and models [here](https://drive.google.com/drive/folders/1W8qKJG2jNQJ_Y2jlqn1w8VJu_c1ahLC2?usp=drive_link) for reproducibility.

Finally, we feel sorry about this absence. Over the recent period, we concentrated on improving and expanding the size of our human-centric dataset (v2), e.g., merging more high-quality human-centric images by filtering other published datasets. More importantly, we have increased the amount of close-ups of hand images.  By manual cropping, we now have collected around **20k high-quality** close-ups of hand images. We present some of them at https://sites.google.com/view/response-close-up-of-hand/homepage . We are willing to release these high-quality close-ups of hand images to boost the development of image-generation communities.

Reviewer comment (Reproducibility):
Reproducibility concerns could be addressed by either providing the trained model and code (which were not found on the provided anonymous webpage near the review deadline) or conducting larger-scale quantitative experiments in Fig. 5 with SD XL or SD v2.1.

Author response:
**Q3:** I have some concerns about reproducibility: whether the good results presented in the paper are carefully selected few ones.

**A3:** For generated images presented in the paper, we repeat 3 times for each prompt and pick the best one for better visualization. And we also follow this process to generate images for other methods to ensure a fair comparison of visualizations in Figure 8 and Figure 9 and qualitatively verify the effectiveness of our method. Besides qualitative results, the quantitative results also demonstrate the effectiveness of our method compared to others. Additionally, we release our code and models [here](https://drive.google.com/drive/folders/1W8qKJG2jNQJ_Y2jlqn1w8VJu_c1ahLC2?usp=drive_link) for reproducibility.

Finally, we feel sorry about this absence. Over the recent period, we concentrated on improving and expanding the size of our human-centric dataset (v2), e.g., merging more high-quality human-centric images by filtering other published datasets. More importantly, we have increased the amount of close-ups of hand images.  By manual cropping, we now have collected around **20k high-quality** close-ups of hand images. We present some of them at https://sites.google.com/view/response-close-up-of-hand/homepage . We are willing to release these high-quality close-ups of hand images to boost the development of image-generation communities.

Reviewer comment (Reproducibility):
The preprocessing of the human-in-the-scene dataset is vaguely explained. For example, they seem to "train a VGG19 to filter out images that contain little information about people" but how such images were identified and how the VGG19 was trained are not present.

Author response:
We are sorry for the confusion. In the preprocessing, we regard an image where the most area is covered by the background as a negative sample except those that major body is approximately centered in the image. We give several intuitive cases of the negative sample in  (See Visualization for Q2 in https://sites.google.com/view/response-reviewer-39sw/homepage). Images that do not meet this standard are regarded as positive samples. However, to be honest, considering that the collection criterion is relatively vague, could not be quantified, and thereby varies among different people, the collected samples inevitably have a certain degree of subjective bias. To alleviate this issue, we try our best to keep justice and repeat the filtering process multiple times as we mentioned in our paper.  

To train the VGG19, we manually collect around 300 positive samples and 300 negative samples as training set, and we also collect around 100 positive samples and 100 negative samples as val set. When training the VGG19, we set batch size to 128, set learning rate to 0.001, and use random flip as data augmentation method. We train the model for 200 epochs and use the best-performing model for subsequent classification. We have added the training details and illustrations of negtive samples in our paper.

Reviewer comment (Experiments):
The usefulness of the fine-tuned SDM in generic image generation is not addressed.

Author response:
We thank the reviewer for this kind suggestion. Our model also can generate non-human-centric images. To verify it, we try some prompts without human-related keywords  (See Visualization for Q3 in https://sites.google.com/view/response-reviewer-39sw/homepage). Results show that our method is capable of generating high-quality and diverse images including non-human-centric entities, e.g., animals, scenery, etc. We speculate this is because the human-centric dataset also contains these entities that interact with humans in an image. As a result, the model learns these concepts. However, it is worth noting that our model may not be better at generic image generation than the generic models as MoLE is trained on human-centric images. We have added the discussion to our paper.

Reviewer comment (Experiments):
The method has obvious limitations, as it requires the collection of substantial amounts of data. How to measure the quality of such data, what about the computational burden of using such data, etc, remain unresolved issues. Besides, the method is ineffective in scenarios involving multiple individuals would also limit its practical value.

Author response:
We may need to emphasize that collecting such a large amount (around 1M) of high-quality images from websites is also one of our contributions. It took us around one month to finish. And to advance the development of the research community, we are willing to release this high-quality dataset, which would significantly save the time and effort of subsequent researchers.

In terms of quality, most of our data is obtained from well-known free websites that contain exquisite high-resolution images, e.g., https://www.pexels.com/search/people/, and https://unsplash.com/s/photos/people. The quality is relatively high.

Regarding the computational load, subsequent researchers can leverage a portion of our dataset to strike a balance between effectiveness and computational burden after the dataset is released. It is worth mentioning that MoLE is resource-friendly and can be trained in a single A100 40G GPU.

Finally, the limitation in scenarios involving multiple individuals is primarily because most of our collected images are single person. Since we have verified the effectiveness of MoLE in scenarios involving a single individual,  we can strategically expand the dataset to involve more images with multiple individuals to mitigate this issue, which we leave as future work.

Reviewer comment (Evaluation):
I am curious on the results in Table 1. The numbers are hard to evaluate on the performance improvement of this work.

Author response:
Since reviewer is curious about the results in Table 1, we describe how we obtain such scores in detail. By doing so, we hope to prove that the score is obtained justly and is effective in validating the performance improvements. Specifically, we randomly sample 3k prompts from two human benchmarks to generate images and calculate metrics, HPS and IR, using their released official evaluation model. Afterward, we average the produced HPS and IR scores for the 3k prompts. We repeat the whole process three times and use the averaged HPS and IR to calculate the mean and standard deviations. Besides, To further verify the effectiveness, we also adopt another two metrics including clip alignment and FID to evaluate the performance. We follow BLIP-Diffusion[4] and HumanSD[5] to report the results using COCO Human Prompts and Human-centric datasets. Clip alignment is to measure the text-image alignment (the higher, the better) and FID is to measure the distribution distance between generated images and real images (the lower, the better). MoLE produces 27.33 for clip alignment and outperform SD v1.5 (26.87). MoLE produces 64.37 for FID and outperform SD v1.5 (69.82). These results indicate that MoLE is also superior to SD v1.5.

[1] Scaling Vision with Sparse Mixture of Experts NeurIPS2021

[2] Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts. CVPR2023

[3] ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. Arxiv2023

[4] BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing. Arxiv2023

[5] HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation. ICCV2023

Reviewer comment (Experiments):
The detailed information and statistics of collected data are missing. To ensure the quality of collected human-centric images, the authors need to analyze the characteristics and attributes of the dataset in a systematic and quantitative manner. For example, the distribution of racial groups and gender, the diversity of text prompts including semantic context, or the distribution of face size and the number of faces have to be provided to understand the characteristics of the dataset.

Author response:
Our human-centric dataset has two aspects of advantages:

- **Firstly, our human-centric dataset is of high quality.** As we mentioned in **A1**, our human-centric dataset consists of images of high resolution (basically over 1024 × 2048) with large file size, e.g., 1M, 3M, 5M, 10M, etc, collected from websites (e.g., [https://www.pexels.com/search/people/](https://www.pexels.com/search/people/), and [https://unsplash.com/s/photos/people](https://unsplash.com/s/photos/people)). The reviewer can simply click these links to have a see. In contrast, the average height and width of the human-centric images from LAION2b-en are approximately 455 and 415 respectively. Most of them are between 320 and 357. Compared to our human-centric dataset, they fail to provide sufficient and high-quality details.
- **Secondly, our human-centric dataset is comprehensive.** As the face and hand are relatively hard to generate naturally compared to other parts as discussed in the AIGC community, besides the collected human-in-the-scene subset, we also provide two high-quality close-up of face and hand subsets, especially the close-up of hand images. **This is absent in human-centric images from LAION2b-en as mentioned in A1**. Additionally, through meticulous manual cropping, we now have curated 20k high-quality hand close-ups (https://sites.google.com/view/response-close-up-of-hand/homepage). **To the best of our knowledge, this quantity of close-up hand images is absent in prior related studies.** It will significantly help to address challenges associated with generating natural-looking hands and propel the advancement of human-centric image generation for subsequent researchers.

  Finally, the contribution of our proposed human-centric dataset is also acknowledged by **Reviewer FSG6 and 39sw**.

Reviewer comment (Theory):
Although the design of the proposed method, MoLE, makes sense, its design is not well-motivated. Please refer to the questions below for the details.

Author response:
The proposed method, MoLE, is primarily motivated by the low-rank refinement phenomenon as shown in Figure 1 where a specialized low-rank module using **a proper scale weight** is capable of refining the corresponding part of person.

We agree with the reviewer that fine-tuning LoRA weights for a specific purpose is a widespread approach. **However, it is only one stage of our method, i.e., Stage 2, to prepare the customized experts.**

In fact, inspired by the low-rank refinement phenomenon, **the novelty of MoLE lies in Stage 3 where we leverage a soft assignment mechanism in MoE to adaptively activate different experts with suitable weights.** Additionally, we introduce an extra **element-wise local assignment** and combine it with a normal global assignment to achieve finer-grained activation. The effectiveness is also verified by the ablation results in Table 3.

Our method is also different from other generation methods leveraging MoE, e.g., ERNIE-ViLG and eDiff-i. They employ different experts to denoise the whole image in respective stages. In contrast, we primarily leverage a soft assignment mechanism in MoE to adaptively activate customized hand and face experts to refine the corresponding part.

Finally, the novelty of our method is also acknowledged by **Reviewer Y1yT**.

Reviewer comment (Experiments):
In Section 1, the authors postulate “the absence of comprehensive and high-quality human centric data within the training dataset LAION5B…”, but the claim does not have supporting analysis. Is there any specific analysis to show that LAION5B does not include comprehensive and high-quality human centric images?

Author response:
Yes, there are two reasons. Firstly, in our preliminary experiments, we analyzed human-centric images from LAION2b-en. We randomly sample 35w human-centric images and find that the average height and width are 455 and 415 respectively. Most of them are between 320 and 357. Compared to the image we collected (basically over 1024 × 2048), the image in LAION2b-en fails to provide sufficient and high-quality details.

Secondly, in human-centric images from LAION2b-en, We randomly sample 1000 images each time and repeat the process three times, in which we find little (almost zero) high-quality close-ups of face and hand (especially hand). This makes the human-centric images from LAION2b-en relatively limited in providing comprehensive human-centric knowledge. In contrast, based on such motivation, our work bridges this gap by proposing a relatively comprehensive and high-quality human-centric dataset. We will add these specific analysis to our paper to support our opinion.

Reviewer comment (Evaluation):
In Section 1, the authors also claim “faces and hands represent the two most complicated parts due to high variability, making them challenging to be generated naturally.” However, although MoLE adopts a soft mixture of LoRAs, the capability of T2I model is not much increased compared with Stable Diffusion v1.5. Can the proposed method synthesize naturally generated faces and hands, while significantly outperforming previous methods? Figure 1 and Figure 8 still show that the proposed method provides unnaturally generated eyes and hands.

Author response:
We do not agree with the reviewer that the capability of MoLE is not much increased compared with Stable Diffusion v1.5. We take metric HPS on COCO for example. As shown in Table 1, compared with SD v1.5, SD-XL improves HPS by 0.93% and increases the model size from 4.3G to 26.4G. In contrast, MoLE improves HPS by 0.36% but only increases the model size by 0.8G, which means MoLE is more efficient. The results also indicate that MoLE is resource-friendly. MoLE can be trained in a single A100 40G GPU.

Moreover, in Figure 8 and Figure 9, by qualitatively comparing the face and hand of generated images from MoLE and SD XL, we find that SD XL also generates unnatural hand and unrealistic face while our results are more realistic. When it comes to SD v1.5 in Figure 8 and Figure 9, the generated results are even worse. Our user study in Figure 5 also supports our conclusions, especially the hand quality, face quality, and overall quality.

Finally, we have to point out that Figure 1 is not the result of our method. It is just an illustration of our motivation.

Reviewer comment (Experiments):
One of the main reasons why a T2I model cannot generate high-quality faces and hands is that complex attributes and details of faces and hands are often located in a small region, considering synthesizing high-quality images with small objects is difficult for T2I models. The reviewer wonders about the authors’ opinion and the reason why the authors focus on close-up faces and hands for fine-tuning in Stage 2 and Stage 3.

Author response:
We agree with the reviewer's opinion that complex attributes and details of faces and hands are often located in a small region. **That is also one of the reasons why we collect the close-up of face and hand besides the human-in-the-scene subset.** Moreover, we use two customized low-rank experts to learn specific knowledge from the two close-up datasets. It is worth mentioning that we are still scaling the data quantity. By manual cropping, we now have collected around 20k high-quality close-ups of hand images.

We also want to supplement another two reasons. Firstly, as we mentioned in our paper, humans have abundant face expressions, (e.g., laughing and crying) and hand gestures (spread and curve), resulting in high variability which is also considerably challenge for generation model. Secondly, the hand and face are the most familiar and sensitive subjects for human observation, If there's even a slight anomaly, humans can readily perceive it. Hence, from this view, the standard of generating natural face and hand is strict, which also makes face and hand difficult to synthesize.

As for the reason, we are motivated by the low-rank refinement phenomenon in Figure 1. We observe that the low-rank module is capable of refining the corresponding part when using a proper scale weight. Hence, in Stage 2, we use the two close-up datasets to train two low-rank modules as experts. In Stage 3, we introduce local and global assignments in a MoE form. By training, we hope them to adaptively activate different experts with suitable weights.

Reviewer comment (Experiments):
Can the proposed MoE exploit numerous LoRA experts more than two? Can the proposed method support the adoption of multiple LoRA experts for face and hand, respectively?

Author response:
Yes is the answer.  It is easy for MoLE to incorporate more LoRA experts. Specifically, we can train numerous LoRA experts in stage 2 and in stage 3 adjust the parameter e in local assignment gating layer in Eq (5) and parameter value 2 in global assignment gating layer in Eq (6) to the corresponding number of involved experts. Considering that we primarily focus on alleviating the unnatural results of face and hand, we adopt two experts, i.e., face and hand experts. Regarding the second question, it parallels the first one, involving multiple experts concentrating on the same aspect, such as the face. Our method is also compatible. For example, we can train three face experts for the white race, black race, and yellow race respectively. Also, we can train two face experts for women and men respectively. Then follow the modification described in the first question.

Reviewer comment (Theory):
What are the details of each low-rank expert $E_\text{face}$ and $E_\text{hand}$? Since they are not formally defined and described, the reviewer wonders about the exact operation (including mathematical formulations) of each expert. In addition, the operation of the learnable gating layer $G$ is not formally defined.

Author response:
We speculate that the reviewer may be confused by the illustration of Figure 4 "Fine-tuned UNet layer" in stage 3. We deeply apologize and feel a profound sense of remorse for this confusion as for the sake of aesthetics,  we overlooked some details. We may need to raise the reviewer's attention to the presented caption in Figure 4 and the description in Eq (8) and in Section 3.1 Low-rank Adaptation (LoRA) part. In MoLE, for **every linear layer of the UNet**, each expert consists of two low-rank matrixes. As for the learnable gating layer $G$, **it is a linear layer parameterized by $\phi$ and $\omega$ in local and global assignment respectively**. We will redraw Figure 4 and emphasize the operation position to avoid potential confusion. We also thank the reviewer for proposing the question to catch our attention.

Reviewer comment (Experiments):
The fairness of the comparison of experimental results is questionable. First, since MoLE further trains SD v1.5, the results that outperform SD v1.5 are natural and do not support the effectiveness of MoLE. In addition, why is the CLIP text encoder further trained together with U-Net? The reviewer suggests considering a user study to compare MoLE and fine-tuned SD v1.5.

Author response:
We understand the reviewer's concern. However, we are not conducting continuous training on the dataset utilized for training SD v1.5; rather, we train it on a new dataset with a distinct distribution, namely, our human-centric dataset. More importantly, comparing the proposed method with baseline is quite common in previous works [1][2][3][4][5][6] that also leverage new datasets and propose new methods. We believe that this comparison is to verify the effectiveness of the whole strategy including collected dataset and proposed algorithm (note that not only the algorithm). Consequently, to verify the effectiveness of each, in our work, we compare the enhancements of stage 1 and stage 3 that correspond to the effectiveness of collected dataset and proposed algorithm. Please see Section 4.3.1 and Tab.2. In Tab 2, stage 3 further improves the performance, verifying the effectiveness of proposed algorithm.

As for the reason of jointly training CLIP text encoder and UNet, we simply follow BLIP-Diffusion[6] and this is verified in BLIP-Diffusion to be helpful for better image-text alignment. See Table 2 in BLIP-Diffusion.

Finally, we follow the reviewer's advice and conduct a user study to compare with fine-tuned SD v1.5 following the process reported in our paper. The results below show that MoLE still obtains higher voting compared to fine-tuned SD v1.5, especially in hand and face quality.

| User Study | Alignment | Hand Quality | Face Quality | Overall Quality |
| -------- | -------- | -------- |-------- | -------- |
| Fine-tuned SD v1.5     | 46%     | 20%    |28%     | 34%     |
| MoLE     | 54%    |80%    | 72%    |66%     |

Reviewer comment (Experiments):
After fine-tuning of SD v1.5 in Stage 1, is the model capable of generating high quality and diverse images including non-human-centric entities such as animals, foods, landscapes, etc.?

Author response:
It is an interesting question. We try some prompts without human-related key words from COCO Caption. Results shows that it is capable to generate high quality and diverse images including non-human-centric entities. (See Visualization for Q7 in https://sites.google.com/view/response-reviewer-b5sw/homepage). We speculate this is because that the human-centric dataset also contains these entities that interact with human in an image. As a result, the model learn these concepts. But we may need to acknowledge that it may not be superior over SD v1.5 in the aspect of non-human-centric generation as our dataset primarily involves human-centric images.

Reviewer comment (Evaluation):
The authors claim that the degradation of performance in Stage 2 results from overfitting. That is, does the training loss decrease but validation loss increase? In addition, can the model after Stable 1 successfully generate images for the prompts in Figure 6?

Author response:
We do not observe such a phenomenon. Here, we mean that the generated image resembles the distribution of the training dataset. For example, in our experiment, face expert is trained on the close-up of face called FFHQ where a human's face covers the most area of an image. Thus, when equipped with face expert, the generated image could be similar to the training data like Figure 6 (left) in which the face also covers most area of the image. We will rephrase the sentence for clarity.

Secondly, for the reviewer's second question, we try to generate image and find the model after Stage 1 can successfully generate images for the prompts in Figure 6, But the details of face and hand are not so good.  (See Visualization for Q9 in https://sites.google.com/view/response-reviewer-b5sw/homepage)

Reviewer comment (Experiments):
This paper describes that 3K prompts are used in Section 4.2 and 1K prompts are used in Section 4.3.2. However, why are the reported performances (mean, std) of MoLE in Table 1 and Table 2 (+Stage 3) exactly the same? Since Table 2 uses random sampling of 1K prompts among the 3K prompts, the HPS and IR scores cannot be exactly the same with the reported values in Table 1, considering the standard deviations (0.07, 1.49).

Author response:
Sorry for the confusion. It is a writing mistake by accident. We also use 3k prompts in Table 2.

Reviewer comment (Experiments):
Instead of using Stage 2 and Stage 3, which use 30K+60K and 50K training steps, respectively, how about the results of Stage 1 when we train the model longer than 300K steps?

Author response:
We may need to emphasize our core purpose, i.e., **improving the generation of the hands and face**, and our stages 2 and 3 are necessary for these two specific purposes. If we only preserve Stage 1, i.e., fine-tuning the model on the human-centric dataset longer, in general, we believe the model would perform better overall than fine-tuned SD v1.5 trained by 300K step, e.g., in image-text alignment and overall quality, if not overfitting. However, considering the proportion of close-ups of hand and face images in the whole human-centric dataset, around 0.7%, like a long-tail problem, it is hard for the model to balance its capability,  and thereby the model could not learn sufficient knowledge about hand and face. So it could also not generate natural hand and face. Intuitively, using two extra customized experts to learn specific knowledge seems more reasonable. The user study in A6 also supports our speculation as the hand quality and face quality are still inferior to that of our MoLE.

Moreover, recall that we train the text encoder and UNet together in Stage 1, this would consume a large amount of computation resources. In contrast, in Stage 2 we freeze the base model and only train the low-rank experts. In Stage 3 we freeze the base model and the low-rank experts and only train local and global gating layers. The whole process is more resource-friendly as less parameters are learnable.

Reviewer comment (Evaluation):
The authors should also report CLIP similarity, FID, and aesthetic score to evaluate the overall quality of generated images in addition to human preferences.

Author response:
We thank the reviewer's advice. We follow [4] and [6] using COCO Human Prompts and Human-centric datasets to evaluate the overall quality of generated images on metrics including the CLIP similarity, and FID as reported below. We also use an aesthetic predictor (https://laion.ai/blog/laion-aesthetics/) to generate the aesthetic score.

| Model | CLIP sim | FID | Aesthetic Score |
| -------- | -------- | -------- | -------- |
| SD v1.5     | 26.87     | 69.82     | 5.19     |
| MoLE     | 27.33     | 64.37    | 5.18     |

We find MoLE is also superior in CLIP similarity and FID. We also find that MoLE is slightly inferior to SD v1.5 in aesthetic score. We deem that it is reasonable as SD v1.5 is especially fine-tuned on **laion-aesthetics v2 5+ dataset** in which **each image's aesthetic score is evaluated with high aesthetics score by exactly the aesthetic predictor we used in this comparison**.

Reviewer comment (Evaluation):
In Table 1, how are the standard deviations measured?

Author response:
As we describe in Section 4.2,  we randomly sample 3k prompts from two human benchmarks to generate images and calculate metrics, HPS and IR, using their released official evaluation model. Afterward, we average the produced HPS and IR scores for the 3k prompts.  We repeat the whole process three times and use the averaged HPS and IR to calculate the mean and standard deviations.

Reviewer comment (Experiments):
Can the proposed approach also improve the performance of SD-XL?

Author response:
Yes is the answer. As shown in Figure 8 and Figure 9, we also see that SD-XL fails to generate correct, natural, and realistic face and hand (especially hand) while our method can be employed on any linear layer. Therefore, it is also feasible to apply MoLE on SD-XL. We are trying to actively reach out to members of stability.ai, hoping them to adopt our method.

Reviewer comment (Experiments):
The reviewer wonders why the authors do not present generated images on user-provided text prompts that rarely appear in the training and benchmark prompts.

Author response:
We may need to explain that the benchmark, i.e., DiffusionDB, is exactly collected from real users. On its website (https://github.com/poloclub/diffusiondb) it is reported that **DiffusionDB is the first large-scale text-to-image prompt dataset. It contains 14 million images generated by Stable Diffusion using prompts specified by real users.** That is also one of the reasons we select DiffusionDB as our benchmark, aiming to evaluate how our method performs under users' prompts in reality. Hence, we actually indeed present generated images on user-provided text prompts, e.g., "portrait front view of a man in a red suit sitting at a desk, with red sunglasses", and "artistic photo of a young beautiful woman with red hair, smiling slightly" in Figure 9, "muscular man wearing a leather harness and hat" and "a smile blond girl lay on chair holding rose" in Figure 10, etc.

Reviewer comment (Evaluation):
In Figure 7(d), why do the face and hand experts also highlight non-human-centric regions?

Author response:
It is easy to explain. When the target, i.e., face or hand does not appear in the image (e.g., in the second row of Figure 7(d) left, the face is absent), the corresponding expert may be disturbed by noise feature in the generation process.
Luckily, due to the existence of global assignment, this interference can be alleviated because global assignment could produce a relatively small scalar as shown in Figure 7(a) and (b) to gradually reduce such interference.