Paper: generative adversarial network with hierarchical semantic prompt constrainting clip for high-quality text-to-image synthesis (ICLR-2024)

Reviewer comment (Writing):
This paper is definitely finished in a rush. The whole manuscript is full of various grammar mistakes, spelling and punctuation errors, and unnatural expressions. For example, the first sentence of the abstract is weird. It should be “How to efficiently synthesize controllable, semantically-aligned and high-quality images based on text prompts is currently a significantly challenging task”. Moreover, there are a lot of spelling and punctuation errors in the sections of the introduction and experiment, making the reviewer so hard to understand.

Author response:
Firstly, we acknowledge and will work to improve the expression issues caused by the haste in writing.

Reviewer comment (Experiments):
The authors hardly provide qualitative comparisons between their method and baselines. Only a few examples in Fig 1,6&7 can demonstrate the superiority of their proposed method. Actually, the experiment section should present enough synthesized images.

Author response:
Regarding qualitative comparisons, although there are fewer images, each experiment includes qualitative comparisons. It is considered unreasonable by the reviewer to claim there are almost none; nonetheless, we will add more synthetic images to address this concern.

Reviewer comment (Presentation):
Why does the result of GALIP appear twice in Table 1?

Author response:
As for the third issue, we find it somewhat confusing. Firstly, we are unsure if the reviewer has thoroughly read the entire paper, because the question “Why does the result of GALIP appear twice in Table 1?”  which has been explained in the paper. One is the literature result, and the other is the baseline we actually reproduced, and the results are different.

Reviewer comment (Novelty):
The paper dives deep into the technical implementation of the proposed method but lacks high-level explanations and discussions of contributions. The reviewer asks: What are the key issues with prior GAN-based text-to-image methods, and why is the proposed method the most efficient way to address them? What could be the alternatives for the proposed extensions? What are the limitations of using all the pre-trained modules (e.g., Text Concept Mining or Scene Graph Generator)? These questions are not answered, making it unclear what high-level lessons the reader can extract.

Author response:
1.The key issues in previous GAN-based text-to-image methods are twofold. Firstly, they are constrained by modality gaps and dataset limitations, making it challenging for GANs to find a good balance point in the latent space for effective feature mapping between text and images. This can be addressed through the use of pre-trained CLIP. Secondly, there is a challenge in semantic constraints on synthesized images – ensuring the consistency between image and text content, which is the focus of our approach. We control this using hierarchical semantic prompts. The choice of hierarchical prompts as a solution stems from their effective use in controlling prompts in text generation, and we aim for a similar effect in image generation.

Reviewer comment (Writing):
The paper contains many typos and uses non-standard terminology. Examples of typos include: "pre-trainde", "demonstrat", "not not excellent", "adptopted", "As Fig 5. shown", "matching-mware Loss", "CLIP ADPTOR", "experiments are conduct". The reviewer believes the paper needs careful revision regarding writing quality.

Author response:
Firstly, thank you very much for your feedback. Regarding the expression issues, due to the short writing time during the submission of this article, there were several typos, and some discussions became unclear after being cut due to length constraints. If the paper is accepted, we will carefully revise and rectify these expression issues in the final version.

Reviewer comment (Novelty):
In the related work section, the "GAN-based image synthesis" part discusses GAN inversion, which is unrelated since the paper is not about GAN inversion. At the same time, there is no discussion of other text-to-image GANs and diffusion models (DMs).

Author response:
2.Regarding the GAN inversion part, this was an oversight on our part. In the initial version, we discussed the correlation between constructing hierarchical semantic prompts and GAN inversion in the modal latent space, and how it influences and interacts with the generator. We failed to adjust this section after its removal in the submitted version, and we will rectify that.

Reviewer comment (Novelty):
It is not clear what kind of technical novelty the paper presents. The method section starts from an existing baseline GALIP. It then combines modules that existed in some forms in other fields or makes slight modifications to the discriminator loss, without analyzing the challenges that arise during these modifications. It remains unclear what the technical novelty is, what technical challenge was addressed, and what the technical lesson for readers is.

Author response:
3.Concerning the novelty of our work, we indicate in the paper that the innovation lies in:

（1）How to construct appropriate semantic prompts from text to guide the synthesis of target and attribute information in images. The current construction of prompts in this field is rough, if not overlooked.
（2）How to use different hierarchical semantic prompts to constrain the adjustment of target synthesis in the generator. This involves determining which prompts to use, which to ignore, and how to adjust the generator's output.
（3）The introduction of hard-negative sample loss in the GAN network is a key aspect. 
Regarding the viewpoint that challenges during the analysis of loss modifications were not addressed, we disagree. Whether it's the construction of hierarchical semantic prompts, feature adjustments based on prompts, or the introduction of hard-negative samples in the discriminator, we have analyzed the purpose of these approaches, the problems they aim to solve, the difficulties in their resolution, and the advantages of adopting such methods.
Certainly, if you feel that our explanation is not clear enough, we can also supplement more content in the formal version's appendix to provide a more detailed explanation of our methods and analysis. Thank you for your feedback; we will strive to ensure that readers have a comprehensive understanding of our work.

Reviewer comment (Theory):
The paper contains an unsupported claim: "This loss can be generalized to apply most generation methods involving semantic matching. We will do more work in the future to prove this viewpoint." The reviewer notes that this is a strong claim without experimental confirmation. The loss form is tied to the GAN-form objective, and it is unclear if it would generalize to any other objectives.

Author response:
4.Regarding the “unsupported claim”，we think this statement is not a strong claim but rather an outlook and consideration for future work.  the new discriminative loss applied in the hinge loss form, we drew inspiration from cross-modal retrieval work and redesigned a new discriminative loss for image matching between synthetic images, real images, and mismatched images in the GAN network. The experiments in the paper demonstrate the effectiveness of the final loss structure. Since the use of hinge loss in image matching is not exclusive to GANs but also includes VAE and some diffusion models in image synthesis, we point out the potential for its generalization to other synthetic methods with a similar structure.

Reviewer comment (Presentation):
The visual results presented are limited, with only 4 images in the main paper and 2 in the supplementary, which is insufficient given the nature of this research area.

Author response:
5.Regarding image issues, we will supplement more image examples in the final version.

Reviewer comment (Evaluation):
The paper introduces several new modules on top of the GALIP model. While the new model achieves somewhat better FID and CLIP scores, it operates noticeably slower (e.g., 0.025s vs 0.017s in Fig. 1). The efficiency aspect and possible trade-offs are not discussed in the paper.

Author response:
6.Regarding the efficiency trade-off, this issue is commonly present in large-scale image synthesis tasks, therefore we aim to pursue higher synthesis quality within the same efficiency scale (0.001s, 0.01s, 0.1s).

Reviewer comment (Experiments):
Compared with the GALIP paper, there is a lack of comparison results of Model Param and Data Size.

Author response:
My model's parameters are 0.3+0.08B, while GALIP's parameters are 0.24+0.08B. The additional parameters are used for generating structural semantic prompts and hierarchical adjustment in the generator. The parameter count is much smaller than that of diffusion generative models. 
We have provided details about the data size in the training section. We adopted GALIP's training settings (includeing the same data size)and trained only on the standard MSCOCO dataset (80000 images).and CUB_BIRD(11,788 Images)

Reviewer comment (Experiments):
Compared with the GigaGAN paper, there is a lack of comparison results of high-resolution generation.

Author response:
Firstly, the primary focus of this paper is to investigate the semantic consistency between text content and synthesized image content; high-resolution image synthesis is not the focal point of this study. We utilized the basic framework of GALIP, which itself is not suitable for synthesizing ultra-high-resolution images. This is in contrast to the foundational model of GIGAGAN, which adopts the basic framework of StyleGAN and incorporates a design for a super-resolution module to synthesize higher-resolution images post the standard model. Therefore, we did not provide a comparison of super-resolved images in the paper because we believe it is not directly relevant to our work. If you deem it necessary and are willing to modify the rating, we would be willing to include comparisons of higher-resolution images in the appendix after the paper is accepted.