Paper: You Only Submit One Image to Find the Most Suitable Generative Model (ICLR-2024)

Reviewer comment (Presentation):
Figure 1 shows that users may use more effort in identifying their target models. However, this figure contains too much information and details. Therefore, a detailed textual explanation should be provided for readers to easily understand this illustration.

Author response:
**[W1] A detailed textual explanation for Figure 1**

Thank you for your valuable suggestion. We have added the following explanation to our revised paper in Appendix B. 

In traditional searches for generative models, the models are initially filtered by the model hubs using tags and then sorted based on download volume. However, this approach presents challenges for users in identifying their desired models, as the model with the highest download volume may not necessarily align with their specific target. Consequently, users select models based on their own judgment, and then download and evaluate each model individually. This consumes a significant amount of network resources and human effort, and there is also the possibility of filtering out the potentially best model due to judgment errors.

In our GMI setting, each model is directly sorted based on its matching score. This means that the most suitable model is recommended at the top, enabling users to effortlessly find their desired models with minimum effort.

Reviewer comment (Experiments):
The authors constructed the model hub and tasks with 16 stable diffusion models. A brief description should be provided for these models.

Author response:
**[W2] A brief description of models**

Thank you for your valuable advice. We will add a brief description of each model in our appendix.

Reviewer comment (Presentation):
Minor problems: (1) The X ticks of the left two sub-figures of Figure 2 are not aligned with the bars; (2) “shows” in the second sentence of section A.2 should be “show”.

Author response:
**[W3] Minor problems**

Thank you for your valuable advice. We have revised these issues in this paper.

Reviewer comment (Presentation):
The paper is more like a technical report instead of an academic paper.

Author response:
**[W1 & W2.1 & Q1] Technical contributions**

Our paper makes technical contributions in three specific aspects:

First, we propose a novel and realistic GMI problem. Generative models have garnered considerable attention, accompanied by the emergence of various generative model platforms. However, users face significant inefficiencies when seeking generative models that satisfy their specific requirements within these platforms, as this necessitates a meticulous browsing and testing process. In this paper, we introduce the Generative Model Identification (GMI) problem, encouraging researchers to develop algorithms describing the capabilities of generative models that enable precise alignment between model functionalities and user requirements, thereby achieving expedited searches in generative model exploration. We posit that such endeavors will substantially propel the advancement of generative models and model platforms.

Second, we take a successful step to address the GMI problem by proposing an effective specification for the generative model. It is noteworthy that the proposal is definitely not a simple combination of existing techniques. For example, we analyze the existing RKME method in Figure 2 and show that the existing method not only can not deal with the mapping between prompts and images which is essential for distinguishing generative models, but also can not address the dimensionality and cross-modality problem which hinders the success of GMI. The results in Table 2 clearly prove that any simple combination of existing methods cannot achieve good performance in GMI, and our proposal provides the best performance evaluated by all metrics. Therefore, it is not fair to consider our proposal as a simple combination of existing techniques.

Third, we construct and release a novel benchmark dataset for the GMI problem, which contains general stable diffusion models. Based on this benchmark dataset, we conduct comprehensive experiments to evaluate the existing baseline methods and proposed methods, and the results demonstrate the effectiveness of our proposed method.  It is expected the dataset could facilitate the development of generative models.

We appreciate your valuable suggestions and will highlight our contributions in our revised paper.

Reviewer comment (Novelty):
The technical contributions are limited. It is quite trivial to calculate the distance between the uploaded image/prompt and the existing images/prompts. The MMD distance is a very naive distance metric, and its robustness is questionable.

Author response:
**[W1 & W2.1 & Q1] Technical contributions**

Our paper makes technical contributions in three specific aspects:

First, we propose a novel and realistic GMI problem. Generative models have garnered considerable attention, accompanied by the emergence of various generative model platforms. However, users face significant inefficiencies when seeking generative models that satisfy their specific requirements within these platforms, as this necessitates a meticulous browsing and testing process. In this paper, we introduce the Generative Model Identification (GMI) problem, encouraging researchers to develop algorithms describing the capabilities of generative models that enable precise alignment between model functionalities and user requirements, thereby achieving expedited searches in generative model exploration. We posit that such endeavors will substantially propel the advancement of generative models and model platforms.

Second, we take a successful step to address the GMI problem by proposing an effective specification for the generative model. It is noteworthy that the proposal is definitely not a simple combination of existing techniques. For example, we analyze the existing RKME method in Figure 2 and show that the existing method not only can not deal with the mapping between prompts and images which is essential for distinguishing generative models, but also can not address the dimensionality and cross-modality problem which hinders the success of GMI. The results in Table 2 clearly prove that any simple combination of existing methods cannot achieve good performance in GMI, and our proposal provides the best performance evaluated by all metrics. Therefore, it is not fair to consider our proposal as a simple combination of existing techniques.

Third, we construct and release a novel benchmark dataset for the GMI problem, which contains general stable diffusion models. Based on this benchmark dataset, we conduct comprehensive experiments to evaluate the existing baseline methods and proposed methods, and the results demonstrate the effectiveness of our proposed method.  It is expected the dataset could facilitate the development of generative models.

We appreciate your valuable suggestions and will highlight our contributions in our revised paper. 

**[W2.1] The MMD distance is a very naive distance metric**

MMD distance is a simple yet effective distance metric actually. For model identification tasks, [1] successfully applies the RKME framework to various image recognition tasks, and [2] applies the RKME framework to both tabular classification and image classification tasks. Additionally, MMD distance is widely used in Domain Adaptation [3] and Domain Generalization [4]. Therefore, the MMD distance is not such a naive and questionable distance metric, and our proposed methods are non-trivial based on it.

Moreover, we should clarify that our proposal goes beyond a simple application of RKME and MMD distances. We analyze the existing RKME method in Figure 2 and show that the existing method not only can not deal with the mapping between prompts and images which is essential for distinguishing generative models, but also can not address the dimensionality and cross-modality problem which is a hinder of GMI. The results in Table 2 clearly prove that any simple combination of existing methods cannot achieve good performance in GMI, and our proposal provides the best performance evaluated by all metrics. Therefore,  our proposal is clearly not  a simple combination of existing techniques.

[1] Lan-Zhe Guo, Zhi Zhou, Yu-Feng Li, Zhi-Hua Zhou: Identifying Useful Learnwares for Heterogeneous Label Spaces. ICML 2023: 12122-12131

[2] Yu-Jie Zhang, Yu-Hu Yan, Peng Zhao, Zhi-Hua Zhou: Towards Enabling Learnware to Handle Unseen Jobs. AAAI 2021: 10964-10972

[3] Wei Wang, Haojie Li, Zhengming Ding, Feiping Nie, Junyang Chen, Xiao Dong, Zhihui Wang: Rethinking Maximum Mean Discrepancy for Visual Domain Adaptation. IEEE Trans. Neural Networks Learn. Syst. 34(1): 264-277 (2023)

[4] Peifeng Tong, Wu Su, He Li, Jialin Ding, Zhan Haoxiang, Song Xi Chen: Distribution Free Domain Generalization. ICML 2023: 34369-34378

Reviewer comment (Writing):
Highlight the technical contributions of this paper.

Author response:
**[W1 & W2.1 & Q1] Technical contributions**

Our paper makes technical contributions in three specific aspects:

First, we propose a novel and realistic GMI problem. Generative models have garnered considerable attention, accompanied by the emergence of various generative model platforms. However, users face significant inefficiencies when seeking generative models that satisfy their specific requirements within these platforms, as this necessitates a meticulous browsing and testing process. In this paper, we introduce the Generative Model Identification (GMI) problem, encouraging researchers to develop algorithms describing the capabilities of generative models that enable precise alignment between model functionalities and user requirements, thereby achieving expedited searches in generative model exploration. We posit that such endeavors will substantially propel the advancement of generative models and model platforms.

Second, we take a successful step to address the GMI problem by proposing an effective specification for the generative model. It is noteworthy that the proposal is definitely not a simple combination of existing techniques. For example, we analyze the existing RKME method in Figure 2 and show that the existing method not only can not deal with the mapping between prompts and images which is essential for distinguishing generative models, but also can not address the dimensionality and cross-modality problem which hinders the success of GMI. The results in Table 2 clearly prove that any simple combination of existing methods cannot achieve good performance in GMI, and our proposal provides the best performance evaluated by all metrics. Therefore, it is not fair to consider our proposal as a simple combination of existing techniques.

Third, we construct and release a novel benchmark dataset for the GMI problem, which contains general stable diffusion models. Based on this benchmark dataset, we conduct comprehensive experiments to evaluate the existing baseline methods and proposed methods, and the results demonstrate the effectiveness of our proposed method.  It is expected the dataset could facilitate the development of generative models.

We appreciate your valuable suggestions and will highlight our contributions in our revised paper.

Reviewer comment (Evaluation):
Discuss the limitation of the proposed method.

Author response:
**[Q2] Discuss the limitations of the proposed method.**

We have discussed the limitation of efficiency in "3.3 Discussion" in our paper. Thank you for your valuable suggestion. We have additionally added a "Limitations" section in Appendix. C to discuss the limitations of the proposed method. 

First, the time complexity of our work has the potential for improvement. At present, our time complexity increases linearly with the number of models in the model platform, which will not become a heavy burden when the number of models is large. Previous research[1] has demonstrated that acceleration techniques can enhance the identification process, such as the use of a vector database. This could be an area for future exploration.

Second, we currently only consider the cases that identify generative models using one uploaded image to describe users' requirements. The assumption is reasonable since users' ideas often rely on existing image templates when they want to generate images, and it is not difficult to find only one image that has a similar style to fulfill the user's requirements. Despite this, it is also interesting to study how to quickly and accurately identify models via other information such as textual prompts. We will study this problem in future work.

[1] Lan-Zhe Guo, Zhi Zhou, Yu-Feng Li, Zhi-Hua Zhou: Identifying Useful Learnwares for Heterogeneous Label Spaces. ICML 2023: 12122-12131

Reviewer comment (Experiments):
The paper assumes that users will always provide an example image, which might not be universally applicable or intuitive.

Author response:
**[W1] Assumption on providing an example image**

First of all, we need to point out that in the case that users cannot provide additional information, they can only browse and search one by one through the tags and illustration examples of the models in the model hub, which results in users not being able to quickly and accurately search for the models that satisfy their requirements. To realize a fast and accurate search, it is necessary to assume that users should provide additional information to describe their requirements. In this paper, we consider the cases that identify generative models with only one uploaded image to describe users' requirements. The assumption is reasonable since users' ideas often rely on existing image templates when they want to generate images, and it is not difficult to find only one image that has a similar style to the user's requirements. Moreover, it is also interesting to study how to quickly and accurately identify models via other information such as textual prompts, we will study this problem in future work.

Reviewer comment (Novelty):
The work heavily relies on existing methods: RKME, a pre-trained vision-language model (possibly CLIP?), and the image interrogator from a GitHub repository. There's a lack of novelty in the proposed method.

Author response:
**[W2] Lack of novelty in the proposed method**

We thank you for your review. It looks like you have some misconceptions about the contribution of our paper. 

First, we propose a novel and realistic GMI problem. Generative models have garnered considerable attention, accompanied by the emergence of various generative model platforms. However, users face significant inefficiencies when seeking generative models that satisfy their specific requirements within these platforms, as this necessitates a meticulous browsing and testing process. In this paper, we introduce the Generative Model Identification (GMI) problem, encouraging researchers to develop algorithms describing the capabilities of generative models that enable precise alignment between model functionalities and user requirements, thereby achieving expedited searches in generative model exploration. We posit that such endeavors will substantially propel the advancement of generative models and model platforms.

Second, we take a successful step to address the GMI problem by proposing an effective specification for the generative model. It is noteworthy that the proposal is definitely not a simple combination of existing techniques. For example, we analyze the existing RKME method in Figure 2 and show that the existing method not only can not deal with the mapping between prompts and images which is essential for distinguishing generative models, but also can not address the dimensionality and cross-modality problem which hinders the success of GMI. The results in Table 2 clearly prove that any simple combination of existing methods cannot achieve good performance in GMI, and our proposal provides the best performance evaluated by all metrics. Therefore, it is not fair to consider our proposal as a simple combination of existing techniques.

Third, we construct and release a novel benchmark dataset for the GMI problem, which contains general stable diffusion models. Based on this benchmark dataset, we conduct comprehensive experiments to evaluate the existing baseline methods and proposed methods, and the results demonstrate the effectiveness of our proposed method.  It is expected the dataset could facilitate the development of generative models.

We will highlight our contributions in our revised paper.

Reviewer comment (Writing):
Perhaps it's just me, but I found the writing in the technical sections really confusing.

Author response:
**[W3] Writing problem**

Thank you for your advice. We will improve the writing for technical sections.

Reviewer comment (Writing):
While the image interrogator is referenced from an existing work, it would still benefit from a brief description within this paper, given that not all readers may be familiar with it.

Author response:
**[W4] A brief description of image interrogator**

Thank you for your advice. We have incorporated the following description into our revised paper in Appendix A.1.

The Image Interrogator tool combines CLIP and BLIP to optimize text prompts for matching a given image. Firstly, it generates candidate captions using the BLIP model. Next, it employs a search process to identify the caption list that maximizes the similarity between the captions and images, as evaluated by the CLIP model.

Reviewer comment (Experiments):
How was the default prompt 'P' determined?

Author response:
**[Q1] Default prompt 'P'**

This paper focuses on improving the identification of the most suitable model when prompts are already provided. Consequently, we utilize a default set of prompts for all generative models during the construction of specifications. This default set of prompts can be found in the "PromptPool.json" file provided in our supplementary material.

Reviewer comment (Evaluation):
The paper suggests that developer/user providing the exact prompt would yield better results. Can this claim be supported with any qualitative or quantitative evaluation? Also, are there any guidelines or recommendations for the type of prompt to be used?

Author response:
**[Q2] Developer / user provided  prompt**

Thank you for your constructive suggestions. 

Users should provide prompts that accurately describe the images they upload. In our proposed method, we generate prompts by employing an image interrogator that demonstrated effective performance in our experiments. When ground-truth prompts are provided, the identification performance will not be worse, and this is supported by our response to Reviewer BSis [Q3].

Developers should provide prompts to guide the model in generating images it excels at. A model-specific prompt set will enhance identification by amplifying the contrast between generated specifications, resulting in improved accuracy. We conducted synthetic experiments to validate this claim. In this experiment, developers will provide 5 prompts for each model, assuming they excel at them, and generate corresponding specifications. The user will query images generated using similar prompts. The results demonstrate that providing prompts for specification generation leads to improved accuracy. However, in actual situations, it is difficult for the model hub to force developers to provide prompts. Therefore, this paper conducts experiments under a default prompt set.

|                             | Top-1 Acc | Top-2 Acc | Top-3 Acc | Top-4 Acc |
| --------------------------- | --------- | --------- | --------- | --------- |
| Default Prompt Set          | 0.455     | 0.614     | 0.734     | 0.812     |
| Developer  Provided  Prompt | 0.777     | 0.937     | 0.985     | 0.987     |

We greatly appreciate the valuable suggestion provided by the reviewer, and we have incorporated this discussion and experiment into our revised paper in Appendix A.5.

Reviewer comment (Experiments):
As I understand, RKME scoring would work better if you have image distributions to match; that is, if you have more than one user provided query image. In that sense, it is not clear to me how the paper could claim that a single image is sufficient to identify the model? How do you substantiate this claim? One thought is: to derive a more general scoring function (in Eq. 2, for example) with multiple user query images and the authors did an ablation that shows that using a single image or when using multiple images, the retrieval performance is approximately similar. Further to the above comment, given there is only a single image query, what is the performance if you used a different statistic for the scoring? For example, using \ell_1 norm or just the \argmin on the kernel features?

Author response:
**[W1.1] Multiple images for querying**

RKME performs more effectively when the provided image set accurately represents the distribution. Thus, our framework, built upon RKME, naturally accommodates situations in which users provide multiple images as queries. We conducted experiments using user queries consisting of 1, 2, 3, 4, or 5 images. The results are shown in the following table:

|                | Top-1 Acc | Top-2 Acc | Top-3 Acc | Top-4 Acc |
| -------------- | --------- | --------- | --------- | --------- |
| 1 image (Ours) | 0.455     | 0.614     | 0.734     | 0.812     |
| 2 images       | 0.578     | 0.743     | 0.840     | 0.908     |
| 3 images       | 0.658     | 0.791     | 0.883     | 0.95      |
| 4 images       | 0.709     | 0.854     | 0.929     | 0.969     |
| 5 images       | 0.755     | 0.873     | 0.946     | 0.979     |

The results demonstrate that the accuracy of identifying the most appropriate model improves as the number of uploaded images increases.

Our claim regarding a single image is grounded in two factors: (1) In practical applications, it is often more convenient to upload a single image rather than multiple images to describe the user's requirements; (2) Using one image can achieve a satisfactory Top-4 Accuracy for identifying the most appropriate model.

In conclusion, our framework has the capability to handle multiple images as a query. However, uploading a single image offers a compromise between performance and convenience. We appreciate the reviewer's valuable suggestion and have added this experiment and discussion to our revised paper in Appendix A.2.

**[W1.2] Different statistics for scoring**

We thank you for your constructive advice. We conduct experiments using some statistical scoring functions, e.g., L1, and Min functions. The results are shown in the following table:

|      | Top-1 Acc | Top-2 Acc | Top-3 Acc | Top-4 Acc |
| ---- | --------- | --------- | --------- | --------- |
| Min  | 0.064     | 0.116     | 0.163     | 0.233     |
| L1   | 0.346     | 0.540     | 0.657     | 0.735     |
| Ours | 0.455     | 0.614     | 0.734     | 0.812     |

The results show that our proposed method can outperform these statistical scoring functions.

Reviewer comment (Reproducibility):
As the selection of the reduced image sets is important for efficiency of the method, the paper must provide details of how this is done for generative models. How do you ensure the size is large enough and also discriminative for each model so that overlap between reduced sets across models is minimal while the model representation is sufficient for identification?

Author response:
**[W2 & W3.1] The selection of the reduced image sets**

We thank you for your professional advice. Selecting suitable prompts is important for building specifications for generative models. 

This paper focuses on improving the identification of the most suitable model when prompts are already provided. Consequently, we utilize a default set of prompts for all generative models during the construction of specifications. This default set of prompts can be found in the "PromptPool.json" file provided in our supplementary material. And, our proposal can achieve satisfactory performance when using the default prompt set for all models. We have also conducted experiments on the following questions, e.g., [W3.2], [Q1], [Q4], to answer your questions.

We appreciate the reviewer pointing out these key points for GMI setting. The selection of prompts for generating specifications could be a future direction for the GMI setting.

Reviewer comment (Experiments):
One of the key innovations in this paper is the inclusion of prompt embeddings in the model specification (Eq. 9). As is clear from Eq. 9, the weighting scheme soft-selects images in the reduced set using the query prompt and compares the kernel embeddings of the respective images with the query image. A natural question that would arise here is: how do you ensure the reduced set prompts are selected so as to cover user provided images or prompts? This part seems less detailed and not evaluated in the experiments. Specifically, how do you produce the (prompt, generated image) pairs for the reduced set? How do you ensure the prompts are different for diverse generated images? How do you account for the domain gaps (if any) between user specified prompts against the submission-time specified prompts? What is the sensitivity of the scoring function against differences in the prompts? It would be useful to see qualitative results showing images and (varied) prompts.

Author response:
**[W2 & W3.1] The selection of the reduced image sets**

We thank you for your professional advice. Selecting suitable prompts is important for building specifications for generative models. 

This paper focuses on improving the identification of the most suitable model when prompts are already provided. Consequently, we utilize a default set of prompts for all generative models during the construction of specifications. This default set of prompts can be found in the "PromptPool.json" file provided in our supplementary material. And, our proposal can achieve satisfactory performance when using the default prompt set for all models. We have also conducted experiments on the following questions, e.g., [W3.2], [Q1], [Q4], to answer your questions.

We appreciate the reviewer pointing out these key points for GMI setting. The selection of prompts for generating specifications could be a future direction for the GMI setting.

**[W3.2] Qualitative results for selection of the reduced image sets**

We conducted experiments with three different prompt sets. Split 1 is our default prompt set used in our experiments. Split 2 and Split 3 are two non-overlapping subsets, each half the size of Split 1. The results are shown in the following table:

|                | Top-1 Acc | Top-2 Acc | Top-3 Acc | Top-4 Acc |
| -------------- | --------- | --------- | --------- | --------- |
| Split 1 (Ours) | 0.455     | 0.614     | 0.734     | 0.812     |
| Split 2        | 0.442     | 0.609     | 0.729     | 0.810     |
| Split 3        | 0.460     | 0.617     | 0.732     | 0.808     |

The results demonstrate the robustness of the identifying performance to varying default prompt sets used in generating specifications. We appreciate the reviewer's valuable suggestion and have added this experiment to our revised paper in Appendix A.3.

Reviewer comment (Experiments):
How does the performance of the method vary when using varied reduced set sizes? It is also important to analyze the scalability of the method when the number of generative models goes higher.

Author response:
**[Q1] Performance with varied reduced set sizes**

We conduct an extreme experiment with a reduced set size = 1. Compared to our proposed method, which set the reduced set size to 550, the performance is relatively stable and still outperforms comparison methods. Therefore, the scalability of the method is guaranteed when the number of generative models increases.

|                    | Top-1 Acc | Top-2 Acc | Top-3 Acc | Top-4 Acc |
| ------------------ | --------- | --------- | --------- | --------- |
| Size = 1 (Ours)    | 0.434     | 0.592     | 0.707     | 0.789     |
| Size =  550 (Ours) | 0.455     | 0.614     | 0.734     | 0.812     |

As discussed in “3.3.Discussion”, the scalability problem can be handled using other techniques. We appreciate the reviewer's valuable suggestion and have added this experiment to our revised paper in Appendix A.4.

Reviewer comment (Reproducibility):
What are the prompts associated with the qualitative results in Figure 4? Can you also include images with user specified prompts?

Author response:
**[Q2] Prompts associated with the qualitative results in Figure 4**

The ground-truth prompt for requirement is `a woman with blonde hair and blue eyes, a detailed painting, by rossdraws, fantasy art, red-purple gradient map, mercy from overwatch, close up of a blonde woman, a brightly colored, lux, sylas`. The prompt generated by image interrogator is `a girl with blonde hair and blue eyes, game icon stylized, glowwave girl portrait, epic legends game icon, app icon, 3 d icon for mobile game, icon for an ai app, darkness background, ios app icon, lux from league of legends, darkness's background, character icon, luxanna crownguard, lightning mage spell icon, darkness aura`. The images generated by each method utilize prompts obtained from the image interrogator. We have added ground-truth prompts and generated prompts for each example in Appendix A.1.

Reviewer comment (Experiments):
Why is the performance of RKME-CLIP so close to that of the proposed method? What is the size of the reduced set for the various diffusion models?

Author response:
**[Q3] Performance of RKME-CLIP**

RKME-CLIP is an ablation variant of the proposed method that addresses the dimensionality challenge by incorporating the CLIP model. Furthermore, our proposal enhances performance by taking into account the mapping between prompts and images.

Reviewer comment (Experiments):
What would be the performance of the model if: i) the query image was selected from one of the generative models, but captioned using I(.)? ii) the query image was selected from one of the models and with the respective prompts? iii) you use the prompt weight w to be an indicator function when using i). These experiments will show the upper-bound on performance of the method (when using kernel mean matching).

Author response:
**[Q4] Three variants of proposed method**

We conducted the experiments to evaluate two variants and our proposed method in the following table. Method (i) is our proposed method, which adopts the prompt generated from the image interrogator. Method (ii) adopts the ground-truth prompts of query images to calculate the weights. Method (iii) adopts an indicator function instead of soft weights. 

|            | Top-1 Acc | Top-2 Acc | Top-3 Acc | Top-4 Acc |
| ---------- | --------- | --------- | --------- | --------- |
| (i) (Ours) | 0.455     | 0.614     | 0.734     | 0.812     |
| (ii)       | 0.455     | 0.615     | 0.733     | 0.813     |
| (iii)      | 0.252     | 0.371     | 0.461     | 0.534     |

The results show that methods (i) and (ii) perform similarly, indicating that prompts generated from an image interrogator can serve as a proxy for the ground truth when identifying the model. Method (iii) performs less effectively compared to the other two methods, suggesting the importance of soft weights in our proposal.

Reviewer comment (Presentation):
What is \gamma in Figure 7?

Author response:
**[Q5] Gamma in Figure 7**

Gamma serves as a hyper-parameter in the calculation of RKME. The results demonstrate the robustness of our proposal to variations in this hyper-parameter.