Paper: GIST: Generating Image-Specific Text for Fine-grained Object Representations (ICLR-2024)

Reviewer comment (Experiments):
The paper lacks of comparison with other fine-tuning techniques such as prompt tuning approaches. Please analyze and compare with other families of VL finetuning.

Author response:
We did provide experimental results for VL finetuning using image captioning models and also compare against two CVPR 2023 VL finetuning papers. Did the reviewer have a specific method in mind?

Reviewer comment (Experiments):
If I understood correctly, the proposed method requires full images for the target domain. In a real-world setting, it may be hard to obtain such images, especially for medical images.

Author response:
Could the reviewer please expand on what is meant by “full images”? GIST assumes access to an image dataset with class labels but has no restrictions on the number of images per class. The full pipeline can be applied to domains that have very few images.

Reviewer comment (Theory):
Why is a summary is required? We can see the performance improvement by doing this, but I found no proper explanations for why.

Author response:
Text summarization is not required, but does improve results (Section 4.5 of submission). Our method using the longer, original text outperforms the baselines. However, we achieve even better results when we summarize the captions. The intuition behind this is that the summarized captions remove extraneous words and are more similar to the format of the original CLIP training data. We add this discussion in the revision and thank the reviewer for the suggestion.

Reviewer comment (Experiments):
Instead of the process of generating captions using LLM and matching with CLIP, how if the trained vision-llm models (e.g., BLIP-2, Flamingo) are used to generate captions?

Author response:
This is an excellent question. We provide experimental results for using BLIP-2 and GIT to generate captions in Section 4.5: Captioning Model Comparison. The full- shot average accuracy and standard deviation is 74.17 (2.72) for GIT, 73.83 (2.75) for BLIP-2, and 75.77 (2.67) for GIST (ours). The 5-shot average accuracy and standard deviation is 42.96 (2.99) for GIT, 42.66 (3.03) for BLIP-2, and 46.96 (3.13) for GIST (ours). We show additional comparison results in the Appendix in Figure 7, Table 5, and Sections A.5 and A.6. Our method outperforms all of these other approaches. Would the reviewer suggest that we move some of the results from the Appendix into the main paper? If so, is there material that the reviewer thinks we could move from the main body into the Appendix?

Reviewer comment (Evaluation):
The generated explanations are exclusively matched to images? Or the same captions can be matched to multiple images? If so, please statistics how frequently each explanation is matched to the images.

Author response:
The same caption can be matched to multiple images since the same fine-grained features can appear in many images. The average number of images matched per captions varies per dataset. For the full Fitzpatrick40 dataset, for example, each caption is matched to an average of 2.8 images. For all five-shot datasets, each matched caption is matched to an average of 1.3 images. Could the reviewer please additionally clarify which other statistics they would like?

Reviewer comment (Novelty):
The contribution is limited since the core idea is the same as typical data labeling workflow that uses LLM. More specifically, during prompting LLM, the user-provided prompt is not coming for free, it is also a kind of human knowledge or preference prior. There is no discussion on this difficulty of prompt preparation as compared to typical easy prompt such as “a photo of class name”. Thus, the workflow of generating image-text pairs has no difference from data labeling workflow that uses LLM.

Author response:
We provide our text prompts in Appendix section A.3 and show that they do not require in-depth domain knowledge. We should have included some of this material in the main text. We do not require prompt tuning and only use one prompt template per domain. The user-provided prompt does not require a significant human load and the prompt can be used across many classes and datasets within the domain. We show in Figure 4 and Figure 10 that our prompting method results in more detailed and relevant text descriptions than the prompting methods used in LaBo and CaFo.

Reviewer comment (Experiments):
In the experiments, Table 1, the proposed workflow which uses GPT-3, CLIP and intricate prompts, does not get significant improvement over FLYP which uses CLIP and a typical easy prompt, though the comparison setting is unfair.

Author response:
Our improvements over FLYP are statistically significant, using a paired t-test. They are also meaningful. For the majority of full-shot and five-shot evaluation settings in Table 1, the absolute accuracy improvement of GIST over FLYP is greater than the absolute improvement of FLYP over off-the-shelf CLIP linear probing (CLIP LP).

Reviewer comment (Experiments):
This paper lacks the ablations about matching texts and images. It's unclear whether the matching based on a pre-trained CLIP will impact the downstream tasks. In addition, I'm concerned about how many captions are matched to one image and whether more captions will help.

Author response:
We include an ablation on the number of matched captions in Table 3 of the main paper. It suggests that matching more captions per image does not necessarily improve performance. The optimal number of captions per image varies per dataset.

Reviewer comment (Experiments):
This paper lacks studies on the impact of fine-tuning CLIP and whether more data (images) will further improve.

Author response:
We compare against CLIP LP and CLIP zero-shot and show that fine-tuning CLIP performs significantly better. We also show ablations for fine tuning CLIP on GIST captions versus fine tuning CLIP on captions from GIT and BLIP-2 or on standard text prompts (e.g.,the  FLYP method). We show that fine tuning on our GIST captions outperforms fine tuning CLIP on all of these alternative text approaches. We demonstrate that fine-tuning with the full dataset of images in comparison to few-shot numbers of images (Table 1,6) improves classification performance.

Reviewer comment (Experiments):
What are text inputs for the proposed GIST, the standard text prompts with labels or the text descriptions?

Author response:
The text inputs to GIST are text prompts using class names. We show all of our text prompts in Appendix section A.3. We add material to the main text about prompting to clarify this.

Reviewer comment (Experiments):
I'm concerned about the performance of fine-tuning CLIP with both short text descriptions and the original texts, and how about fine-tuning the CLIP with longer texts while fine-tuning classifiers with shorter descriptions.

Author response:
In Section 4.5, we provide experimental results on fine tuning CLIP on short text descriptions versus fine tuning CLIP on the original descriptions. Both methods outperform the baselines, but fine tuning CLIP on short text descriptions does best. The text descriptions are used to finetune CLIP to improve the embedding space. No text descriptions are used to finetune the classifier.