Paper: Democratizing Fine-grained Visual Recognition with Large Language Models (ICLR-2024)

Reviewer comment (Novelty):
Although the novelty and technique score is satisfactory for ICLR, a major issue is whether it is necessary to only leverage LLM for FGVC under the proposed setting. In fact, as far as the reviewer concerns, some other methods only learning prompt on VLM can already achieve more than 90% accuracy for zero-shot FGVC. For example: [1] Conditional Prompt Learning for Vision-Language Models. CVPR 2022. [2] Learning to Prompt for Vision-Language Models. IJCV 2022. This issue is critical, at least from the vision community. What should be the correct role of LLM for FGVC? Should only the concepts from LLM be used for FGVC? Or should it be jointly used with visual representation to aid FGVC? Solely relying on LLM for vision tasks is in a way that the reviewer does not appreciate.

Author response:
Thank you for your positive, attentive and constructive feedbacks! We are encouraged that you, as an expert in vision and FGVC, recognized our method as "**novel and interesting**" and "experimentally better than existing state-of-the-art methods" which provides "a new way to do training-free FGVC based on LLM". We address your thoughts point by point below.

>**Q1:** As there is already some good VLM solution to do zero-shot FGVC and achives nearly 90% accuracy, is it necessary to solely address FGVC by pure concepts from LLM? Besides, is it necessary to solely rely on LLM instead of VLM or visual representation for FGVC?
>
>**A1:** Great question! Thanks for recoginizing the novelty and technique score of our work are statisfactory for ICLR. Following your suggestion, we have now also added a discussion section in **Appendix K.4** of the revision to carefully compare our setting and method with respect to VLM zero-shot transfer and prompt enrichment/tuning methods. Here, we breakdown this question into three subsequent questions and address them below:
>
>1. **Setting perspective:** In fact, as recoginized by ```Reviewer oRxX```, we would like to emphasize and clarify that the novel task we propose and study in this work is **vocabulary-free FGVC with only few unlabeled samples as natural observation**. We compare our setting and existing VLM solutions apple-to-apple in the **Table 1 and Table 2 (in Response 2/4)**. To be more specific, under vocabulary-free FGVC setting, there is no target class names (vocabulary) pre-defined by domain experts (*e.g.*, ornithologist) for VLM-based zero-shot transfer methods or VLM prompt tuning methods to construct classifiers. This makes existing VLM solutions (*e.g.,* CLIP, CoOp[2], CoCoOp[1]) that rely on the pre-defined vocabulary **fail to function** in this novel task, although VLM solutions such as [1] and [2] can achieves good performance for standard zero-shot FGVC task where the class names are already known. To be best of our knowledge, vocabulary-free FGVC task is underexplored in the existing literature. We believe this is an important yet realistic novel task for the community to explore. This newly proposed vocabulary-free FGVC task is inspired by the natural case:
>
> > "An bird lover gathered several unlabeled images from a webcam set up in the Amazon jungle. Confronted with the task of identifying the fine-grained bird species existing the the region from these few unlabeled observations, and without any pre-defined species names as prior knowledge from ornithologists, the bird lover encountered a significant challenge. In this context, our proposed FineR system is functional, enabling the bird lover to not only discover the bird species but also to classify them effectively in the incoming data stream from the webcam.“
>
>2. **Method perspective:** Our proposed method **is not** solely rely on LLM. In fact, the proposed FineR system jointly integrates the visual information translation capability of the **VQA model**, the reasoning capability of the **LLM**, and the zero-shot discrimination capability of **contrastive VLM** to tackle the proposed vocabulary-free FGVR task. To further clarify, FineR primarily operates in 3 steps: (i) first uses BLIP-2 to **translate** visual information (super-category and attribute descriptions) from the few (3 per class) unlabeled images to natural language; (ii) then uses the visual cues in text to elicit the reasoning capability of ChatGPT for **discovering fine-grained categories**; (iii) lastly uses CLIP to classify/recognize fine-grained categories in the test data stream based on the discovered concetps.
>
>3. **The necessity of using LLMs:** Insightful question! In our work, **the role of LLM is a Reasoning Engine**, just like a human expert would reason fine-grained categories based on the visual input. Furthermore, to further study "Is it necessary to use LLMs?", we constructed strong baseslines that adopt different solutions: (i) Learning-based Sinkhorn; (ii) Knowledge base based WordNet (use CLIP to retrive categories from WordNet); (iii) and BLIP-2 (VQA), as well as compared  with SOTA methods (SCD and CASED) based on large corpus knowledge bases. As shown in the quantitative results in Tab. 1 and qualitative results in Fig. 6 of the paper, our FineR system not only performs better but also makes more plausible mistakes. This demonstrates the advantages of using LLMs to **reason** in vocabulary-free FGVC task.

[1] Conditional Prompt Learning for Vision-Language Models. CVPR 2022.
[2] Learning to Prompt for Vision-Language Models. IJCV 2022.

Reviewer comment (Novelty):
The idea to use LLM to generate concepts for FGVC is already not new now. Some recent works implement this idea: [3] Learning concise and descriptive attributes for visual recognition. ICCV 2023.

Author response:
>-----
>**Q2:** The idea of this paper is actually not very new now. The ICCV paper [3] implements similary idea. The difference and comparison should be made and clarified.
>
>**A2:** We would like to clarify that the settings and methods of our paper and [3] are very different.
> * **Setting**. The work in [3] assumes the fine-grained categories are **known**. On the contrary, we do not assume any knowledge about the class names.
> * **Implementation**. In [3], a LLM is queried to simply enhance the descriptions of the already known concepts (or class names) to improve classification. Instead, we use VQA model to first identify super-category (e.g., "Bird") from an image and then acquire useful attribute names (no description) for distinguishing the fine-grained sub-categories. Then we use a VQA model to describe the attribute values in the image. The acquired attribute-description paris are then used to invoke LLM reasoning for discovering the fine-grained categories that are **unknown**. **In a nutshell, [3] only uses LLM to describes known classes, while our method uses the interaction between LLM and VLM to reason unknown classes.**
>
>In fact, [3] falls into the same group of LLM-based VLM prompt enrichment methods as the ICLR 2023 work [4] that we discussed in the second paragraph of Section 4. We have now also included [3] in the second paragraph Line#3 of Section 4 in the revision. Thanks for the suggestion!

[3] Learning concise and descriptive attributes for visual recognition. ICCV 2023.
[4] Visual Classification via Description from Large Language Models. ICLR 2023.

>-----

Reviewer comment (Novelty):
The novelty of this paper should be further demonstrated. The proposed method seems an intuitive combination of existing large-scale models, such as the visual question answering model, large-language model and vision-language model, etc. Besides, extracting visual attributes from images for recognition is widely used in generalized zero-shot learning methods such as [a].

Author response:
>**Q1.1:** The novelty should be clarified.
>
>**A1.1:** Thanks for the suggestions. Our work, we believe, is novel in several key aspects:
>* **Innovative Application/Task:** To the best of our knowledge, our work is the first to tackle the novel task of **vocabulary-free FGVR with only a few unlabeled samples as natural observation**. To tackle this novel task we harness the reasoning capabilities of LLMs and leverage the Internet-scale expert knowledge encoded in their weights to first discover the fine-grained categories that are present in a dataset, which is otherwise not know a priori. While CaSED (Conti et al., 2023) previously addressed vocabulary-free classification by retrieving concepts from a knowledge-base, our method is tailored specifically for FGVR by leveraging the knowledge in LLM. This also resonates with the comments from `Reviewer 3hkC` and `Reviewer WYuk` who commented that utilization of LLM for FGVR as "**a new way**" and "**a promising way**", respectively.
>
>* **Methodological Advancements:** We introduced **FineR**, a novel system that integrates the visual information translation capability of VQA models, the reasoning capability of LLMs, and the zero-shot discrimination capability of contrastive VLMs to address the vocabulary-free FGVR task. This approach is recognized by `Reviewer WYuk` as "**novel**" and "**interesting**". Furthermore, as shown in the preliminary experiments in Fig. 2, we observed that other LLM-based methods requiring training alignments (such as BLIP-2 learning Q-former, LLaVA, and MiniGPT-4 learning a projector) struggle to discover fine-grained categories effectively due to the limitations of alignment data. These training-based methods cannot fully utilize the reasoning capabilities of LLMs. In contrast, our FineR system translates visual information from unlabeled images into natural language in order to fully harness the reasoning power of LLMs for fine-grained concepts. Thus, we showed that visual and textual concepts can be aligned *without* the need of expensive training.
>
>* **Resource Efficiency and Accessibility:** Our approach is both training-free and vocabulary-free, significantly reducing the computational and data resources typically required for vision model and LLM alignments. This enhances the accessibility and feasibility of our method for a broader range of users, potentially democratizing the application of FGVR across various sub-fields.
>
>* **Scalability and Generalization:** Being training-free enhances the scalability and generalizability of our methods. As demonstrated in Fig. 7 (Section 3.2, page 8) and Fig. 24 (Appendix K.3, page 31), our method can effectively discover wild and novel fine-grained Pokémon categories, outperforming other methods that fail in these wild scenario.
>
> > We have summarized our novelty and contributions at the end of the Introduction section (Sec. 1) in the revision.

Reviewer comment (Experiments):
The effectiveness of the proposed method should be further verified. The recognition accuracy on the CUB dataset is relatively low. Besides, existing generalized zero-shot learning methods such as [a] should be compared and more corresponding analyses should be added.

Author response:
>**Q2:** The effectiveness of the proposed method should be further verified. The recognition accuracy on the CUB dataset is relatively low. Besides, existing generalized zero-shot learning methods such as [a] should be compared and more corresponding analyses should be added. More corresponding experiments and analyses should be added.
>
>**A2:** We first would like to clarify that it is not fair to directly compare between our approach with generalized zero-shot learning methods. It's crucial to highlight that our approach is both **vocabulary-free** and **training-free**, which fundamentally differs from GZSL methods. Directly comparing our method's recognition accuracy with that of GZSL approaches, which are **supervised** on numerous base seen classes with class name and attribute annotation and jointly evaluated on both seen and unseen classes, would be inappropriate. For instance, GZSL methods' accuracy on the CUB-200 dataset is jointly evaluated on 150 seen and 50 unseen classes. In contrast, all 200 classes of CUB are unseen in our evaluation. Moreover, as outlined in the setting comparison in **Table 1 and Table 2 above**, all the prerequisites required by GZSL during training and inference, including class names, annotated base class data, and ground-truth attribute-description pairs are not available in our setting. In our vocabulary-free FGVR task, both GZSL and standard VLM zero-shot transfer methods, which depend on predefined vocabulary, **fail to function** due to the absence of class names. A thorough review of [a] leads us to conclude that a comparison with **GZSL approaches is not feasible in our setting due to the absence of class names**. In fact, standard VLM zero-shot transfer method (CLIP) was considered and present as the upper bound in our experiments. Notably, we have achieved a significant performance improvement on the CUB-200 dataset compared to the methods we evaluated against. Nevertheless, **we appreciate your suggestion and have added a discussion section in Appendix K.4** of the revised manuscript to specifically clarify the differences between our vocabulary-free FGVR setting and GZSL, along with comparisons to other related settings.

Reviewer comment (Presentation):
There lacks a complete recognition example for explaining the whole procedure, which should be added for better understanding.

Author response:
>**Q3:** There lacks a complete recognition example for explaining the whole procedure, which should be added for better understanding.
>
>**A3:** Thank you for your valuable suggestions! Indeed, **we have demonstrated the entire recognition process using an unlabeled "Italian Greyhound" image as an example in Appendix B.2, from Fig.12 to Fig.15**. This includes the visual input, the identified super-category ('Dog'), attribute names obtained from the LLM, attribute descriptions extracted by the VQA model, the complete LLM reasoning prompt, and the LLM's output results. In response to your advice, we have added a new figure in **Appendix B.2 Fig.11** of the revised manuscript to more clearly illustrate the full pipeline with actual inputs and outputs.

Reviewer comment (Novelty):
Add the citation of some highly related missing works: (1) Transhp: image classification with hierarchical prompting; it also focuses on the fine-grained image classification task. Also, it takes advantage of the recently proposed prompting technique in CV. Is your used LLM better or prompting better? (2) V2L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval; it also focuses on the vision language model and fine-grained visual recognition. The paper is the champion of FGVC 9. A discussion of comparison seems necessary.

Author response:
>**Q1:** Add the citation of some highly related missing works.
>
>**A1:** Thanks you for your helpful advice. In the revision, we have now included these two highly related papers in the second paragraph of Related Work section (Sec.4) of the main paper. Furthermore, we have now also added a discussion section in Appendix K.4 of the revision to carefully compare our setting and method with respect to related works. In response to your question, we discuss the main differences between our work and TransHP and V2L below:
>* **TransHP**: We note that the very recent NeurIPS'23 FGVR work TransHP (arXived on 2023-October-23, which is after ICLR'24 deadline) is related to our study. Our method differs from TransHP in two ways: (i) We study the task of **vocabulary-free** FGVR, where neither coarse-grained class names nor the target fine-grained class names are known, unlike TransHP where coarse grained classes are known; (ii) Our method is also **training-free**. The only resources our method has access are few unlabeled images (3 per-class or [1,10] per class) as an observation for discovering the fine-grained class names. Under this realistic vocabulary-free FGVR setting TransHP *cannot operate* due to the lack of class names.
>
>
>* **V2L**: V2L learns and complementarily ensembles vision (ResNeSt, ResNeXt, CotNet, HS-ResNet, NAT, ViT, and ViT) and vision-language models (BLIP, ALBEF, XVLM, METER and SLIP) with the supervision from coarse-grained labels and textual description. Similar to TransHP, the lack of class names and supervisory signals makes V2L inoperable in our vocabulary-free scenario.

Reviewer comment (Experiments):
Could you show more examples of Fig. 2? Or any failure cases of Fig. 2?

Author response:
>**Q2:** Could you show more examples of Fig. 2? Or any failure cases of Fig. 2?
>
>**A2:** In the Appendix Fig. 20 we have provided more examples of Fig. 2. In Fig. 20 we have compared our method with several baselines. We observe that our FineR predicts the correct class more number of times than the baselines.
>
>In addition, following your suggestion we have added a thorough qualitative failure case analysis in **Appendix I, Fig. 21**. Specifically, Fig. 21 qualitatively presents the failure cases of our method across each dataset (including five fine-grained datasets and the proposed Pokémon dataset) and compares these with vision-language models (BLIP-2, LENS, LLaVA, MiniGPT-4). We observe that most of the baselines fail to correctly identify the finegrained categories. On the contrary our FineR, albeit not being able to predict the exact fine-grained category, it can still capture some fine-grained attributes. For eg., it predicts the dog breed *Brabancon Griffon* as *Brussels Griffon*, demonstrating that it can capture the characteristics of a *Griffon* dog. This suggests that our FineR makes *less severe* mistakes than the baselines. 
>
>Intriguingly, on the novel Pokémon dataset, almost all compared methods predict only the real-world counterparts of the Pokémon characters (e.g., 'Squirtle' as 'Turtle' and 'Pidgey' as 'Bird'), highlighting a domain gap between real-world and virtual categories. In contrast, although not completely accurate, our method offers predictions that are closer to the ground truth (evolutionized Pokemon characters, 'Pidgey' as 'Pidgeotto' and 'Squirtle' as 'Wartortle') for the two Pokémon objects. This failure case analysis further confirms that our system effectively captures fine-grained visual details from images and leverages them for reasoning. It demonstrates that FineR not only generates more precise and fine-grained predictions when correct but also displays high semantic awareness when it errs.

Reviewer comment (Novelty):
What is your view of NOVELTY of a work using LLMs without training anything?

Author response:
>**Q3:** What is your view of NOVELTY of a work using LLMs without training anything?
>
>**A3:** Thank you for your insightful question! Following your suggestion, we have outlined the novel aspects of our work at the end of the Introduction section (Sec. 1) in the revision. Our work, we believe, is novel in several key aspects:
>* **Innovative Application/Task:** To the best of our knowledge, our work is the first to tackle the novel task of **vocabulary-free FGVR with only a few unlabeled samples as natural observation**. To tackle this novel task we harness the reasoning capabilities of LLMs and leverage the Internet-scale expert knowledge encoded in their weights to first discover the fine-grained categories that are present in a dataset, which is otherwise not know a priori. While CaSED (Conti et al., 2023) previously addressed vocabulary-free classification by retrieving concepts from a knowledge-base, our method is tailored specifically for FGVR by leveraging the knowledge in LLM. This also resonates with the comments from `Reviewer 3hkC` and `Reviewer WYuk` who commented that utilization of LLM for FGVR as "**a new way**" and "**a promising way**", respectively.
>
>* **Methodological Advancements:** We introduced **FineR**, a novel system that integrates the visual information translation capability of VQA models, the reasoning capability of LLMs, and the zero-shot discrimination capability of contrastive VLMs to address the vocabulary-free FGVR task. This approach is recognized by `Reviewer WYuk` as "**novel**" and "**interesting**". Furthermore, as shown in the preliminary experiments in Fig. 2, we observed that other LLM-based methods requiring training alignments (such as BLIP-2 learning Q-former, LLaVA, and MiniGPT-4 learning a projector) struggle to discover fine-grained categories effectively due to the limitations of alignment data. These training-based methods cannot fully utilize the reasoning capabilities of LLMs. In contrast, our FineR system translates visual information from unlabeled images into natural language in order to fully harness the reasoning power of LLMs for fine-grained concepts. Thus, we showed that visual and textual concepts can be aligned *without* the need of expensive training.
>
>* **Resource Efficiency and Accessibility:** Our approach is both training-free and vocabulary-free, significantly reducing the computational and data resources typically required for vision model and LLM alignments. This enhances the accessibility and feasibility of our method for a broader range of users, potentially democratizing the application of FGVR across various sub-fields.
>
>* **Scalability and Generalization:** Being training-free enhances the scalability and generalizability of our methods. As demonstrated in Fig. 7 (Section 3.2, page 8) and Fig. 24 (Appendix K.3, page 31), our method can effectively discover wild and novel fine-grained Pokémon categories, outperforming other methods that fail in these wild scenario.
> > We have summarized our novelty and contributions at the end of the Introduction section (Sec. 1) in the revision.