Paper: Tag2Text: Guiding Vision-Language Model via Image Tagging (ICLR-2024)

Reviewer comment (Reproducibility):
The tag system works based on pre-existing vocabulary. What happens with out-of-vocabulary terms?

Author response:
In response to your query about out-of-vocabulary terms:


* **Captioning Impact.** Our findings indicate that the out-of-vocabulary tags do not affect the caption resulting. We analyze the reason is that only tags within the vocabulary are utilized for the Image-Tag-Text generation training task.

* **Out-of-Vocabulary Recognition.** For the image tagging branch, Tag2Text remains in the pre-existing vocabulary of 3,429 categories, unable to recognize out-of-vocabulary categories. Nevertheless, for out-of-vocabulary recognition, the image-text alignment branch can be utilized. We anticipate its performance to be similar with CLIP and BLIP under the same training dataset.


* **Future Enhancements.** We are exploring enhancements to advance Tag2Text into an stronger open-set tagging model in the future works. One potential research route involves employing an off-the-shelf text encoder to encode pre-existing tag embeddings for training, enabling it to generalize open-vocabulary tag embeddings based on embedding similarity during inference. Current experiments in this direction have yielded effective results.

Reviewer comment (Experiments):
Have you tested retrieval based on short queries? E.g. retrieval using a term within the vocabulary vs some short retrieval query that is not part of the vocabulary (e.g. adjective + noun) vs retrieval with long captions typical of flickr/coco.

Author response:
Thank you for raising the aspect of retrieval with short queries, which is indeed a practical and valuable scenario.


* **Lack of Benchmarks.** However, there is currently a lack of established benchmarks for short query retrieval. Our experiment attempts of only using short query tags indicated inferior performance compared to longer captions on the Flickr benchmark. 


* **Real-World Evaluation.** We have conducted internal evaluations with practical examples. Our findings demonstrate that when handling short queries including in-vocabulary terms, Tag2Text significantly outperforms CLIP and BLIP. However, for short queries involving out-of-vocabulary terms, Tag2Text did not show a marked improvement.


* **Future Enhancements.** In light of these findings, we are committed to enhancing Tag2Text's efficacy in processing short queries involving out-of-vocabulary terms. We are exploring advancements in open-set tagging capabilities to address this challenge.

Reviewer comment (Reproducibility):
The detail of text semantic parser is not clear. Although it is based on existing work of [Wu et al. 2019], it should make clear how to obtain the corresponding tags in the paper.

Author response:
Thank you for the constructive suggestion. 

The parser operates based on the dependency tree, a grammatical structure that maps syntactic relationships within a sentence. With a set of established semantic rules is applied to the dependency tree, the parser identifies *'entities (=head + modifier)'* and their *'relationships'* based on the given sentence. For instance, given the sentence "A red alarm clock is on a wooden desk", the parser automatically parse this as: *'head': ['alarm clock', 'desk']*, *'modifier': ['red', 'wooden']*, *'relation': ['on']*.


We have included these details and examples in the Section 3.2.

Reviewer comment (Presentation):
In Fig. 2, it is difficult to understand which are users’ input desired tags, since they share the same forms of the recognized image tags.

Author response:
We are sorry for this confusion. In Figure 2, all green underlined tags within *'Tag2Text (User Specified)'* refer to user-specified tags. And we try to clarify Figure 2:

*  Benefits from the powerful tagging capability of Tag2Text, Tag2Text can integrate the recognized image tags to automatically generate comprehensive and detailed captions.


*  Tag2Text also allows users to input specified tags to generate corresponding captions. This flexibility, as highlighted by Reviewer Nwx3, offers a way of controlling caption generation through the use of input tags.


We have refined the captions of Figure 2 to clearly distinguish them.

Reviewer comment (Presentation):
In the Image-Tag-Text Generation paragraph, the introduction of this task is not clear. According to Fig. 4(c), it seems the text embedding should not be used as input, which is conflicting with the introduction in the corresponding paragraph.

Author response:
Sorry for this confusion. During the Image-Tag-Text Generation training process, the text embeddings are utilized as ground truths to optimize the model via Language Modeling Loss. In the Image text generation comparison of Figure 4, all the text embeddings are utilized as ground truths during training and as model outputs during inference.


Thanks for your question, we have clarified these details Section 3.3.

Reviewer comment (Experiments):
Data and Pre-training Settings: The author of the paper used a 4M setting, which includes training data from COCO (Common Objects in Context) and Visual Genome. NoCaps data is sourced from OpenImages and COCO. Importantly, this work does not incorporate any out-of-distribution data. The success of this approach on COCO-related tasks is attributed to the similarity between the pre-training data and COCO style, and it's noted that many works beyond BLIP face similar challenges.

Author response:
### Evaluation on Out-Of-Distribution Dataset. ###
> *"This work does not incorporate any out-of-distribution data. The success of this approach on COCO-related tasks is attributed to the similarity between the pre-training data and COCO style, and it's noted that many works beyond BLIP face similar challenges. "*

We recognize your major concern about evaluating on out-of-distribution datasets. We try to address this from two perspectives.

* **Text Perspective**. In our study, we follow the official NoCaps benchmark [1] and other well-established studies [2-8] as our training and evaluation settings. Notably, NoCaps categorizes benchmarks into in-distribution and **out-of-distribution subsets** based on COCO labels. In Table 5 of our paper, we report these metrics demonstrating that Tag2Text outperforms existing models, particularly in out-of-distribution scenarios. This evidences the ability of Tag2Text to generalize beyond COCO-style datasets in handling a **diverse range of textual contexts**.

* **Image Perspective**. In addition to textual contexts, we shared the same viewpoint on the necessity of including more out-of-distribution images in evaluation benchmarks. Therefore, in Table 1 of our paper, we have made progress for conducting comprehensive evaluation on the out-of-distribution OpenImages benchmark [9], which is a well-established benchmark in image tagging [10-12]. Significantly, Tag2Text achieves superior zero-shot performance even comparable to fully supervised models (**Tag2Text: 83.4 (zero-shot) vs. ML-Decoder 85.5 (full supervised)**), **without including any OpenImages training images in its training stage**. This result demonstrates the effectiveness of Tag2Text across **diverse image domains**.


Based on above analysis, we respectfully disagree with the view of *"this work does not incorporate any out-of-distribution data."* Since our experiments includes evaluation on out-of-distribution datasets to proves the effectiveness of our proposed methods, which is appreciated by Reviewer Kr1V and Nwx3.

Reviewer comment (Novelty):
Tag Introduction in Pre-training: The introduction of tags in pre-training is not a novel idea. There are existing works, like OSCAR, that have explored the concept of using tags to improve vision-language pre-training.

Author response:
### Novelty Concerns in Tag Introduction. ###

> *"Tag Introduction in Pre-training: The concept of using tags in pre-training is not new, with examples like OSCAR in the field. "*


We acknowledge this point and have tried our best to articulate our novel contributions, especially in comparison to works like OSCAR, in the Introduction of our paper:


* **Unidirectional vs. Bidirectional Improvements.** OSCAR utilizes tags to enhance vision-language models in a unidirectional manner. For getting image tags, OSCAR directly utilizes an off-the-shelf object detector (Faster RCNN).
In contrast, Tag2Text offers **bidirectional enhancements on both image tagging and vision-language models**. For image tagging, **Tag2Text pioneer utilizing large-scale image-text pairs instead of manual annotations, achieving superior zero-shot ability.**


* **Object Detector vs. Image Tagging**. Although using tags based on the object detector is useful, the trend in recent studies [4-8,13-14] is **shifting towards detector-free vision-language models**. Since the frozen object detector restricts the model's capacity and is time-consuming. Tag2Text **re-introduces the valuable tag guidance** into detector-free models via image tagging. This approach enables data scaling, and is both efficient and effective in improving VL tasks.

Reviewer comment (Experiments):
In Figure 1(b), the authors mention 'Actions.' Could you please clarify the source of this particular type of data or tag? Are there any documented instances of failure cases for Tag2Text?

Author response:
### Clarifications on 'Actions' in Figure 1(b) ###

> *"In Figure 1(b), the authors mention 'Actions.' Could you please clarify the source of this particular type of data or tag?  "*

We are sorry for this confusion. 

In Section 3.2, we detail the source of image tags. Specifically, we parse 4 million image-text pairs and select the most frequently occurring tag categories as our label system, including objects, scenes, attributes, actions. The action categories  (e.g., 'fly', 'stand', 'climb') are obtained from the *'relationship'* items after text parsing. We also provide more tag category details in Appendix B, including tag category statistics in Table 7, an illustration in Figure 8, and the overlapping with other public datasets in Table 8 and Table 9.

We have added an example in Section 3.2 of our revised manuscript.

### Failure Cases of Tag2Text ###
> *"Are there documented instances of failure cases for Tag2Text?"*

* **Hallucinatory Captions**. Tag2Text benefits from its powerful tagging capabilities. As depicted in Figure 6, there is a strong correlation between captioning performance and tagging guidance performance. In practical applications, we observe that incorrect user-provided tags may lead to hallucinatory captions. 

* **Small Objects**. In addition, evaluating a tagging model capable on 3,429 categories is also challenging. Our quantitative comparison and visual validations reveal that Tag2Text efficiently recognizes common objects and scenes, yet struggles in small objects (e.g., spoon or baseball). Our empirical experiments indicate that increasing the resolution during fine-tuning significantly improves performance on these small objects.

Thanks for your constructive question, we have added the Section G of "Limitation" in our revised manuscript.

Reviewer comment (Theory):
It would be beneficial for this paper to incorporate specific mathematical formulations to provide a more comprehensive description and discussion of the fundamental problems that require resolution.

Author response:
### Mathematical Formulations. ###
> *" It would be beneficial for this paper to incorporate specific mathematical formulations to provide a more comprehensive description and discussion of the fundamental problems that require resolution."*

Thanks for your valuable suggestion. 

We have provided mathematical formulations to detail different pre-training tasks in Appendix A.2.

Reviewer comment (Evaluation):
Metrics such as Precision, Recall, and F1 score, which are commonly used for image tagging and multi-label classification, are notably absent from the results.

Author response:
### Evaluation Metrics for Image Tagging. ###
> *" Metrics such as Precision, Recall, and F1 score, which are commonly used for image tagging and multi-label classification, are notably absent from the results."*

Thanks for your question. As detailed in Table 1 of our paper, we have indeed included a variety of evaluation metrics, encompassing mAP, Precision, and Recall, to comprehensively assess our model's performance.

* **mAP**. We utilize mAP as the primary metric when comparing with other tagging models, due to its well-established status in the field of image tagging.

* **Precision and Recall**. For comparison with vision-language models, where direct calculation of mAP is challenging, we employ Precision and Recall as our main metrics. 

In response to your feedback, the additional F1 score comparisons are presented below. Notably, Tag2Text demonstrates significant enhancements across various datasets. These results have been included in Table 2 of our revised manuscript.



| F1 Comparison | Training Images | Evaluation Paradigm | TagEval | OpenImages | COCO |
|---------------|-----------------|---------------------|---------|------------|------|
| CLIP| 400M  | Alignment | 63.4| 63.0| 48.2 |
| BLIP| 129M  | Alignment | 65.7| 64.8| 54.3 |
| BLIP| 129M  | Captioning| 58.6| 56.6| 55.7 |
| BLIP-2 | 129M  | Captioning| 58.2| 58.1| 59.1 |
| Tag2Text| 14M   | Captioning| 65.9| 62.7| 62.7 |
| Tag2Text| 14M   | Tagging   | **78.6**| **72.7**| **71.5** |

Reviewer comment (Writing):
The construction of the tag category system appears to involve human annotation in the process (Section 3.2), which contradicts the earlier claim of being an "automatic" approach (Section 1).

Author response:
### Clarification on Tag Category System Construction ###
> *" The construction of the tag category system appears to involve human annotation in the process (Section 3.2), which contradicts the earlier claim of being an "automatic" approach (Section 1)."*

We are sorry for this confusion. 

Extracting large-scale tags from texts through automatic text parsing is entirely automated with no manual annotation, as stated in Section 1: *"The pre-training image tags are obtained through automatically text semantic parsing"*

In Section 3.2, in order to build a more comprehensive label system, we filter the tag categories with a minimal amount of manual effort. For example, we merge synonyms such as 'person' and 'human'. We have added this example in Section 3.2 of our revised manuscript.

Reviewer comment (Experiments):
Conducting an ablation study on the choice of vocabulary set (tag set) size to be mined could offer valuable insights into the proposed method.

Author response:
### Impact on Vocabulary Set Size ###
> *"Conducting an ablation study on the choice of vocabulary set (tag set) size to be mined could offer valuable insights into the proposed method.."*

We appreciate this constructive suggestion!

In response, we have expanded the vocabulary set from 3,429 to 4,585 categories and compared the performances, as presented in the table below. Notably, the tagging performance decrease with the larger vocabulary set. We attribute to two possible reasons:

* The increased complexity in training with more categories.

* The additional categories leading to more noise, as they lack sufficient training data, thereby impacting the model's efficiency.

These results and analyses have been incorporated in Section B of our revised manuscript.

| Vocabulary Set Size|TagEval|OpenImages|
|-|-|-|
| 3,429 | 81.7| 84.1|
| 4,585 | 80.3| 83.1|

Reviewer comment (Writing):
Could the author give a short explanation of the meaning of title "Tag2Text"?

Author response:
### Clarification on Model Name "Tag2Text" ###
> *"Could the author give a short explanation of the meaning of title "Tag2Text"?."*

Sorry for the confusion. The name "Tag2Text" is chosen to vividly represent the core functionality of our model, which refers to the integration of image tags as guiding elements into text generation. Benefits from its powerful tagging capabilities, Tag2Text is able to generate more comprehensive and and detailed text descriptions, as illustrated in Figure 2 of our paper. 

In addition, Tag2Text also allows users to input specified tags to generate corresponding captions. This flexibility, as highlighted by Reviewer Nwx3, offers a way of controlling caption generation through the use of input tags.

Reviewer comment (Evaluation):
Performing a multi-label classification with 3,429 categories is non-trivial. The reviewer wonder if the authors come across any difficulties? And what the performance of the learned classifiers on those category (for example, on a validation set of VL pretraining)?

Author response:
### Image Tagging with 3,429 Categories ###
> *"Performing a multi-label classification with 3,429 categories is non-trivial. The reviewer wonder if the authors come across any difficulties? And what the performance of the learned classifiers on those category?"*

Thank you for raising this important point!

We share the same viewpoint that performing a tagging model with 3,429 categories presents significant challenges, primarily due to the scarcity of training data for such a large number of categories. To address this, we pioneer the utilization of large-scale image-text pairs instead of manual annotations. This approach, combined with our multi-task and pretrain-finetune manners, are proven effective for image tagging as detailed in Table 5 of our paper.


Evaluating a tagging model on such a wide category range is also challenging. Our quantitative comparison and visual validations reveal that Tag2Text efficiently recognizes common objects and scenes, yet struggles in small objects (e.g., spoon or baseball). Our empirical experiments indicates that increasing the resolution during fine-tuning significantly improves performance on these small objects.

Thanks for your constructive question, we have added the Section G of "Limitation" in our revised manuscript.

Reviewer comment (Experiments):
Have the authors ablate VL pretraining framework with a single task, for example, Image-Text alignment only based VL, Image-tagging only based VL, Image captioning only based VL?

Author response:
### Ablation Study with Single Task ###
> *"Have the authors ablate VL pretraining framework with a single task, for example, Image-Text alignment only based VL, Image-tagging only based VL, Image captioning only based VL?."*

Thanks for your insightful question.

In our paper, we have reported the results with only either VL tasks or tagging task. We summarize them in the table below. The table clearly demonstrates the effectiveness of our bidirectional enhancements on both image tagging and VL tasks.

|Pre-Training Objectives|Source of our Paper|Tagging|Captioning| Retrieval |
|-|-|-|-|-|
|Tagging|Table 5 Row 5|74.7|-|-|
|Captioning + Alignment |Table 4 Row 2 |-|123.3|74.4|
|Tagging + Captioning + Alignment|Table 5 Row 6 / Table 4 Row 3|76.4 / 82.9(+ft)|124.6|74.9|