Paper: Integrating Visual Cues via Prompting for Low-Resource Multimodal Named Entity Recognition (ICLR-2024)

Reviewer comment (Novelty):
The contribution of this paper is quite limited. The authors use a large number of FLAN-based models (both text-only and multimodal) for multimodal NER tasks.

Author response:
The contribution is indeed limited by the inclusion of few varieties of models. We have done additional experiments, not discussed in the paper due to limited space.  We believe the significance of our MNER study was illustrated well by the experiments and models we have included in the updated version of the paper.

Reviewer comment (Novelty):
The method is not new. QA for NER can be traced back to [1] and there are recent works using QA for all multimodal IE tasks in both zero-shot and few-shot settings.

Author response:
QA is indeed not new, but the paper mentioned by the reviewer is unimodal, while we focus on the multimodal setting.  Our use of MLLMs specifically to MNER in this prompting fashion on the other hand is. You mention one work [2] published AFTER the ICLR submission deadline, we do not regard it to be fair to use it as an argument against our paper’s contribution. We anticipate that there were numerous recent works in zero and few-shot learning last time. These developments likely span both unimodal and multimodal applications, which may perform similar tasks.

Reviewer comment (Experiments):
The evaluation is very limited. More models, including both instruction-tuned (IlIava, MiniGPT4, etc.) and base multimodal models (Open Flamingo), should be evaluated. Additionally, analysis of more tasks would be more interesting. While multimodal NER is to-some-extend important, it is not representative enough. Evaluations on other multimodal information extraction tasks would be beneficial.

Author response:
We agree with the reviewer that the inclusion of more models would have been a benefit.  Indeed, we have worked on and have results for this, even though we did not publish it here due to limited space. While additional tasks indeed are interesting, this is again a question about depth version breadth for a 9-page ICLR paper.  Our opinion upon submission was that the scope of our paper was similar to that of previously published ICLR papers.

Reviewer comment (Novelty):
The methodology section of this work primarily focuses on the design of instructions and lacks technical innovation, making it unsuitable for publication in ICLR.

Author response:
We respect the reviewer’s opinion. At the same time, we would like to point to a lack of concrete arguments about what this weakness consists of. In support of our work, we would like to mention that papers studying prompting methods used in novel ways have been accepted to ICLR previously (Progressive Prompts: Continual Learning for Language Models, for example). We agree with the reviewer that the instruction part of our paper is not innovative.  However, we strongly argue that the multimodal NER setting is innovative.

Reviewer comment (Experiments):
Table 7 presents results from previous baselines in a fully supervised setting, but the experiments conducted earlier in this paper do not compare against these baselines, making it unclear how the conclusion "Results against state-of-the-art models" is derived.

Author response:
You correctly point out that we don’t have comparative results of low-resource MNER experiments. The reason for this is that we have not found other low-resource results for these datasets.  As a result, apples-to-apples comparisons indeed become challenging. Our assumption, on the other hand, was that our low-resource results should not have an unfair advantage over other researchers’ full-resource results.  (Most people would argue that it is the other way around.)  We have provided some more context and discussion for our results, emphasizing that our low-resource results are at a disdvantage to the full-resource results,  but it’s correct that this is a bit of an apples to oranges comparison.

Reviewer comment (Evaluation):
How can we draw the conclusion with the statement "Results against state-of-the-art models for MNER are provided in Table 7"?

Author response:
This was not worded well enough in the original version of the paper but has now been clarified.   We mean to say that we can look at our results in the context of fully fine-tuned results.

Reviewer comment (Novelty):
The related works are not comprehensive enough. For example, it lacks some pre-trained models and multimodal NER models in the recent two years (2022-2023). It is unclear with the current presentation compared to previous works. For example, the below reference also considers the multimodal NER.  
- Fei Zhao, Chunhui Li, Zhen Wu, Shangyu Xing, Xinyu Dai. Learning from Different text-image Pairs: A Relation-enhanced Graph Convolutional Network for Multimodal NER. ACM Multimedia 2022: 3983-3992.  
- Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, Fangsheng Weng. RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER. AAAI 2021: 13860-13868.  
- Lin Sun, Jiquan Wang, Yindu Su, Fangsheng Weng, Yuxuan Sun, Zengwei Zheng, Yuanyi Chen. RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER. COLING 2020: 1852-1862.

Author response:
It’s correct that not all works on MNER have been referenced.   This was due to limiting our references to the most relevant related work, as MNER is a broad task where the methods used differ widely. Two of the papers you mention (RpBERT and RIVA)  were in fact already referenced and included in Table 7 (now Table 6).  Further, we have included the last reference you mentioned in the revised version of the paper, as it is a highly relevant paper.  Thanks for the reference to this work.

Reviewer comment (Evaluation):
Based on the statistics on the datasets used, it is category unbalanced. Therefore, using the micro-f1 as the evaluation only may be not convincible.

Author response:
It’s a very good point that the micro-F1 score is not necessarily the best metric.  This was somewhat alleviated by including the entity-level F1 scores, which then alleviates this somewhat. Still, we have mentioned the evaluation metrics and other possible metrics possible in the revised paper. The further inclusion of precision and recall could have shown more nuance in the results, that’s correct, but also other metrics and so we refuge to mentioning this as caveats and suggest future work.

Reviewer comment (Novelty):
The author(s) need to comprehensively research the related datasets. It is not like the authors claimed that “Notably, these are currently two of the few openly accessible multimodal datasets for named entity recognition.”  
- Dianbo Sui, Zhengkun Tian, Yubo Chen, Kang Liu, Jun Zhao. A Large-Scale Chinese Multimodal NER Dataset with Speech Clues. ACL/IJCNLP (1) 2021: 2807-2818  
- Xigang Bao, Shouhui Wang, Pengnian Qi, Biao Qin. Wukong-CMNER: A Large-Scale Chinese Multimodal NER Dataset with Images Modality. DASFAA (3) 2023: 582-596

Author response:
Related literature (including the abovementioned papers) only handles the two mentioned Twitter datasets.  There is an additional dataset from Snapchat. However, it is not publicly available. You interestingly mention other available multimodal datasets. However, the first one is audio and text, not image and text, which we focus on, in addition to being in Chinese (while our focus is on English text). The last mentioned dataset is also in Chinese and was recently released in 2023, which is perhaps why we have overlooked it. Still, there seem to be very few datasets available, so while an interesting point, it’s not necessarily wrong.

Reviewer comment (Novelty):
The low-resource MNER ability of the proposed method mainly comes from the existing zero-shot learning of multimodal LLMs and a series of manually designed prompts, lacking further optimization for MNER.

Author response:
Indeed, there have not been added several optimizing steps in the process specifically for MNER, but on the other hand this is due to the method being interesting because it’s so versatile (simple prompting). Adapting to various contexts without needing specialized optimization makes it broadly applicable/usable/implementable.

Reviewer comment (Experiments):
Lacking the zero-shot and few-shot experimental results of existing MNER methods (such as a series of BERT-based models and LLMs-based models) in evaluation. Unable to accurately measure the advantages of the proposed method compared to other methods.

Author response:
Such experimental results are lacking for the following reasons. Among the literature we reviewed, no other methods have done zero-shot ML on MNER as we have.  The reviewer correctly points out that we did not implement another method using BERT (for example) that could zero-shot. This is perhaps a good suggestion for future work.  It should be noted that there are many considerations to be made if we want to be able to zero-shot BERT (assuming with a custom classifier head) and also compare it to a larger causal LM. We assume that the low-resource case in our experiment, the “true few-shot setting” should not do better than full-resource case exemplified by the surveyed results in Table 7 (now 6), and we therefore thought it was a not perfect but acceptable comparison under constraints of limited time.

Reviewer comment (Evaluation):
It is not clear that the author(s) show the Zero-shot F1 in Figure 5 and Figure 6, which is not consistent with the previous claim. Also, the micro F1 scores by model with images are lower than that without images in Table 5 and Table 6. If there are some clerical errors?

Author response:
We are not entirely sure what you mean regarding Figures 5 and 6, as there are no Figures 5 and 6 in the paper, we assume that tables are what is meant, as the Zero-shot F1 scores are mentioned in Table 5 and Table 6 (now 4 and 5). Indeed, experiments with images sometimes have lower scores than those without. This is not a mistake. We state and reflect on this in the submitted paper, and this has been further clarified in the revision.

Reviewer comment (Reproducibility):
How to compute the c(A_i) is not clear.

Author response:
This is a good point. We have clarified how to compute this in the revision. It’s the outputted logits from the model.

Reviewer comment (Presentation):
Table 4 is not referenced.

Author response:
Well pointed out, it should be fixed in the revision.