In-context Learning for Few-shot Multimodal Named Entity Recognition (2023)

Abstract: Thanks in part to the availability of copious annotated resources for some entity categories, existing studies have achieved superior performance in multimodal named entity recognition (MNER). However, in the real-world scenario, it is infeasible to enumerate all entity categories in advance. Therefore, in this paper, we formulate a new few-shot multimodal named entity recognition (FewMNER) task, which aims to effectively locate and identify named entities for a text-image pair only using a small number of labeled examples. Further, we explore the merit of in-context learning (ICL) and propose a novel framework to deal with FewMNER, where three points are taken into account: i.e., converting visual modality, selecting useful examples, and designing an effective task demonstration. Specifically, we first employ an image caption model to convert images into textual descriptions, enabling large language models to absorb information from visual modality. Then, we use the ranking of the sum of similarity rankings from both text and image modalities to select k-nearest examples, which form a demonstration context. Finally, we utilize the MNER definition and the meaning of each entity category as effective instruction. Extensive experimental results demonstrate that our framework outperforms baselines under several few-shot settings.

## introduction multimodal named entity recognition (mner) aims to identify named entities of different categories from the text with extra image assistance. consider the example in figure 1 (a), we need to recognize three named entities from the text, * corresponding author "suarez" (per), "barcelona" and "la liga" (org), to finish the mner task. most existing methods commonly employ pre-trained models followed by fine-tuning to accomplish the mner task and achieve superior performance @xcite @xcite . in terms of existing research efforts, their superior performance generally relies on sufficient annotated data, which is time-consuming and laborintensive. in addition, in practice, entity categories will continue to emerge rather than remain fixed. therefore, it is impractical to define all entity categories in advance. to address these issues, motivated by the fewshot named entity recognition (fewner) task that involves learning unseen entity categories from a small number of labeled examples @xcite , we extend the mner task to the few-shot field, named the few-shot multimodal named entity recognition (fewmner) task, which aims to locate and identify named entities for a text-image pair only using a small number of labeled examples. as illustrated in figure 1 further, to address the fewmner task, we propose leveraging the powerful in-context learning (icl) capability of the large language model (llm). specifically, we argue that this paradigm can provide a promising direction for solving the fewmner task by learning from a few examples in context without training. however, there are three problems while solving fewmner using the in-context learning paradigm: (i) for the fewm-ner task, each sample is represented by textual and visual modalities, while the input of llm is limited to natural language. thus, we first require seeking ways to convert visual modality into natural language form. (ii) the key to performing icl is to select a few examples to form a demonstration context. although there are some example selection studies @xcite ) targeting text classification tasks, selecting some useful examples for icl in multimodal scenarios has not been approached. (iii) in addition, good demonstration designing precisely is essential to obtain satisfactory performance. unlike simple classification tasks, the task instruction and output format of mner need to be constructed according to the extractive nature. to apply icl to solve the fewmner task, we propose corresponding solutions to the abovementioned problems. first, we employ an image caption model @xcite to generate textual descriptions from images, which not only converts images into natural language form but also aligns image features into the text space. second, for selecting examples, we design an efficient sorting algorithm based on image and text similarity ranks, which can mitigate the similarity bias caused by different modality models. then, we utilize this algorithm to select top-k examples with the highest similarity to the current test sample. third, the demonstration design consists of two parts: instruction construction and demonstration construction @xcite . the former aims to inform llm about the current task. to provide more detailed information, we add the description of entity category meaning to the instruction. the latter is to define the demonstration template and order selected examples into the demonstration template. for the demonstration template, it consists of three components: image description, sentence, and output, where output is the label information. then, we pack selected top-k examples into the demonstration template in ascending order of similarity rank, such that the most similar example is nearest to the current test sample. finally, we concatenate instruction, demonstration, and the test sample as the input and feed it into llm to obtain the prediction output. the contributions of this paper are as follows: ## conclusion in this paper, we formulate multimodal named entity recognition as a few-shot learning problem, named few-shot multimodal named entity recognition (fewmner), to extend entity detection to unseen entity categories. to tackle fewmner, we propose a framework based on in-context learning by addressing three problems. experimental results show that our framework outperforms baselines in several few-shot settings. moreover, we conduct analysis experiments and find that selecting similar examples, sorting them in ascending order, and using more examples improve the performance of in-context learning. we also perform error analysis and observe that increasing the number of examples reduces boundary errors but not category errors. these results provide novel insights for future work on the fewmner task.