SCANNER : Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities (2024)

Abstract: Recent advances in named entity recognition (NER) have pushed the boundary of the task to incorporate visual signals, leading to many variants, including multi-modal NER (MNER) or grounded MNER (GMNER). A key challenge to these tasks is that the model should be able to generalize to the entities unseen during the training, and should be able to handle the training samples with noisy annotations.To address this obstacle, we propose SCANNER (Span CANdidate detection and recognition for NER), a model capable of effectively handling all three NER variants.SCANNER is a two-stage structure; we extract entity candidates in the first stage and use it as a query to get knowledge, effectively pulling knowledge from various sources.We can boost our performance by utilizing this entity-centric extracted knowledge to address unseen entities.Furthermore, to tackle the challenges arising from noisy annotations in NER datasets, we introduce a novel self-distillation method, enhancing the robustness and accuracy of our model in processing training data with inherent uncertainties.Our approach demonstrates competitive performance on the NER benchmark and surpasses existing methods on both MNER and GMNER benchmarks.Further analysis shows that the proposed distillation and knowledge utilization methods improve the performance of our model on various benchmarks.

## introduction named entity recognition (ner) is a fundamental task in natural language processing to identify textual spans that correspond to named entities in the given text, and classify them into pre-defined categories, such as persons, locations, and organizations @xcite . the extracted information can be utilized for various downstream tasks, including entity linking and relation extraction. the rapid growth of the amount of multi-modal contents on social media platforms has given rise to the multi-modal variants of ner. the most prominent example is multi-modal ner (mner; @xcite ) , which extends traditional ner to identifying named entities in the text based on additional image input paired with the text (fig. 1b ). another recent example is the grounded mner (gmner; @xcite ); here, one additionally aims to predict the bounding boxes of named entities appearing in the given image (fig. 1c ). a major challenge in ner, mner, and gm-ner tasks is the presence of unseen entities in the test datasets, which are not found in the training datasets. traditional models often struggle with low performance on these unseen entities (see text dataset [the [oval] org ] org conll2003 [the [world cup] misc ] misc twitter-2015 [taste of [toronto] loc ] misc twitter-2015 [mrs. [brozik] per ] per twitter-2017 [[robert downey ] per jr] per twitter-2017 our approach demonstrates competitive performance on ner and surpasses existing methods on both mner and gmner. further analysis shows that the proposed distillation and knowledge utilization methods improve the performance of our model on various benchmarks. the contributions of scanner are summarized in three key aspects: ## related work prior works on mner typically operates by first extracting the ner-related features from the image, and then combining these features with text features to recognize name entities. roughly, ex-isting works fall into two categories according to how they extract image features. textual features. several works extract the textual metadata from the given image and utilize them as features for the subsequent ner task @xcite . for instance, ita @xcite extracts object tags, image captions, and ocr results from the given image. similarly, @xcite also extracts image captions, but additionally utilizes large language model as an implicit knowledge source to further refine the features. more @xcite takes a slightly different approach, using an image-based retrieval system to retrieve textual descriptions of the closest images in the database. ## method in this section, we first introduce the architecture of the proposed method, which comprises the span candidate detection module and the named entity recognition module (sec. 3.1). then, we describe the named entity recognition module, which performs entity recognition and visual grounding in the image for each entity candidate (sec. 3.2). finally, we explain a novel distillation method, named trust your teacher, which is designed to entity recognition module input sentence entity candidate object knowledge extractor wiki knowledge extractor image captioner span candidate detection module ## conclusions we introduce scanner, a novel approach for performing ner tasks by utilizing knowledge from various sources. to efficiently fetch diverse knowledge, scanner employs a two-stage structure, which detects entity candidates first, and performs named entity recognition and visual grounding on these candidates. additionally, we propose the novel distillation method, which robustly trains the model against dataset noise, demonstrating superior performance in various ner benchmarks. we believe that our method can be easily extended to utilize knowledge from multiple sources that were not covered in this paper.