DAMO - NLP at S em E val-2023 Task 2: A Unified Retrieval-augmented System for Multilingual Named Entity Recognition (2023)

Abstract: The MultiCoNER II shared task aims to tackle multilingual named entity recognition (NER) in fine-grained and noisy scenarios, and it inherits the semantic ambiguity and low-context setting of the MultiCoNER I task. To cope with these problems, the previous top systems in the MultiCoNER I either incorporate the knowledge bases or gazetteers. However, they still suffer from insufficient knowledge, limited context length, single retrieval strategy. In this paper, our team DAMO-NLP proposes a unified retrieval-augmented system (U-RaNER) for fine-grained multilingual NER. We perform error analysis on the previous top systems and reveal that their performance bottleneck lies in insufficient knowledge. Also, we discover that the limited context length causes the retrieval knowledge to be invisible to the model. To enhance the retrieval context, we incorporate the entity-centric Wikidata knowledge base, while utilizing the infusion approach to broaden the contextual scope of the model. Also, we explore various search strategies and refine the quality of retrieval knowledge. Our system wins 9 out of 13 tracks in the MultiCoNER II shared task. Additionally, we compared our system with ChatGPT, one of the large language models which have unlocked strong capabilities on many tasks. The results show that there is still much room for improvement for ChatGPT on the extraction task.

## introduction the multiconer series shared task @xcite aims to identify complex named entities (ne), such as titles figure 1 : an example of wrong prediction in raner @xcite (one of the top systems in the multiconer i task @xcite ) . this case illustrates that the knowledge covered is not sufficient for fine-grained complex ner. the previous top systems @xcite of the multiconer i task incorporate additional knowledge in pre-trained language models, either a knowledge base or a gazetteer. raner @xcite builds a multilingual knowledge base based on wikipedia and the original input sentences are then augmented with retrieved contexts from the knowl-edge base, allowing the model to access more relevant knowledge. gain @xcite proposes a gazetteer-adapted integration network with a gazetteer built from wikidata to improve the performance of language models. although these systems achieve impressive results, they still have some drawbacks. first, insufficient knowledge is a common problem. as shown in figure 1 , the knowledge used in raner can help the model to identify erving goffman as a person, but cannot further determine the fine-grained category artist. second, these methods mostly suffer from the limited context length. @xcite discards stitched text that is longer than 512 after tokenizing, which means that plenty of retrieved context is not visible to the model, leading to resource waste. third, these systems have a single retrieval strategy. @xcite acquires knowledge by text retrieval, while @xcite accesses knowledge by dictionary matching. this single way of knowledge acquisition will result in the underutilization of knowledge. to tackle these problems, we propose a unified retrieval-augmented system (u-raner) for finegrained multilingual ner. we use both wikipedia and wikidata knowledge bases to build our retrieval module so that more diverse knowledge can be considered. as shown in figure 1 , if we locate the entry for erving goffman in wikidata, we can make use of fine-grained entity category information to facilitate predictions. also, we discover that the retrieval context dropped by the model may also contain useful knowledge. thus, we explore the infusion approach to make more context visible to the model. in addition, we use multiple retrieval strategies to obtain the most relevant knowledge from two knowledge bases, further improving the model performance. our main contributions are as follows: ## related work named entity recognition (ner) @xcite is a fundamental task in natural language processing. because of the long-term attention and the rapid development of pre-trained language models, various models @xcite @xcite @xcite have achieved state-of-the-art results and performance in general ner scenarios and datasets, such as conll 2002 @xcite , conll 2003 @xcite , and ontonotes 5.0 @xcite . considering that the previous task settings or datasets are monolingual and scenarioconstrained, the task of multilingual complex named entity recognition (multiconer) is proposed to promote the ner research to be more oriented to real scenarios @xcite . our work focuses on this task and we will introduce the related work from the dataset and method of multiconer respectively: challenges of multiconer dataset to address contemporary in the ner field, malmasi et al. construct multiconer, a large and complex dataset for multilingual complex named entity recognition. this 26m token dataset covers 3 domains (including wiki, question, and search query) and 11 languages (12 languages for semeval-2023). in particular, aiming at the main challenges of ner research, the multiconer dataset sets 4 key characteristics: (1) low context: existing ner methods perform poorly if the context is less informative @xcite , thus, texts in multiconer are low in context to assess the model's performance on the more realistic and difficult setting. note that in the dataset version of semeval-2023, this challenge and setting do not exist. ## data the multiconer ii corpus @xcite aims to recognize the complex named entities and pose new challenges for current ner systems. to meet these challenges, we first reproduce the results of the top system @xcite and perform error analysis on validation sets. we observe that the performance bottleneck of the system lies in the lack of knowledge. then, we investigate to break this bottleneck from data and model perspectives and improve model robustness. following @xcite , we build a multilingual kb based on wikipedia of the 12 languages to search for the related documents. we download the latest (2022.10.21) version of the wikipedia dump from wikimedia foot_0 and convert it to plain texts. we execute the official system on multiconer ii corpus and categorize the results according to whether the annotated entity appears in the retrieval context or not. as shown in retrieval strategy query retrieval result text2text from 1995 to 2011 deal hudson was the magazine's publisher. 1. in 1995 hudson became publisher of the conservative roman catholic magazine, crisis. 2. hudson is the former publisher and editor of 3. hudson also hosts the radio show church and culture on ave maria radio ... text2ent from 1995 to 2011 deal hudson was the magazine's publisher. 1. deal w. hudson 2. deal wyatt hudson 3. s. hudson ... ent2ent [deal hudson] type: human description: hudson is the former publisher and editor of crisis magazine and insidecatholic.com. while @xcite uses wikidata to build their gazetteer, we explore to enhance our retrieval system with wikidata. wikidata is a free and entitycentric knowledge base. every entity of wikidata has a page consisting of a label, several aliases, descriptions, and one or more entity types. as shown in figure 2 , base indicates that only the wikipedia knowledge base is used, and more database indicates that we use both wikipedia and wikidata knowledge bases. the entity coverage improves on all 4 languages and achieves the maximum gain of 12.6% on zh. in addition, as more context shows, expanding the length of the retrieval context also brings more entity knowledge. thus, we use the infusion approach to make more retrieval context visible to model. more details are described in section § 4.2. ## methodology overview as depicted in fig. 3 , u-raner is comprised of two parts: a retrieval augmentation module and a ner module. the retrieval augmen-tation module utilizes multiple retrieval strategies and the ner module adopts a modified transformer structure to utilize the retrieved knowledge. given an input sentence, u-raner retrieves similar texts and entities as external knowledge, which are then utilized in the form of text and vectors to help the ner module obtain improved predictions. ## conclusion in this paper, we propose a unified retrievalaugmented system (u-raner) for the multi-coner ii shared task, which wins 9 out of 13 tracks in the shared task. we expose that the bottleneck of the previous top system is the lack of knowledge. accordingly, we use both wikipedia and wikidata knowledge bases with three retrieval approaches so that more diverse knowledge can be considered. also, we explore the infusion approach to make more context visible to the model so as to make the best use of the resources. and the error analysis indicates that the entity typing sub-task is the bottleneck in the current system. in the future, we plan to exploit the knowledge in the large language model such as chatgpt or llama by self-verification or fine-tuning some adapters, in order to achieve robust generalization performance.