DAMO - NLP at S em E val-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition (2022)

Abstract: The MultiCoNER shared task aims at detecting semantically ambiguous and complex named entities in short and low-context settings for multiple languages. The lack of contexts makes the recognition of ambiguous named entities challenging. To alleviate this issue, our team DAMO-NLP proposes a knowledge-based system, where we build a multilingual knowledge base based on Wikipedia to provide related context information to the named entity recognition (NER) model. Given an input sentence, our system effectively retrieves related contexts from the knowledge base. The original input sentences are then augmented with such context information, allowing significantly better contextualized token representations to be captured. Our system wins 10 out of 13 tracks in the MultiCoNER shared task.

## introduction the multiconer shared task @xcite aims at building named entity recognition (ner) systems for 11 languages, including english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, and bangla. the task has three kinds of tracks including one multilingual track, 11 monolingual tracks and one code-mixed track. the multilingual track requires training multilingual ner models that are able to handle all languages. the monolingual tracks require training individual monolingual models where each model works for only one language. the code-mixed track requires handling code-mixed samples (sentences that may involve multiple languages). the datasets mainly contain sentences from three domains: wikipedia, web questions and user queries, figure 1: a motivating example from the english test set. in the retrieval results, the bold phrases are the title of the retrieved page and the underlined phrases contain the hyperlinks to other pages. loc and grp are entity labels representing location and group respectively. in this paper, we propose a general knowledgebased system for the multiconer shared task. we propose to retrieve the related documents of the input sentence so that the recognition of difficult entities can be significantly eased. based on wikipedia of the 11 languages, we build a multilingual kb to search for the related documents of the input sentence. we then feed the input sentence and the related documents into the ner model. moreover, we propose an iterative retrieval approach to im-prove the retrieval quality. during training, we propose multi-stage fine-tuning. we first train a multilingual model so that the ner model can learn from all annotations. next, we train the monolingual models (one for each language) and a code-mixed model by using the fine-tuned xlm-roberta (xlm-r) @xcite embeddings in the multilingual model as initialization to further boost model performance on monolingual and code-mixed tracks. for each track, we train multiple models with different random seeds and use majority voting to form the final predictions. besides the system description, we make the following observations based on our experiments: 1. knowledge-based systems can significantly improve both in-and out-of-domain performance compared with system without knowledge inputs. 2. our multi-stage fine-tuning approach can help improve model performance in all the monolingual and code-mixed tracks. the approach can also reduce the training time to speed up our system building at different stages. 3. our iterative retrieval strategy can further improve the retrieval quality and result in significant improvement on the performance of codemixed track. 4. searching over wikipedia kb performs better than using online search engines on the multi-coner datasets. 5. comparing with other model variants we have tried, our ner model enjoys a good balance between model performance and speed. fine-tuning pretrained contextual embeddings is a useful and effective approach to many nlp tasks. recently, some of the research efforts propose to further train the fine-tuned embeddings with specific training data or in a larger model architecture to improve model performance. @xcite proposed two-stage fine-tuning, which first trains a general multilingual enhanced universal dependency @xcite parser and then finetunes on each specific language separately. @xcite proposed to train models through concatenating fine-tuned embeddings. we extend these ideas as multi-stage fine-tuning, which improves the accuracy of monolingual models that use finetuned multilingual embeddings as initialization in training. moreover, multi-stage fine-tuning can accelerate the training process in system building. ## our system we introduce how our knowledge-based ner system works in this section. given a sentence of n tokens @xmath0, • • • , x n }, the sentence is fed into our knowledge retrieval module. the knowledge retrieval module takes the sentence as the query and retrieves top-k related paragraphs in kb. ## results and analysis in this section, we use language codes foot_5 to represent languages, and use multi and mix to represent multilingual and code-mixed tracks respectively foot_6 . ## conclusion in this paper, we describe our knowledge-based system for the multiconer shared task, which wins 10 out of 13 tracks in the shared task. we construct multilingual kbs and retrieve the related documents from kbs to enhance the token representations of input text. we show that the ner models can use the retrieved knowledge to facilitate complex entity prediction, significantly improving both the in-domain and out-of-domain performance. multi-stage fine-tuning can help the monolingual models learn from the training data of all the languages and improve the model performance and training efficiency. we also show that the system presents a good balance between the model performance and prediction efficiency to meet the time requirement in the test phase. we believe this system can be widely applied to other domains for the task of ner. for future work, we plan to improve the retrieval quality and adopt the system to support other kinds of entity-related tasks.