USTC - NELSLIP at S em E val-2023 Task 2: Statistical Construction and Dual Adaptation of Gazetteer for Multilingual Complex NER (2023)

Abstract: This paper describes the system developed by the USTC-NELSLIP team for SemEval-2023 Task 2 Multilingual Complex Named Entity Recognition (MultiCoNER II). We propose a method named Statistical Construction and Dual Adaptation of Gazetteer (SCDAG) for Multilingual Complex NER. The method first utilizes a statistics-based approach to construct a gazetteer. Secondly, the representations of gazetteer networks and language models are adapted by minimizing the KL divergence between them at the sentence-level and entity-level. Finally, these two networks are then integrated for supervised named entity recognition (NER) training. The proposed method is applied to several state-of-the-art Transformer-based NER models with a gazetteer built from Wikidata, and shows great generalization ability across them. The final predictions are derived from an ensemble of these trained models. Experimental results and detailed analysis verify the effectiveness of the proposed method. The official results show that our system ranked 1st on one track (Hindi) in this task.

## introduction named entity recognition (ner) is a fundamental and important natural language processing (nlp) task, which aims at finding entities and recognizing their type in a text sequence. recently, deep neural networks have achieved great performance on simple ner with abundant labeled data @xcite @xcite . in practical and open-domain settings, it is difficult for machines to process complex and fine-grained named entities @xcite . for example, "the old man and the sea" is the title of a movie as well as a book, which has different categories in different contexts and cannot be recognized easily by present ner systems. this issue may become even more serious in multilingual settings @xcite . however, it has not received sufficient attention from the research community. to alleviate the issue, semeval-2023 task 2 @xcite formulates this task which challenges participants to develop ner systems for 12 languages (english, spanish, swedish, ukrainian, portuguese, french, farsi, german, chinese, hindi, bangla and italian), focusing on recognizing semantically complex and fine-grained entities in short and low-context settings. each language constitutes a single track, while multilingual is added as track 13. the datasets @xcite ) mainly contain sentences from three domains: wikipedia, web questions and search queries. besides, simulated errors are added to the test set to make the task more realistic and difficult. recent studies have found that integrating external knowledge or gazetteers into neural architectures is effective in solving this problem @xcite @xcite . for example, the two representations respectively from a language model and a gazetteer network are integrated as one representation, which is further fed into a classifier such as a conditional random field (crf) @xcite . however, for the fine-grained entities, due to the closer semantic distance between these entity categories, the coverage rate of the constructed entity gazetteer is difficult to improve. besides, the interaction between the two representations in previous work @xcite only focus on sentence-level, which ignore the entity-level representation gap between them and lead to information loss. in this paper, we propose a method named statistical construction and dual adaptation of gazetteer (scdag) for multilingual complex ner based on gain @xcite . firstly, based on wikipedia of the 12 languages, we build a multilingual gazetteer to search for the entities in input sentence. different from gain, we use a statistics-based approach to maximize the coverage of the gazetteer. afterwards, the scdag adopts a two-stage training strategy to dually adapt the gazetteer network to the language model. during the first training stage, the parameters of a language model are fixed. then a sentence and its annotation are fed into the two networks separately. the representations of gazetteer networks and language models are adapted by minimizing the kl divergence between them at the sentence-level and entity-level. this process helps the gazetteer network understand the meaning of ner tags and strengthen the model adaptation ability to ner. a gazetteer is applied to sentences to generate pseudo tags which are fed into the two pre-trained networks separately in the second stage. finally, the two output representations are integrated for classifying. the proposed method achieves great improvements on the validation set @xcite of semeval-2023 task 2 compared to baseline models with gazetteers. ensemble models are used for all thirteen tracks in the final test phase, and our system officially ranked 1st on one track (hindi). the outstanding performance demonstrates the effectiveness of our method. to facilitate the reproduction of our results, the code is available at https://github.com/mjy1111/scdag . ## related work ner has a lot of applications in various domains and languages. recently, with the introduction of contextual pre-trained models, such as bert, roberta @xcite and xlm-r @xcite , the performance of ner systems has been significantly improved. these models are trained on large-scale unlabeled data such as wikipedia, which can significantly improve the contextual representations abilities. semeval-2023 task 2 is a continuation of the multilingual ner task started in 2022 @xcite . there are many challenges that can make ner extremely difficult. in @xcite , they explain that named entity recognition is especially difficult in situations with low-context or in scenarios where the named entities are exceptionally complex and ambiguous. another work has extended this to multilingual and codemixed settings @xcite . these are the key challenges of the 2022 datasets @xcite and we have participated in this competition @xcite . besides, ner requires abundant well-annotated data, which is too expensive in low-resource languages @xcite . lots of methods are proposed to improve ner performance and a general discovery is that the use of external knowledge bases is very effective. @xcite retrieve related contexts from a search engine as external contexts of the inputs to take advantage of long-range dependencies for entity disambiguation and successfully achieve stateof-the-art performance across multiple datasets. @xcite recognize the importance of gazetteer resources, even in the case of state-ofthe-art systems making use of pre-trained models. they propose a contextual gazetteer representation encoder, combined with a novel mixtureof-expert (moe) gating network to conditionally utilize gazetteer information. @xcite employ multilingual gazetteers embedded with transformer models in an moe approach to improve the recognition of entities in code-mixed web queries, where entities are in a different language from the rest of the query. ## data the multiconer dataset @xcite is provided in a column-based format and divided into the training, development and testing sets. the text is lowercase with the named entity annotation in bio format @xcite . the first token (or the single token) of an entity contains the "b-" prefix. other entity tokens (in the case of multi-token entities) start with an "i-" prefix, while non-entity tokens are denoted with "o". it consists of 6 coarse-grained entity types and 33 fine-grained entity types, and the coarse to fine level mapping of the tags is as follows: location (facility, otherloc, humansettlement, station), creative work (visualwork, musicalwork, writ-tenwork, artwork, software), group (musical-grp, publiccorp, privatecorp, aerospacemanufacturer, sportsgrp, carmanufacturer, org), person (scientist, artist, athlete, politician, cleric, sportsmanager, otherper), product (clothing, vehicle, food, drink, otherprod), medical (medication/vaccine, medicalprocedure, anatomical-structure, symptom, disease). two data augmentation methods are used following @xcite . for the basic training set provided officially, an entity replacement strategy is adopted using our own gazetteer to construct a data-augmented set. this part of data is called "data-wiki", which mainly consists of rich-context sentences. in order to improve the performance of our models on low-context instances, a set of annotated sentences are generated from the ms-marco qna corpus (v2.1) @xcite and the orcas dataset @xcite , which are mentioned in @xcite . our trained models and existing ner systems (e.g., spacy) are applied to identify entities in these corpora, and only templates identically recognized by all models are reserved. then 3,379 english templates for ms-marco and 11,754 english templates for orcas are obtained. next, we slot the templates by our own gazetteer and translate them to the other 11 languages. this part of data is called "data-query". finally, the constructed data is used together with the official data for training. ## methodology this study focuses on making better use of external entity knowledge. to describe our system clearly, in this section, we first introduce three basic mainstream ner systems used. then we show the process of constructing a gazetteer from wikipedia using a statistics-based approach to maximize the coverage rate, and how the gazetteer representation is generated and utilized. finally, we illustrate the dual adaptation between gazetteer network and language model. the overall structure of the proposed system is shown in figure 1 . ## conclusion this paper presents the implementation of the ustc-nelslip system submitted to the semeval-2023 task 2 multiconer ii. different from multiconer i, it has a fine-grained taxonomy which greatly increased the difficulty of the task. the scdag method is proposed to statistically construct gazetteer and dually adapt the gazetteer network to the language model, achieving great improvements on the fine-grained ner task. some construction methods for gazetteers and augment data are also provided. in future works, we will improve the gazetteer quality and apply this method to more tasks.