USTC - NELSLIP at S em E val-2022 Task 11: Gazetteer-Adapted Integration Network for Multilingual Complex Named Entity Recognition (2022)

Abstract: This paper describes the system developed by the USTC-NELSLIP team for SemEval-2022 Task 11 Multilingual Complex Named Entity Recognition (MultiCoNER). We propose a gazetteer-adapted integration network (GAIN) to improve the performance of language models for recognizing complex named entities. The method first adapts the representations of gazetteer networks to those of language models by minimizing the KL divergence between them. After adaptation, these two networks are then integrated for backend supervised named entity recognition (NER) training. The proposed method is applied to several state-of-the-art Transformer-based NER models with a gazetteer built from Wikidata, and shows great generalization ability across them. The final predictions are derived from an ensemble of these trained models. Experimental results and detailed analysis verify the effectiveness of the proposed method. The official results show that our system ranked 1st on three tracks (Chinese, Code-mixed and Bangla) and 2nd on the other ten tracks in this task.

## introduction named entity recognition (ner) is a core natural language processing (nlp) task, which aims at finding entities and recognizing their type in a text sequence. in practical and open-domain settings, it is difficult for machines to process complex and ambiguous named entities @xcite . for example, "on the beach" is the title of a movie but cannot be recognized easily by present ner systems. this issue may become even more serious in multilingual or code-mixed settings @xcite . however, it has not received sufficient attention from the research community. to alleviate the issue, semeval-2022 task 11 @xcite formulates this task which focuses on detecting semantically ambiguous and complex entities in short and low-context settings for 11 languages. one of the classic approaches to solving this problem is to integrate external entity knowledge or gazetteers into neural architectures @xcite @xcite . typically, the two representations respectively from a language model like bert @xcite and a gazetteer network like bilstm @xcite ) are combined as one merged embedding, which is further fed into a ner classifier such as a conditional random field (crf) @xcite . however, there is a sense of "gap" between the two networks. the gazetteer network has no explicit semantic learning goal itself, which means it is just a more complex but almost isolated embedding layer for gazetteer information and cannot obtain the true meaning of ner tags actively. almost no semantic information can be gained by the classic gazetteer network. to effectively connect the two networks, a gazetteer-adapted integration network (gain) is proposed. the gain adopts a two-stage training strategy to adapt the gazetteer network to the language model. during the first training stage, the parameters of a language model are fixed. then a sentence and its annotation are fed into the two networks separately. the two outputs are adapted by minimizing the kl divergence between them. this training process helps the gazetteer network truly understand the meaning of ner tags by transferring semantic information from the pretrained language model to the gazetteer network, as the randomly initialized gazetteer network is gradually adapted to the pre-trained language model during the training. in the second stage, with maintaining the training process above, a gazetteer built from wikidata is applied to generate pseudo annotations searched by string matching. a sentence and the corresponding pseudo annotation are then fed into the two pre-trained networks separately. finally, integration methods like the concatenation or weighted summation are utilized on the two output representations for classifying. the proposed method achieves great improvements on the validation set @xcite of semeval-2022 task 11 compared to baseline models and ordinary ner systems with gazetteers. ensemble models are used for all thirteen tracks in the final test phase, and our system officially ranked 1st on three tracks (chinese, code-mixed and bangla), and 2nd on the other ten tracks among nearly 50 teams overall. the outstanding performance demonstrates the effectiveness of our method. fine-grained results show that our method significantly improves scores of difficult labels like "creative-work" and "product", which is the key challenge of this task. to facilitate the reproduction of our results, the code is available at https://github.com/mckysse/gain . ## task description semeval-2022 task 11 challenges participants to develop complex ner systems for 11 languages (english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, and bangla), focusing on recognizing semantically ambiguous and complex entities in short and lowcontext settings @xcite . each language constitutes a single track, while multilingual and code-mixed are added as track 12 and 13. the task adopts the wnut 2017 @xcite taxonomy entity types: person (per for short, names of people), loca-tion (loc, locations/physical facilities), cor-poration (corp, corporations and businesses), groups (grp, all other groups), product (prod, consumer products), and creative-work (cw, movie/song/book/etc. titles). the task also aims at testing the domain adaption capability of the systems by adding additional test sets on questions and short search queries. for each language, a training set with 15300 samples and a validation set with 800 samples are provided. for the code-mixed track, 1500 training samples and 500 validation samples are provided. the test data for each track have instances between 150k+ and 500k+ @xcite . ## system description this study focuses on making better use of the external entity knowledge. to describe our system clearly, in this section, we first introduce three basic mainstream ner systems used. then we show the process of constructing a gazetteer with wikidata, and how the gazetteer representation is generated and utilized. finally, we illustrate the gazetteer-adapted integration network (gain). the overall structure of the proposed system is shown in figure 1 . ## data preparation for the basic training set provided officially, an entity replacement strategy is adopted using our own gazetteer to construct a double data-augmented set. this part of data is called "data-wiki", which mainly consists of rich-context sentences. in order to improve the performance of our models on low-context instances, a set of annotated sentences are generated from the ms-marco qna corpus (v2.1) @xcite and the orcas dataset @xcite , which are mentioned in @xcite . our trained models and existing ner systems (e.g., spacy) are applied to identify entities in these corpora, and only templates identically recognized by all models are reserved. finally, 3753 english templates for ms-marco and 13806 english templates for orcas are obtained. after slotting the templates by our own gazetteer and translating them to the other 10 languages, we get approximately 16k annotated low-context sentences for each language. this part of data is called "data-query". a special operation is conducted for the codemixed track, because only 2000 annotated instances are provided officially. the multilingual function parameter value hidden size for language models 1024 for large, 768 for base learning rate for language models 2e-5 for large, 1e-5 for base learning rate for gazetteer networks 2e-4 for large, 1e-4 for base learning rate for the crf layer 2e-3 for large, 1e-3 for base first-stage training epochs 5 second-stage training epochs 20 batchsize 32 dropout rate 0.1 α for the second stage training 5 for softmax and span, 100 for crf optimizer adamw activation function relu model name lang xlm-r large @xcite multi chinese-roberta-wwm @xcite ) zh luke-large @xcite en klue-roberta @xcite ko bert-base-turkish @xcite tr bert-fa-base @xcite fa in wikidata is utilized, as every entity in the wiki page has several expressions in other languages. for every sentence in "data-wiki" and "data-query", the entities inside are randomly replaced with their translations recorded by wikidata. in this way, a set of annotated code-mixed data are built. ## analysis 6.1 effectiveness of the gazetteer to explore the effectiveness of the proposed gain methods, a large number of trials are conducted on the official data mentioned in section 2. all scores under the concatenation integration setting on the validation set are listed in ## conclusion this paper presents the implementation of the ustc-nelslip system submitted to the semeval-2022 task 11 multiconer. the gain method is proposed to adapt the gazetteer network to the language model, and achieves great improvements on the complex ner task. some construction methods for gazetteers and code-mixed data augmentation are also provided. in future works, we will keep exploring effective ways to integrate gazetteer networks with encoders to make better use of the external entity knowledge.