Qtrade AI at S em E val-2022 Task 11: An Unified Framework for Multilingual NER Task (2022)

Abstract: This paper describes our system, which placed third in the Multilingual Track (subtask 11), fourth in the Code-Mixed Track (subtask 12), and seventh in the Chinese Track (subtask 9) in the SemEval 2022 Task 11: MultiCoNER Multilingual Complex Named Entity Recognition. Our system’s key contributions are as follows: 1) For multilingual NER tasks, we offered a unified framework with which one can easily execute single-language or multilingual NER tasks, 2) for low-resource mixed-code NER task, one can easily enhanced his or her dataset through implementing several simple data augmentation methods and 3) for Chinese tasks, we proposed a model that can capture Chinese lexical semantic, lexical border, and lexical graph structural information. Finally, in the test phase, our system received macro-f1 scores of 77.66, 84.35, and 74 on task 12, task 13, and task 9.

## introduction semeval 2022 task 11: multiconer multilingual complex named entity recognition @xcite focuses on extracting semantically ambiguous complex named entities @xcite in short, low-context and code-mixed @xcite scenarios. the domain adaptability capacity of the system is in high demand for this shared task, which contains 13 tracks in english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, bangla, multilanguage, and code-mixed. among them, multilanguage, and code-mixed tracks are in the mixed language, while the other tracks are in the single language. the difference between multi-language track and code-mixed track is that the corpus of multi-language track is multilingual, but the words in each sentence are in a single language, while the corpus of code-mixed track is a corpus in which the * these authors contributed equally to this work. words or phrases in each sentence may be from different languages, for example the sentence i liebe sie (i love you) consists of english, french and german. furthermore, the datasets offered by the organizers , which may mainly from wikipedia, web questions and user queries, comprise data from 11 languages, with each language containing around 15,000 training samples and 800 development (dev) samples @xcite . the corpus contains a total of six entity categories, namely location (loc), person (per), product (prod), group (grp), corporation (corp) and creative works (cw). the main contributions of our system are as follows: 1) we propose a unified framework for multilingual ner tasks, using which one can easily perform monolingual or multilingual ner tasks; 2) for low-resource code-mixed ner tasks, we provide several simple and effective data augmentation methods to easily increase the amount of data; 3) for chinese tasks, we propose a model that captures chinese lexical semantics, lexical boundaries and lexical graph structure information in a model. ## related work named entity recognition (ner)-a classic and fundamental task that aims to extract named entities from a sentence-plays an important role in a variety of downstream tasks in the field of nlp, including relation extraction @xcite , knowledge graph construction @xcite , question answering @xcite and so on. for a long time, the development of ner was slow, especially before the rise of neural networks, and ner mostly used statistical machine learning methods like hmm @xcite and crf @xcite . although these methods were effective at the time, they were still stretched for complex scenarios. after the rise of neural networks, especially structured networks such as rnns @xcite and cnns @xcite , ner has been greatly developed, and the accuracy has been greatly improved compared with traditional statistical learning methods. there are also methods that combine neural networks with traditional statistical learning methods, such as bilstm+crf @xcite . although these methods take ner to another higher level, they also have certain problems. for example, rnns cannot model bi-directionally context, and then there is the inability of rnns to capture long-term dependencies, and though cnns can model bi-directionally, but due to the size of convolutional kernels, they can only model local contextual information at the same time and cannot capture global contextual information. recently, self-training pre-training models on large-scale corpus such as bert(kenton and toutanova) and its variant versions have greatly improved the accuracy of ner tasks and effectively solved the problem that rnns cannot capture long-range dependencies as well as dynamic contexts. from statistical models like hmm and crf to deep learning models like cnn, rnn, and transformer-based pre-trained models, the accuracy of ner tasks is increasing and has reached commercial levels in many scenarios @xcite . despite its remarkable progress, ner still faces some challenges @xcite , such as the problem of discontinued and nested named entities @xcite , the challenge of cross-domain and cross-lingual transfer learning @xcite , the lack of lexicon information when using sub-character tokenization strategy and word boundary information of chinese ner tasks @xcite , and the ambiguity of named entities @xcite under different semantic circumstance. ## experimental setup in this session, we are going to describe the implementation details of our system. dataset and word embeddings: train and dev datasets we used are provided by official and we don't use test dataset before testing phase because it was not available. besides, experiments at model ensemble stage and pseudo label fine-tuning stage are carried on test dataset.since leroberta and gcn4ner techniques require word embedding and considering the lack of embedding of relevant words in the pre-trained model xlm-roberta, we use word embedding sgns.merge.word † trained by skip-gram. processing and hyper parameters: limited by the length of the paper, we will only briefly introduce our main processing and part of hyper parameters, with which our system can reach the best result. learning rate was set to 1×10 -6 , warming up proportion to 0.06, drop out rate to 0.2, batch size † https://drive.google.com/file/d/1zh9zceu8_esqqkyvqufqdnkpc4mtekr/view. to 32, epoch to 30, random seed to 42 and the number of voting models to 7. augmentation method for full data fine-tuning stage was mulsent-uni, decoder strategy was softmax and adversarial training method was pgd with ϵ = 1.0, α = 0.1, @xmath0. for more details, please check the appendix a. through out our system we almost only use softmax as our decoder strategy because we have found crf has no effect on improving the scores compared to softmax but consumes more computing resources and storage space. so, we drop crf strategy. one plausible explanation for this is that pretrained models have already caught the relations between tokens which may not be well captured by non-pretrained models. we selected the best epoch and the best hyper parameters using performance (measured in terms of macro-f1 score) on corresponding dev dataset. ## results in this section, we will report our main experiment and provide an analysis of the results. unless otherwise specified, the hyper parameters in our experiments are configured in accordance with section 3. full data fine-tuning stage: as we can see from chinese (zh) track fine-tuning: as for chinese (zh) track, we employ leroberta, gcn4ner and leroberta-gcn4ner for finetuning. gcn4ner and leroberta-gcn4ner have a positive effect on improving macro-f1 score. we can prove our hypothesis that leroberta only integrates the information of words from the bottom encoding layer, and that it does not make full use of the graph relation of containing, transition, and lattice between words and characters, by observing the differences in results between our modified models. as a result, leroberta just climbs by 0.18 percent. similar to gcn4ner, it may just consider character graph relationships while ignoring lexical semantic information. so, gcn4ner increases by 0.32 percent. when these two approaches are combined, we discover that leroberta-gcn4ner improves by 1%, giving it the highest score on the dev dataset in our system. multilingual track fine-tuning: this experiment continually use the best checkpoint from full data fine-tuning stage . we try r-drop technique and private feature extractor and shared feature extractor (pfe-sfe) method to improve the performance of our model on this track. in the r-drop experiment, we use the average method for the kl divergence, and set coefficient parameter α to 0.1 and it obtains a marco-f1 score of 87.15 on the dev dataset which means that this method has a good effect on multilingual tasks. what's more, we adopt pfe-sfe method and the marco-f1 score achieve 87.21 in our experiment. it can be seen from mix-code track fine-tuning: as for mix-code track, we used mention replacement (mr) and shuffle within segments (sis) techniques to resolve the insufficiency issue of mix-code data and increase the generalization ability of our model on this track. we not only utilize the parameter setting at full data fine-tuning stage, but also employ the best checkpoint at full data fine-tuning stage. as shown in model ensemble stage: at this stage, test data is available and we choose the top-7 models based on marco-f1 score under 7 sets of different parameters to vote on the final results. for details, please check the appendix b. the second row in next, same as model ensemble stage we choose 7 sets of different parameters to train on the pseudo label dataset and get 7 sets of checkpoints. for more details about these models, please check the appendix a. the results are shown in ## conclusion in this paper, we have introduce our system step by step which can be regarded as an universal framework for multilingual ner task. besides, we proposed a graph-based model to make up for the lack of understanding capacity of chinese lexicon. adequate ablation experiments shows that our methods work for this task. in future efforts, we plan to further improve our system to better handle polysemous scenarios.