UM 6 P - CS at S em E val-2022 Task 11: Enhancing Multilingual and Code-Mixed Complex Named Entity Recognition via Pseudo Labels using Multilingual Transformer (2022)

Abstract: Building real-world complex Named Entity Recognition (NER) systems is a challenging task. This is due to the complexity and ambiguity of named entities that appear in various contexts such as short input sentences, emerging entities, and complex entities. Besides, real-world queries are mostly malformed, as they can be code-mixed or multilingual, among other scenarios. In this paper, we introduce our submitted system to the Multilingual Complex Named Entity Recognition (MultiCoNER) shared task. We approach the complex NER for multilingual and code-mixed queries, by relying on the contextualized representation provided by the multilingual Transformer XLM-RoBERTa. In addition to the CRF-based token classification layer, we incorporate a span classification loss to recognize named entities spans. Furthermore, we use a self-training mechanism to generate weakly-annotated data from a large unlabeled dataset. Our proposed system is ranked 6th and 8th in the multilingual and code-mixed MultiCoNER’s tracks respectively.

## introduction recent named entity recognition (ner) models have achieved great performance for many languages and using various benchmark datasets such as conll2003 and ontonotes 5.0 @xcite . however, it is unclear whether or not these systems can handle ambiguous and complex entities, especially in the case of short and low-context settings @xcite . it is also unclear weather these systems can be deployed in real-world scenarios where the input data can be in different languages or code-mixed @xcite . in fact, to illustrate these issues, if consider the example of complex named entities such as the titles of creative works (movies, songs, books ...), they are hard to be recognized by simple ner systems. this is due to their syntactic ambiguity and the form they can take from one context to another. for instance, they can be as an imperative clause ("dial m for murder") or a proposition ("on the beach") which refers to the name of a movie. thus, it is important to check the performance of ner systems in these scenarios. the complexity of named entities can be due to three main reasons: it is well known that the state-of-the-art performance reached by current ner systems is mainly due to the presence of easy entities and well-formed input texts @xcite . nevertheless, they yield weak performance when applied in multilingual/code-switched input texts having complex or unseen entities. it is also worth mentioning that the reached state-of-the-art results by current ner systems have been achieved using transformers-based pre-trained language models @xcite that are known to encode the context of input texts and tokens. in this paper, we introduce our participating system to the multiconer shared task @xcite , in particular, the complex multiconer tracks of multilingual and code-switched queries. our system relies on a deep learning model based on the multilingual transformer-based pre-trained language model xlm-roberta @xcite . to handle the complexity of the two tracks, our model is trained using two optimization objectives, as well as for self-training. the main components of our system can be summarized as follow: we incorporate a crf layer on top of the transformer representations to fully exploit the mutual information between tokens in an input sentence. the rest of this paper is organized as follows. section 2 presents the related work. section 3 describes the dataset and the sub-tasks of semeval-2022 task 11 @xcite . in section 4, we present our system overview. section 5 summarizes and discusses the obtained results for both multilingual and code-mixed tracks. finally, section 5 concludes the paper. ## related work during the last years, neural network-based approaches have contributed to improving the performance of ner systems @xcite . this was mainly achieved thanks to word embeddings. static word embeddings are fed to bilstm-crf models and have helped eliminate manual feature engineering while achieving better performance. on the other hand, transformer-based language models @xcite have greatly improved the ner results thanks to their contextualized word representations. however, these models may fail when recognizing new or complex entities @xcite . these challenges are reflecting the real-world setting. recently, @xcite have proposed an approach to tackle these challenges by using a contextual gazetteer representation encoder which can be fused with word-level models. the results have shown that this method enhances the f1-score by +49% in the uncased setting. this work has been mainly applied to the english language. finally, in another work, @xcite have explored the code-mixed ner scenario using multilingual transformers. they have combined the mixtureof-experts model with existing multilingual transformers models to incorporate the multi-lingual gazetteers. the experiments have demonstrated that their proposed approach enhances the f1-score by +31% in the code-mixed ner over the baseline model. ## data the dataset of the multiconer (multilingual complex named entity recognition) shared task (malmasi et al., 2022a) is provided to tackle thirteen different tracks, eleven tracks cover the monolingual cases, while the remaining two tracks cover code-mixed and multilingual data. in this paper, we focus on the last two tracks. the multilingual track covers eleven languages (english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, and bangla), while the code-mixed track covers a subset of these languages. in both tracks, the entities are annotated into six types: per, loc, grp, corp, prod, and cw. the provided datasets are labeled using the iob format which is used for sequence labeling tasks. ## methodology before discussing the methodology of our proposed system, we assume that we have n labeled sentences with named-entities @xmath0, y t i )} n 1 and m unlabeled sentences @xmath1, where x @xmath2,1 , x i,2 , ..., x i,l } represents a sentence containing l tokens, while y @xmath3,1 , y i,2 , ..., y i,l } are the l labels corresponding to these l tokens. the proposed system incorporates 4 components used on top of the pre-trained transformer encoder. in the following, we describe each component of our system. ## conclusion in this paper, we present our named entity recognition (ner) system for complex scenarios on multilingual and code-mixed queries. our system relies on 4 components: a multilingual transformer en-coder, an entity span classification module, a crflayer, and a self-training mechanism that leverages information from unlabeled data. we use our system to submit our predictions in the semeval-2022 task 11 within the multilingual and code-mixed tracks. the results show that the use of multilingual transformer and self-training enhances the results in both multilingual and code-mixed cases. moreover, the incorporation of the span classification module and the crf layer allow better recognition of complex named entities.