CMNERO ne at S em E val-2022 Task 11: Code-Mixed Named Entity Recognition by leveraging multilingual data (2022)

Abstract: Identifying named entities is, in general, a practical and challenging task in the field of Natural Language Processing. Named Entity Recognition on the code-mixed text is further challenging due to the linguistic complexity resulting from the nature of the mixing. This paper addresses the submission of team CMNEROne to the SEMEVAL 2022 shared task 11 MultiCoNER. The Code-mixed NER task aimed to identify named entities on the code-mixed dataset. Our work consists of Named Entity Recognition (NER) on the code-mixed dataset by leveraging the multilingual data. We achieved a weighted average F1 score of 0.7044, i.e., 6% greater than the NER baseline.

## introduction named entity recognition (ner) is a fundamental task in nlp. it aims to identify and classify entities in a text into predefined types. it is an essential tool for information retrieval, question answering, @xcite ) and text summarization tasks @xcite . however, except for some resource-rich monolingual languages, ner annotated data for most other languages are still very limited @xcite . moreover, it is usually time-consuming to annotate such data, particularly for low-resource languages such as multilingual and code-mixed @xcite . therefore, transfer learning and leveraging the other datasets for multilingual and code-mixed ner has attracted growing interest recently, especially with the influx of deep learning methods. this paper presents the system description for named entity recognition on the code-mixed dataset. code-mixing is defined as using two or more languages in a single sentence or utterance @xcite . the use of codemixed language is prevalent in most multilingual societies. due to linguistic complexity arising sentence hamem this magic moment languge hi en en en ner tags o b-cw i-cw i-cw the lack of annotated data is a crucial issue for code-mixed datasets. lack of data poses a problem of data overfitting and poor entity recognition. the language models trained on such low resource datasets cannot generalize the training data, thus performing low on the test datasets. several previous studies have used monolingual data as training signals for transfer learning, and these data can also be used in the form of pre-training. thus, we used a similar approach of including the multilingual data along with the code-mixed dataset. we used the multilingual pre-trained bert model as our model for code-mixed ner. the model uses code mixed training data along with the multilingual training and mulilingual validation data. we have analyzed that bi-lstm with crf models has shown an improved accuracy on the token classification tasks such as pos tagging, language identification, and ner. the ensemble of bert or xlm-roberta with bi-lstm and crf would have shown a further improvement in the codemixed ner. also, using language identification as a downstream task with the current method might have improved the ner's accuracy. the paper is organized as follows. section 2 provides related work on named entity recognition on cm social media text. section 3 provides information on the task and examples of datasets. section 4 describes the proposed work. section 5 presents the experimental setup and section 6 project the performance of the model. section 7 concludes our work. ## related work code-mixed ner has attracted a lot of attention in the nlp community this decade. this section lists the latest works on code-mixed named entity recognition. priyadharshini et al. (2020) ( @xcite presented a novel cm ner model. they proposed a gated architecture that enhances existing multilingual transformers by dynamically infusing multilingual knowledge bases, a.k.a gazetteers. the evaluation of code-mixed queries shows that this approach efficiently utilizes gazetteers to recognize entities in code-mixed queries with an @xmath0, an absolute improvement of +31% over a non-gazetteer baseline. @xcite mentioned that including gazetteer features could cause models to overuse or underuse them, leading to poor generalization. they proposed a new approach for gazetteer knowledge integration by including context in gazetteer representation using encoder and mixture-of-experts gating network models. these models overcome the feature overuse issue by learning to conditionally combine the context and gazetteer features instead of assigning them fixed weights. ## task setup the shared task detects semantically ambiguous and complex entities in short and low-context settings. complex nes, like the titles of creative works (movie/book/song/software names), are not simple nouns. usually, they take imperative clauses, or they often resemble typical syntactic constituents. such nes are harder to recognize @xcite . syntactic parsing of such complex noun phrases is hard, and most ner systems fail to identify them. inside-outside-beginning (iob) format (ramshaw and marcus, 1999) is used for annotating entities. a few examples of complex nes and ambiguous nes from the code-mixed dataset are given in so the multiconer shared task encourages the models to handle such complex nes. a huge dataset @xcite is released for this task @xcite . the languages focused on in this shared task are: english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, and bangla. the shared task also offered an additional track with code-mixed and multilingual datasets. in this paper, we will be concentrating on the code-mixed dataset. ## system overview we finetuned the pre-trained multilingual bert model by using the multilingual training and validation datasets for code-mixed named entity recognition. we found that the training data is insufficient for the deep learning language model to identify the named entities in the validation data correctly. the data scarcity of low-resource languages has been a significant challenge for building nlp systems since they require a large amount of data to learn a robust model. we observed that the multilingual ner training data is similar to the code-mixed dataset. also, it is relatively large when compared to the code-mixed dataset. in our approach, the multilingual training and validation data is combined with the code-mixed training dataset. using the combined dataset, we finetune the deep neural network model. our method thus attempts to learn language-agnostic features by using the combined multilingual and code-mixed dataset. this finetuned model can be used to infer named entity information at a token level on a code-mixed low resource language. we used the pre-trained mbert @xcite model for code-mixed ner. mbert is a transformer-based self multi-headed attention model that is pre-trained on a massive collection of multilingual data and can be finetuned for our ner task. as the model is pre-trained on a large corpus, the semantic and syntactic information is well modeled and can be directly finetuned for a specific task. bert is a bi-directional transformer model @xcite . it analyzes the meaning of a term depending on its context given on both sides. the transformer part in the bert works like an attention mechanism capable of learning the contextual relationships between the terms in a sentence. ## experimental setup the section presents the baselines, hyper-parameter settings, and analysis of observed results. ## results and analysis our submission, the pre-trained mbert by leveraging the multilingual dataset, performed better than the multiconer baseline by 6%. even the multiconer baseline with the multilingual dataset performed better than our submission. the confusion matrices with and without multilingual data of our submission on the code-mixed ner validation dataset are shown in the figures ?? and 2. by using confusion matrices, we observed that the multilingual data given in figure 2 helped better identify the cw, prod, crop, and loc entities when compared to the baseline model. ## conclusion and future work in this paper, we addressed the shared task on named entity recognition for the code-mixed dataset. as the code-mixed data is a low resource language and there are no pre-trained models, we leveraged the multilingual dataset for training the ner model. the model used for testing our method is the pre-trained multilingual bert. we finetuned the pre-trained mbert for the codemixed ner task by using the code-mixed training data and multilingual training and validation datasets. the use of meta embeddings for dealing with code-mixed datasets has recently attracted a lot of attention. it might be possible that meta embedding-based ner will work better on this code-mixed dataset. unlike the social media data where code-mixed sentences/words are written in roman script, the native script is used for each word, so the language identification will work better on this dataset. using language identification or pos tagging as a downstream task for ner on this dataset will help in improving the code-mixed ner.