LLM - RM at S em E val-2023 Task 2: Multilingual Complex NER Using XLM - R o BERT a (2023)

Abstract: Named Entity Recognition(NER) is a task ofrecognizing entities at a token level in a sen-tence. This paper focuses on solving NER tasksin a multilingual setting for complex named en-tities. Our team, LLM-RM participated in therecently organized SemEval 2023 task, Task 2:MultiCoNER II,Multilingual Complex NamedEntity Recognition. We approach the problemby leveraging cross-lingual representation pro-vided by fine-tuning XLM-Roberta base modelon datasets of all of the 12 languages provided - Bangla, Chinese, English, Farsi, French,German, Hindi, Italian, Portuguese, Spanish,Swedish and Ukrainian.

## introduction named entity recognition(ner) is the task of recognizing entities (e.g., person, location, organization) in a piece of text. most of the ner datasets like conll2003 (erik f. @xcite are focussed on highresource languages like english and are specific to a given domain like news. the semeval-2023 task 2, multilingual complex named entity recognition (multiconer ii) @xcite contains ner datasets curated for 12 languages which are low resource and from other domains as well. the task also provided a large number of 30 classes and simulated errors being added to the test set to make it more challenging. it contains the following languages: english, spanish, dutch, russian, turkish, portugues, korean, farsi, german, chinese, hindi, and bangla. with the advent of deep learning based models, transformer models like bert, and bidirectional lstm based elmo became state of the art. bert's multilingual counterpart, mbert became state of the art in multilingual ner tasks. another model xlm-roberta(xlm-r) (alexis conneau and stoyanov, 2020) has shown to outperform mbert on low resource languages on the ner tasks. xlm-r is suitable for our task as it is pretrained for more than 100 languages including the 4 languages which we worked upon. therefore, we have used xlm-roberta-base transformer model and fine-tuned it on each of the given 12 languages. ## related work the earliest use of the neural networks to solve ner was by @xcite who attempted to use an lstm network to solve the problem. recent transformer networks like bert (jacob devlin and toutanova, 2018) and elmo (matthew e. peters and zettlemoyer, 2018) have further led to the state of the art results in the ner task for english language. crf layer @xcite was also proposed to be used as a final layer for token classification. to expand ner in a multilingual setting, (fetahu et al., 2021) introduced a multilingual and code-mixed dataset concerning the search domain. they also used an xlm-roberta model combined with gazetteers to build their multilingual ner classification system. in a paper for the first edition of this task in 2022, xlm-roberta has been applied to hindi and english dataset and have shown to perform better than mbert with a similar set of languages as in this current task (sumit singh and tiwary, 2022). ## data the dataset was first released in the semeval task in 2022, called multiconer dataset @xcite . in the 2nd edition of the task, the organisers provided a new dataset called multiconer v2 @xcite comprising of individual language datasets in 12 languages. the test dataset is used for the final evaluation of the leaderboard and is further split into corrupted and non-corrupted sets. it is to be noted that the uncorrupted test set size is quite large compared to the training set for all the languages. ## methodology we utilised xlm-roberta model for all 12 languages we participated in. xlm-roberta is a massive transformer trained on 100+ languages on 2tbs of commoncrawl data. figure 1 describes the architecture of transformer encoder models like roberta. it is trained with a multilingual mlm objective using only monolingual data. xlm-roberta has already been shown to outperform mbert on cross-lingual classification by upto 23 percent accuracy on lowresource languages. it also gave competitive results with respect to the state of the art monolingual models. entity name entity type facility, otherloc, humansettlement, station location visualwork, musicalwork, writtenwork, artwork, software creative work musicalgrp, publiccorp, privatecorp, org, aerospacemanufacturer, sportsgrp, carmanufacturer group scientist, artist, athlete, otherper politician, cleric, sportsmanager, person clothing, vehicle, food, drink, otherprod product medication/vaccine, medicalprocedure, anatomicalstructure, symptom, disease medical ## results from ## conclusion in this paper, we presented using xlm-robertabase to solve the shared task of multiconer. future work can include exploring more recent transformer-based models like xlm-v with very large vocabularies. also, data augmentation techniques like entity replacement can be tried.