N amed E ntity R angers at S em E val-2022 Task 11: Transformer-based Approaches for Multilingual Complex Named Entity Recognition (2022)

Abstract: This paper presents the two submissions of NamedEntityRangers Team to the MultiCoNER Shared Task, hosted at SemEval-2022. We evaluate two state-of-the-art approaches, of which both utilize pre-trained multi-lingual language models differently. The first approach follows the token classification schema, in which each token is assigned with a tag. The second approach follows a recent template-free paradigm, in which an encoder-decoder model translates the input sequence of words to a special output, encoding named entities with predefined labels. We utilize RemBERT and mT5 as backbone models for these two approaches, respectively. Our results show that the oldie but goodie token classification outperforms the template-free method by a wide margin. Our code is available at: https://github.com/Abiks/MultiCoNER .

## introduction this paper describes two submissions to the multilingual complex named entity recognition (mul-ticoner) shared task, held by semeval-2022 @xcite . this shared task aims at recognizing named entities with the ambitious end goal of building systems that support up to 11 languages. multilingual setups are complicated when the dataset mixes languages from different groups. this way, the multiconer dataset comprises 11 languages from different families, and multiple scripts @xcite . this setup hayj s become increasingly popular recently since it allows to test for transfer learning across languages within a single pre-trained model. the dataset, proposed in the shared task, has several unique features previously neglected in ner evaluation: syntactically complex entities, ambiguous entities, divers, and low-frequency (aka long-tail) entities. this makes the shared task setup closer to real-life settings, where datasets are way noisier and nonuniformly distributed. our solution consists of two state-of-the-art approaches adopted to the task. first, we use a mainstream ner technique, token classification. under this approach, the model is trained to assign a label to each input token. the second approach falls into the group of prompt-based techniques, at the core of which are the capabilities of auto-regressive language models to memorize and reproduce input texts. in this case, we train an encoder-decoder model to replace entities in the input sentence with predefined labels. we utilize @xcite and mt5 @xcite as backbone models for these two approaches, respectively. these two models provide state-of-the-art results for the common test-beds of cross-lingual experiments @xcite . in other words, we explore the following questions: 'how do pre-trained transformer-based models perform in the multilingual complex named entity recognition task?' and 'which of the two approaches perform better?'. our results show that plain fine-tuning of the above-mentioned state-of-the-art multilingual transformer-based models can give moderate re-sults. results are within 10% margin of the top solution for the multilingual ner task. we analyzed the errors of both models. as expected, for two novel 'complex' entity types ('creative work' and 'product') error rate is higher than for 'common' types ('group', 'location', 'person'). however, some of the errors might be caused by inconsistencies in the labeling of the multiconer dataset. the performance of the template-free approach suffers from such inconsistencies more than the performance of the token classification approach. ## related work named entity recognition is one of the central tasks in natural language processing, which attracts a lot of research efforts. @xcite encode morphology and context information via character and word embeddings. recent studies @xcite @xcite employ syntactic dependencies, lexical similarity, gazetteers, etc. in the word representations before feeding them to context encoding layers. the authors show that additional information may lead to improvements in ner performance. however, ner still faces multiple challenges @xcite such as detection of fine-grained and nested named entities @xcite @xcite , ner in domain-specific areas @xcite , ner from noisy data @xcite and code-mixed data @xcite . recent research in this area considers not only standard types of entities (person, location, organization) but also semantically ambiguous and complex entities @xcite . for example, a system has to recognize the titles of movies, books, or songs, which may contain verbs, adverbs, prepositions, etc. @xcite propose a template-based method, treating ner as a language model ranking problem in a seq2seq manner. original sentences and statement templates filled by a candidate named entity span are the source sequence and the target sequence, respectively. ma et al. (2021 ) induces a language model to predict label words at entity positions during fine-tuning. this method demonstrates the effectiveness under the few-shot setting. the semeval-2020 shared task multiconer @xcite focuses on a more exciting and challenging problem of building a ner sys-tem for multiple languages. the results of a recent multilingual named entity challenge in six slavic languages @xcite have also confirmed the complexity and significance of the task. training competitive multilingual ner systems requires either manually labeled text collections or large automatically annotated datasets @xcite . ## discussion we provide the performance analysis of the models that we developed. the confusion matrices for both approaches demonstrate similar commonly occurring errors. moreover, while working on experiments, we noticed several dataset issues which we suppose are worth mentioning. the problems potentially contribute to the low performance of the template-free approach. ## conclusion in the paper, we implemented and evaluated two straightforward yet different approaches to the multilingual ner subtask with complex types of entities (multiconer). the first approach uses a state-of-the-art transformer-based model and finetuning for token classification, while the second one applies a template-free information extraction paradigm. our results are within the 10% margin of the top solution for multilingual ner tasks and much higher than organizers' baseline performance. therefore, we can conclude that out-of-the-box approaches generalize quite well for complex ner tasks and provide a viable alternative. the performance of the template-free approach can suffer from inconsistent annotation. the second finding is the relatively low performance of an ensemble model, but this issue needs further investigation. evaluation of systems for multilingual ner with complex entity types is still challenging. our study analyzed the dataset and found several issues that can be addressed in future versions of multiconer track or in similar competitions.