S eq L at S em E val-2022 Task 11: An Ensemble of Transformer Based Models for Complex Named Entity Recognition Task (2022)

Abstract: This paper presents our system used to participate in task 11 (MultiCONER) of the SemEval 2022 competition. Our system ranked fourth place in track 12 (Multilingual) and fifth place in track 13 (Code-Mixed). The goal of track 12 is to detect complex named entities in a multilingual setting, while track 13 is dedicated to detecting complex named entities in a code-mixed setting. Both systems were developed using transformer-based language models. We used an ensemble of XLM-RoBERTa-large and Microsoft/infoxlm-large with a Conditional Random Field (CRF) layer. In addition, we describe the algorithms employed to train our models and our hyper-parameter selection. We furthermore study the impact of different methods to aggregate the outputs of the individual models that compose our ensemble. Finally, we present an extensive analysis of the results and errors.

## introduction named entity recognition (ner) is the task of identifying proper names in a text and categorizing them into predefined entity types such as person (per), location (loc), or creative work (cw). for example, given a sentence "michael jeffrey jordan was born in brooklyn, new york" the goal is to label entities correctly with their corresponding category using the bio-scheme: existing systems are often trained on standard news text and strongly rely on surface form features such as capitalization and punctuation. these approaches do not scale well to usergenerated content because of the increased variation in language and context in an ever-expanding domain @xcite @xcite . examples of challenging scenarios for named entity recognition are: (a) entities in very short text inputs such as search queries with limited or no context @xcite structurally complex entities such as movie or book titles ranging from complex noun phrases to full clauses (c) recognizing named entities in dynamically evolving contexts in which novel entities emerge @xcite . although many named entities are shared between languages, named entity recognition systems usually rely on language-specific cues (e.g. capitalization of nouns in german, compounding phenomena in agglutinative languages such as korean, japanese and turkish @xcite ). such kind of detection cues do not scale well to other languages. as a consequence, most models need to be fine-tuned for each language separately on manually annotated high-quality training data which is a costly process. task 12 of semeval 2022 provides a test bench for more robust systems which can detect complex named entities in 11 languages @xcite . the dataset intentionally contains semantically ambiguous entities with limited contexts. we focus on the multilingual tracks of the competition which requires the prediction of named entities in all 11 languages by a single model (track 12). as an additional challenge, the model is also evaluated on code-mixed data (track 13). we summarize the main finding of our analysis as follows: in the rest of the paper, we discuss related work, experimental setup and model training, and extensive error analysis. ## related work named entity recognition can be modeled as a sequence labeling problem. deep-learning based approaches learn suitable representations in an end-to-end fashion and outperform rule-based and handcrafted feature-based approaches @xcite . @xcite proposed a bilstm-crf architecture for sequence tagging which is used by most state-of-the-art models. these models combine long short-term memory layers in a bidirectional fashion to use both past and future input and predict named entity sequences using a conditional random field layer. significant performance gains have been obtained by initializing the model with pre-trained contextual embedding models such as bert @xcite , flair @xcite and luke @xcite . subsequent works explore some of the limitation of using a vanilla transformer. @xcite show that a transformer architecture is less effective for modeling sequence labeling that strongly relies on left and right context and longrange dependencies which is the case for named entity recognition. luke @xcite is the state of the art in the conll-03 ner dataset. it is pre-trained by contextualized representations based on bi-directional transformers on entity-annotated corpus of words and entities. the complexity of the task and its multilingual nature are the main factors in choosing our modeling approach. the task complexity entails that our approach should rely on token context and in capturing relationships between labels since the surface form cues (e.g capitalization ) are normalized in the training data. the multilingual aspect entails choosing a crosslingual pre-trained model which can handle the target languages. we choose xlm-roberta-large @xcite and microsoft/infoxlm-large @xcite . xlm-roberta-large model is a cross-lingual version of roberta. xlm-roberta has outperformed cross-lingual bert and it is the state of the art on many crosslingual tasks including named entity recognition. microsoft/infoxlm-large is similarly a multilingual pre-trained model for over 100 languages with a new cross-lingual pre-training task named cross-lingual contrast (xlco). a comparison of the two pre-trained models shows both to be competitive on tasks such as cross-lingual natural language inference (xnli) and microsoft/infoxlm-large to be significantly better on cross-lingual question answering (mlqa) and cross-lingual sentence retrieval on the tatoeba dataset. we provide a direct comparison of these two models for named entity recognition (which was previously missing in the literature) and explore an ensemble of the two models. ## data analysis we use the training dataset provided as part of se-meval 2022 task 11 multiconer: track 12 (multilingual) and track 13 (code-mixed) @xcite . the dataset consists of training and development data in 11 languages annotated with six named entity types. ## system description the system that we proposed for both track 12 and track 13 is based on an ensemble of two pre-trained transformer models (ptms). the first one is the xlm-roberta-large model @xcite which is a cross-lingual version of roberta.the second one is microsoft/infoxlm-large @xcite which is also multilingual pre-trained model that supports over 100 languages and includes a new cross-lingual pre-training. ## experiment and result all our models were implemented with pytorch @xcite , on top of the pre-trained transformer models provided by huggingface @xcite . for the ptms, the output of the last attention layer was used as input for the classifier layer. the crf classifier was implemented using the allennlp library @xcite . adam optimizer (kingma and ba, 2014) was used to update model parameters. finally, cosine annealing decay with t _max = 20 and @xmath0.0e-8 was applied for the learning rate after (early_stopping_patience / 2) consecutive epochs without improving @xcite . sentence majority voting (hard voting) crf (soft voting) labels scores labels scores labels scores scores labels labels figure 2: voting ## analysis and conclusion in this section, we analyze our model output for the multilingual task and the code-mixed task on the development dataset because the gold labels for the test data were not released. we see that the model performance varies strongly between languages. the best result is obtained for german and is 10 percentage points higher than the lowest result which is obtained for russian. when we compare the different entity types, we observe the highest variance for chinese. the large differences are surprising as the training data size is equal for all languages and the entity types are roughly evenly distributed (with the exception of chinese). we therefore analyse these differences further below.