A alto NLP at S em E val-2022 Task 11: Ensembling Task-adaptive Pretrained Transformers for Multilingual Complex NER (2022)

Abstract: This paper presents the system description of team AaltoNLP for SemEval-2022 shared task 11: MultiCoNER. Transformer-based models have produced high scores on standard Named Entity Recognition (NER) tasks. However, accuracy on complex named entities is still low. Complex and ambiguous named entities have been identified as a major error source in NER tasks. The shared task is about multilingual complex named entity recognition. In this paper, we describe an ensemble approach, which increases accuracy across all tested languages. The system ensembles output from multiple same architecture task-adaptive pretrained transformers trained with different random seeds. We notice a large discrepancy between performance on development and test data. Model selection based on limited development data may not yield optimal results on large test data sets.

## introduction semeval-2022 shared task 11: multiconer @xcite was about complex named entity recognition (ner) in the multilingual context. transformer-based models such as bert @xcite have reached high accuracy in standard ner tasks. however, the models still struggle with complex and code-mixed named entities @xcite . the shared task tries to address the challenges regarding complex ner in a multilingual context. out of the 13 available tracks, we focus on 5 monolingual tracks. we build models for bangla, english, farsi, german, and korean. our strategy is to build an ensemble approach to increase accuracy compared to the baseline model. to tackle multilingualism, we build an approach that starts from the same xlm-roberta @xcite encoder and after fine-tuning, achieves good performance across languages. we propose two ensembling approaches: (1) naive ensembling and (2) end-to-end (e2e) ensembling. naive ensembling is a test-time ensembling where class scores from individually trained models are added together. e2e ensembling trains two encoder models jointly by concatenating their results and passing the concatenated predictions to the final layer. in addition to these strategies, we adapt the encoder model to data context using continued pretraining @xcite . we discover that naive ensembling produces good results with few models and by just using different random seeds for training. we also discover that performance can drastically vary between data sets and model selection based on development data may not yield good results on large test sets. our final rankings are 11th for farsi, 12th for bangla, 13th for german, 15th for korean, and 25th for english. for each language, our ranking is on the lower half of the participants. the code for our submission has been released foot_0 . ## background in ner, complex and ambiguous entities are hard to classify correctly. out of the 13 available tracks, we participated in the monolingual tracks for bangla, english, farsi, german, and korean. for training, we used only the competition data sets @xcite provided by the organizers. the task is to detect named entities in given sentences. figure 1 contains an example of the task setting. this example can be considered complex, as madagascar is ambiguous as it could refer to the country or the movie. transformer-based models @xcite ) such as bert @xcite and roberta @xcite produce state-of-the-art results on variety of natural language processing (nlp) tasks including ner. transformer models trained on multilingual data, such as xlm @xcite , can be used for cross-lingual tasks. xlm-roberta (xlm-r) @xcite combines the xlm and roberta architectures and produces state-of-the-art results across languages. xlm-r is used as the basis in our system, which we built on top of. @xcite showed that continuing pretraining of a transformer model with the domain or task-specific data often increases the performance. we utilize this finding and apply task-adaptive pretraining on multilingual models to increase performance in a specific language. ensemble learning is widely used in machine learning, in which, predictions from several different models are combined. the ensemble learning method combines multiple weak learners and provides a strong learner @xcite . this works especially well when the weak learners 2 are diverse and make different errors. @xcite showed that ensemble methods can significantly increase ner accuracy. combining transformer-based model predictions has been shown to increase ner accuracy @xcite @xcite . to keep applicability on languages with limited number of pretrained models, our approach is to form an ensemble model based on training the same model with different random seeds. ## system overview the motivation for our system is to produce an approach that can be applied across languages. initial testing with the baseline system provided by the organizers suggested that xlm-r large @xcite produces comparable or even better performance than tested monolingual pretrained models for targeted languages. therefore, we resort to building on top of a multilingual encoder model xlm-r large and the provided baseline system. inspired by results using task-adaptive pretraining @xcite , we start by training 2 also called base models task-adaptive language models. we hypothesize that task-adaptive pretraining could provide two sources of improvement: (1) adapting to the data domain and (2) improving the capability in the specific language. task-adaptive pretraining is conducted for the xlm-r large using the provided training data, which we prepare for pretraining by removing labels and constructing line-by-line sentences. the pretraining is applied to each language separately yielding us five encoder models named koala/xlm-roberta-large-xx. the xx is replaced with the corresponding language identifier. the pretraining is continued from the xlm-r large checkpoint with only the preprocessed task data. the pretraining objective remains the same. the models are available at the huggingface model repository foot_1 . in this paper, we refer to the models as koala-xlm-r large interchangeably for all languages. after obtaining the enhanced encoder models, we propose two ensemble learning approaches, i.e., naive ensemble and e2e ensemble. ## experimental setup our experimental setup was based on the provided baseline model @xcite and the data splits @xcite . models were trained only using the training data and the best model was selected based on performance on development data. after selecting the best model, we did not do any training with development data. for continued pretraining, we preprocessed the training data. the data were converted from conll format into lineby-line text. hence, we removed the labels and reconstructed the sentences into a single line of text data. we selected suitable hyperparameters based on performance on the development data. we selected two learning rates, first for the encoder model and a second one for all subsequent layers (decoder). we set the encoder learning rate to 10 -6 and the decoder learning rate to 10 -3 . to prevent overfitting, we used a dropout rate of 0.1. we also used early stopping with development data. the training was stopped when accuracy on development data had not increased in the last three epochs. we selected batch size based on the hardware constraints. for baselines and sub-models in the naive ensemble, we used batch size 64. for e2e ensemble, batch size is 20. the main python libraries we used are pytorch, pytorch lightning, allennlp, and transformers. further details available in the repository foot_2 . the performance in this shared task is evaluated using prediction accuracy on unseen test data set. the teams are ranked by their macro-averaged f1 classification score. ## results the official metrics were based on the evaluation phase with test data. the participants were ranked by their macro f1 scores. the performance of built models on test data is shown in we noticed a rather large discrepancy in accuracies between the development and test data set. the details of model performance on development data are presented in ## conclusion from our efforts, we conclude that naive ensembling improves accuracy with just four models of the same architecture trained with different random seeds. the ensemble of four models outperforms the best individual model across all the tested languages. the e2e ensemble model can provide good accuracy, but the results vary drastically between runs. task-adaptive pretraining, which has in some cases improved accuracy significantly, yielded only a slight improvement in this task. we also notice a large difference between performance in development and test data. with such limited development data and a large test set, model selection on solely development data yields sub-optimal results. more attention should be paid to model selection and model's generalizability.