UC 3 M - PUCPR at S em E val-2022 Task 11: An Ensemble Method of Transformer-based Models for Complex Named Entity Recognition (2022)

Abstract: This study introduces the system submitted to the SemEval 2022 Task 11: MultiCoNER (Multilingual Complex Named Entity Recognition) by the UC3M-PUCPR team. We proposed an ensemble of transformer-based models for entity recognition in cross-domain texts. Our deep learning method benefits from the transformer architecture, which adopts the attention mechanism to handle the long-range dependencies of the input text. Also, the ensemble approach for named entity recognition (NER) improved the results over baselines based on individual models on two of the three tracks we participated in. The ensemble model for the code-mixed task achieves an overall performance of 76.36% F1-score, a 2.85 percentage point increase upon our individually best model for this task, XLM-RoBERTa-large (73.51%), outperforming the baseline provided for the shared task by 18.26 points. Our preliminary results suggest that contextualized language models ensembles can, even if modestly, improve the results in extracting information from unstructured data.

## introduction named entity recognition (ner) is a natural language processing (nlp) technique to extract relevant information from unstructured natural language data, identifying and categorizing entities in texts and thereby supporting other nlp tasks. however, processing complex and ambiguous entities is a challenging task that has not received sufficient attention from the research community @xcite . these complex entities can be formed by any linguistic constituent (long noun phrases and sometimes complete sentences), as titles of creative works such as books and movies, different from traditional entities like person names and locations. as the creative work titles can be semantically ambiguous, the challenge is recognizing entities based mainly on their context. the semeval 2022 task 11: multiconer (multilingual complex named entity recognition) @xcite targets the recognition of lowercase complex entities in multilingual and multi-domain texts, encouraging researchers to develop new approaches to extract diversified entity types. the challenge involves the use of human language modeling in the ner task, on cross-domain texts. the semantic structure has six types of entities: person, location, group, corporation, product and creative work. the dataset was provided for 11 languages, and, in addition, this task also provided multi-language and code-mixed foot_0 features, encouraging the development of more generic and adaptive systems. as the contextualized pre-trained language models based on the transformer architecture have reached the state-of-the-art in several nlp tasks @xcite , in our method, we explore transformer-based models and combined the results into an ensemble, which can make better predictions and have a superior performance than any single contributing model (as demonstrated by @xcite , @xcite and @xcite ). the models were fine-tuned to ner task on the english, spanish and code-mixed datasets. the paper is organised as follows: section 2 provides a brief explanation of related works, section 3 provides details about the data used for training our models, section 4 describes the proposed method with implementation details, section 5 presents our results with some discussions, and in section 6 we present the conclusions obtained from the observed results. these contextual word embedding models use the learned representations over the large data and, in a process called fine-tuning, have their last layers updated to adapt for a downstream task, using taskspecific training data. the bidirectional encoder representations from transformers (bert) @xcite is a language representation model designed to pre-train deep bidirectional representations from unlabeled text conditioning on both left and right context in all layers. a transformer is a encoder-decoder architecture, but in bert-based models only the encoder is used, with the attention mechanism that learns contextual relationships between words in a text. the bert-based models are pre-trained using two language generic tasks: masked language modeling (mlm) and next-sentence-prediction (nsp) tasks. roberta was built on bert's language masking strategy, where the system learns to predict hidden sections of text within otherwise unannotated language examples. roberta modifies key hyperparameters in bert, as removing next-sentence pretraining objective. also, it was trained with much larger mini-batches and learning rates, improving the masked language modeling objective and leading to better downstream task performance @xcite . distilbert learns a distilled version of bert, maintaining almost all performance but using only half the number of parameters with a technique called distillation, which approximates the large neural network by a smaller one @xcite . we also chose distilbert on our ensemble because, although it has 40% less parameters than bert-base and runs 60% faster, can preserve over 95% of bert's performances @xcite . xlm-roberta is a transformer-based masked language model trained on one hundred languages, which outperformed multilingual bert on a variety of cross-lingual benchmarks @xcite . some researches have focused on the combination of transformer-based models into an ensemble, for nlp tasks. in the work of @xcite , the ensemble of contextualized language models resulted in an effective approach for ner in chemical patent documents, outperforming individual transformer models. @xcite achieved high results with ensemble approach for ner and their study indicates that the more models were used in the ensemble, the more the performances tend to be high and stable. the research of @xcite shows that the combination of three bert-based models obtained superior results than the individual models, in the classification of texts from social media. motivated by the success of transformer-based models along with the ensemble approach, we trained several models and combined the results into an ensemble. ## data this shared task encourages nlp researchers to develop complex ner systems for 11 languages, with semantically ambiguous and complex entities in short and low-context settings. complex entities as creative works are really challenging as they are harder to recognize. an example of entities for english and spanish can be seen in figure 1 . according to the organizers @xcite , 15,300 sentences were made available for training and 800 sentences for evaluation in the mono-lingual tracks. the training file has 15,000 sentences for code-mixed and the evaluation dataset has 500 sentences. test files have varying sizes, with 219,652 sentences for english, 272,887 sentences for spanish, and 100,000 for code-mixed tracks. data enrichment: to improve the performance of our models, we enriched our datasets with other ner annotated corpus containing similar entities. for english, we concatenated to the original mul-ticoner dataset the corpus conll-2003 @xcite , emerging entities @xcite , and in media res @xcite . conll-2003 is a ner dataset released in conll-2003 shared task for language-independent ner. we used the english data @xcite , provided from reuters corpus containing entities of location, person, organization, and miscellaneous. we converted this data into multiconer format, transforming organization into corporation. the emerging entities @xcite is a corpus released by wnut2017 shared task with focus on rare and emerging entity recognition from noisy user-generated data. this corpus contains the entities person, location, corporation, product, creative-work, and group, the same entity types we have on multiconer. in media res is a corpus for evaluating named entity linking with creative works (braşoveanu et al., 2020), annotated from wikipedia articles with entities like person, organization, location, (creative) work, event and other. we converted it to multiconer format, excluding events and others and converting work to creative work. for spanish, we enriched the dataset with conll-2002 @xcite and wikiner @xcite . conll-2002 contains entities of type persons, organizations, locations, times and quantities. we use the spanish texts, converting organization to corporation, and eliminating entities we do not use. wikiner @xcite is an annotated dataset of wikipedia texts for 9 languages. it has entities of person, location, organization, and other (misc). we used the texts in spanish, and we did the conversion from organization to corporation and an analysis of entities of type misc, where could be categorized as products, creative works and others (ignored). because of this analysis of misc-type entities, we only processed 10% of the corpus. all the datasets were converted to lowercase format, and concatenated to the original multiconer train dataset. ## system description our method consists on an ensemble of transformer-based models, trained for ner task and joined by the soft voting method (figure 2 ). from the 11 languages, we decided to participate in the english and spanish tracks, since we already have some related research on them @xcite and akhtyamova et al. (2020) ), and also in the multilingual and code-mixed tracks to assess our strategy on multilingual texts. in this section, we detail the development of ner models with its parameters and training setup, the ensemble method, and metrics used in this shared task. ## results and discussion official results: the share-task proceedings reports the results (macro f1-score) by the participating systems in the semeval 2022 task 11 @xcite , where our results are shown as uc3m-pucpr team. these results, as well as the analysis performed in the discussion section, are exclusively computed on the test set. our results refer to experiment 2 (where we used weights for each model), which performed better for all tracks. for the english track, our approach was ranked in 21 th place from 30 participants, with a f-score of 69.24, outperforming the baseline provided for the shared task by 8.04 points. on the other hand, for the spanish track, our system had a significant drop, ranking in the last position (18 th from 18 teams), with a f1-score of 56.79, i.e., 33.15 points behind first place and 0.6 points behind baseline. discussion: the ensemble method modestly improved the f1 performance in relation to the use of individual models for english (1.92 of ensemble improve in relation to the best individual model, bert-large) and code-mixed (2.85 of ensemble improve in relation to the best individual model, xlm-roberta-large). however, for spanish track, no improvement was seen in the ensemble compared to beto individually. for english track, where we work with eight different pre-trained models, the ensemble model outperforms the other models for all classes, as can be seen in to help understand the reason for the low results in spanish, we present a confusion matrix, in figure 3 . in this track, an analysis of the errors indicates that the model had difficulty recognizing the entities products, groups and creative works. by labeling them with "o" instead of the correct class, lowers the recall value for this entities. for english and code-mixed tracks, we noticed that the large models performed better than the base models, which was already expected, agreeing with the results of previous research. for english, the best-isolated model was bert-large-uncased, and for code-mixed, xlm-roberta-large, both for evaluation and testing phases. on the other hand, on spanish track, the uncased version of beto model performed better than the new roberta models generated from more than 201 million documents (570 gb) from the spanish national library. despite improving performance, only the ensemble strategy may not be enough to have competitive results, when analyzing the results of the other teams, especially for the spanish task, which had poor results. the enrichment of the train databases may have helped for some entities, but it was not enough to significantly improve performance either, since our best results were precisely in code-mixed track, in which we did not use this technique. the entity types between the external databases may have minor semantic differences, for example, there is a fine line between group and organization. maybe a closely analyze and even manual re-annotation on these entities could have improved performance. also, given such a large amount of test data (files with more than 200,000 sentences), the use of ensemble with transformer-based models may not be the most suitable for processing on regular computers, without much processing gpu power, due to the need to process this large amount of data with several models. in addition to the prediction process, the weighted sum of probabilities also consumes much memory. our team could not process the test files for the multilingual track before the deadline, and we could not process the megatron model on time to put into the ensembles (which could improve the results). we realize that complex entities still represent a great challenge to the nlp community. in addition, lowercase texts, sentences without punc-figure 4 : an example of predictions by different models from english evaluation dataset, where blue represents person entity type, green represents creative works and pink, group type. tuation or grammatically incomplete, in a crossdomain dataset without any standardization of sentence length, make it even more challenging to extract information from unstructured data, as was the case in this task. ## conclusions we presented our method for recognizing complex entities, which consists in a transformer-based models ensemble. although our team's results are not among the first, the ensemble method worked for both english and code-mixed tracks, in which we obtained an improvement in the f1 value compared to individual transformer models. our method outperformed in macro f1 the baseline provided by the organizers in 8.04 points on english track, 18.26 on code-mixed track and 12.31 on multilingual track (unofficial). as our method is based only on machine learning, without fixed rules, this indicates that the transformer models were able to take advantage of natu-ral language contexts to capture the most relevant features, on lowercase cross-domain texts. since extraction of complex entities represents a challenge and these entities are increasingly present, we would like to continue improving our method. future work intends to train models using a different split (hold out) of the data, training models with more data and for more epochs. also, since creative works had the worst performance on the three tracks, a hybrid approach that includes a dictionary with creative works can contribute to better results. furthermore, an analysis of which models should participate in the ensemble can lead to better results and lower processing costs. for example, less robust models like distilbert that are faster and lighter, but do not improve performance, possibly do not contribute to the improvement of the results. despite having the lowest task evaluation score for the spanish track, this method exhibited competitive performance at english and code-mixed tracks.