Benchmarking Pre-trained Language Models for Multilingual NER : T ra S pa S at the BSNLP 2021 Shared Task (2021)

Abstract: In this paper we describe TraSpaS, a submission to the third shared task on named entity recognition hosted as part of the Balto-Slavic Natural Language Processing (BSNLP) Workshop. In it we evaluate various pre-trained language models on the NER task using three open-source NLP toolkits: character level language model with Stanza, language-specific BERT-style models with SpaCy and Adapter-enabled XLM-R with Trankit. Our results show that the Trankit-based models outperformed those based on the other two toolkits, even when trained on smaller amounts of data. Our code is available at https://github.com/NaiveNeuron/slavner-2021 .

## introduction this paper describes the traspas submission to the third shared task of the balto-slavic natural language processing (bsnlp) workshop at eacl 2021. the task focuses on recognizing named entities (ner), their normalization and linking across six slavic languages. the participants are provided training data comprised of news articles crawled from the web, centered around four specific topics and labeled on the document level, while the testing data consists of news articles focused on two topics, which are completely different from those included in the training data. this setup poses a significant research challenge, as it features languages with latin as well as cyrillic scripts, substantial class imbalance in the training data, specific set of named entity tags (which makes it difficult to find comparable datasets) as well as documentlevel annotation, which does not map directly to the more generally used token-level annotation and requires custom preprocessing. our solution utilizes three open-source nlp toolkits, namely trankit * these authors contributed equally to the work @xcite , spacy and stanza @xcite , yielding the name traspas. we chose these particular toolkits for their popularity, ability to exploit large pre-trained language models for the ner task, as well as their potential to be used in production deployments. the research question we are trying to pose in this work can be paraphrased as follows: can the universal open-source nlp toolkits (such as spacy, stanza or trankit) be used to achieve competitive performance on multilingual named entity recognition? this has been tested to some extent in the previous edition of the shared task with nlp-cube foot_0 , albeit with not very encouraging results: on average it reached f1 scores of around 0.2 whereas the top three best performing systems regularly reported f1 scores of over 0.85. we hypothesize that the situation has changed significantly since then, mostly thanks to large pretrained transformer-based language models, which are being made accessible thanks to huggingface's transformers @xcite . hence, we believe that the customized setup that was previously required in order for models like bert to be used may no longer be necessary and that competitive performance can be obtained using an off-the-shelf toolkit. furthermore, this comparison sheds more light on multilingual transfer learning, as stanza uses lstm-based pre-trained language models and both spacy and trankit generally utilize variants of bert. moreover, since spacy and stanza defaults to using single-source models (i.e. one model per each language) and trankit works primarily in the multi-source setup, our experiments allow us to compare these two approaches as well. we further investigate the impact of various tokenizers and additional data on final performance and perform a detailed error analysis. our results indicate that the multi-source approach implemented using trankit yielded the strongest performance, both on the development as well as the test set. the impact of additional data is pronounced in the case of single-source approach but does not lead to significant increase in performance of multi-source systems. the error analysis shows that all of the presented system struggle with the evt (events) tag, which is the least populated in the training set. the highest error rate is between the pro (products) and org (organizations) tags. we speculate that this might be due to many ambiguities between these two domains, such as "uber", a product of the uber company or "ryanair choice", a specific product which contains the name of the ryanair airline. some of these errors might be caused by the disambiguation rules outlined in the task guidelines. ## related work the balto-slavic nlp workshop has previously organized two shared tasks. in the first one, held in 2017, no training data was provided and as reported in @xcite , due to the relatively high complexity of the task, only two teams have submitted the results within the task's deadline. the second one was held in 2019 and featured document-level annotated data in four languages (bulgarian, czech, polish and russian) on four specific topics, two of which were released as training set, with the remained serving as the test set. as @xcite reports, most teams participating in this task used embeddings provided by the bert model @xcite , often in its multilingual version, coupled with cross-lingual training @xcite and crf @xcite or ncrf++ @xcite as the top-most layer. the work presented in @xcite is of particular note, as the authors show that multi-source training outperforms the single-source approach. as one of the standard tasks of natural language processing (nlp), named entity recognition (ner) has been extensively studied in the past. in recent years, performance improvements have been obtained by the introduction of the conditional random field (crf) @xcite , especially when combined with neural representation of the input text, yielding the bilstm-crf model @xcite . further improvements can be attributed to the widespread use of contex-tual embeddings @xcite @xcite provided by large-scale language models. an example of such a model in the multilingual setup would be multilingual bert @xcite , which was trained on the text of top 104 largest wikipedias. its performance on the ner task has recently been improved by xlm @xcite which leverages cross-lingual language model pre-traning, and xlm-r @xcite which applies this pre-training approach to commoncrawl data, which is significantly larger than data obtained from wikipedia. the increasing proliferation of open-source nlp toolkits has significantly lowered the barrier for inclusion of sophisticated language processing tools in larger systems. these include corenlp @xcite , flair @xcite , ud-pipe @xcite , nlp cube @xcite and spacy foot_1 , which is focused on industry usage and hence optimized for efficiency. historically, the nlp toolkits have primarily supported a few major languages but recently the trend has reversed, with toolkits aiming to support the processing of multilingual text. stanza @xcite ) is one such toolkit, which provides pretrained models for 66 languages. these models are generally based on bilstm architecture, which is then amended for the target downstream task. trankit @xcite is a recently introduced toolkit based on the xlm-r language model, which also makes use of adapters architecture @xcite . this allows the xlm-r model to be reused across languages and only a small number of parameters to be finetuned for a specific language and/or component of the toolkit. ## the bsnlp2021 shared task the bsnlp2021 shared task is the third installment of the multilingual named entity challenge in slavic languages, organized as part of the 8th workshop on balto-slavic natural language processing. similarly to its previous editions, it focuses on recognizing named entities in text of web news articles, their normalization and linking across languages. ## results the main results can be seen in the table is split into four parts. in the first column are the names of the evaluated models, along with the aforementioned tags that alter their training set. the following six columns present the performance of respective models on the perlanguage subsets of the development set. the next five columns visualize the performance across the ner tags, while the last column is the f1 score computed on the whole development set. the presented metrics were computed using the official evaluation tool provided by the organizers of the shared task foot_6 . the test set results can be found in ## discussion in this section we discuss several observations which follow from the results presented in in the following subsections we discuss the impact of various pre-built models, the various evaluated tokenization methods, the inclusion of additional data and conduct an error analysis on the considered system. ## conclusions this paper describes the traspas submission to the bsnlp 2021 shared task on multilingual named entity recognition in which three opensource nlp toolkits were benchmarked. the results show that the approach implemented using the trankit toolkit, based on the xlm-r pre-trained language, produced a model with the best results on both the development as well as the test set. while additional data did help in the case of the other two frameworks, the effect on the trankit-based models was mostly negative. error analysis on the development set indicates that many errors of our models could be attributed due to ambiguities in the training data.