Priberam Labs at the 3rd Shared Task on S lav NER (2021)

Abstract: This document describes our participation at the 3rd Shared Task on SlavNER, part of the 8th Balto-Slavic Natural Language Processing Workshop, where we focused exclusively in the Named Entity Recognition (NER) task. We addressed this task by combining multi-lingual contextual embedding models, such as XLM-R (Conneau et al., 2020), with character- level embeddings and a biaffine classifier (Yu et al., 2020). This allowed us to train downstream models for NER using all the available training data. We are able to show that this approach results in good performance when replicating the scenario of the 2nd Shared Task.

## introduction this document describes our participation at the 3 rd shared task on slavner, part of the 8 th balto-slavic natural language processing workshop, held in conjunction with the 16 th conference of the european chapter of the association for computational linguistics (eacl). it includes three different subtasks: named entity recognition (ner), including detection and classification, lemmatization, and cross-lingual entity linking. the differentiating feature of this shared task is the focus on six slavic languages: bulgarian (bg), czech (cs), polish (pl), russian (ru), slovene (sl), and ukrainian (uk). we focus our participation exclusively on the task of ner, and we base ourselves on recent developments that show that cross-lingual embeddings produce good results for a wide range of languages and tasks @xcite . our overall approach is heavily based in @xcite , and uses both contextual and characterlevel embeddings as input to a sequence of models which culminates in a biaffine classifier. due to the multilingual nature of this task we explore multilingual contextual embedding models, such as multilingual bert @xcite and xlm-roberta @xcite , as a way of leveraging all the available training data at once. it differs from the approaches presented in the previous edition of the shared task @xcite in the following aspects: (i) we explore more contextual embedding approaches; (ii) we further finetune the contextual embedding model while training the downstream model; (iii) we use the same topics in our train and development sets; and (iv) we use a different classifier architecture. our approach resulted in strong performance when replicating the scenario of the 2 nd shared task on slavner. ## related work named entity recognition corresponds to the task of both finding and classifying named entities in text. some of the most common approaches to tackle ner that make use of neural networks involve combining models such as conditional random fields (crfs, @xcite , bidirectional long-short-term-memory neural networks (bilstms, @xcite @xcite . two examples of such approaches are @xcite and @xcite , which use bilstms and cnns to build both wordand character-level features, diverging in the fact that the latter performs decoding with a crf, while the former uses a linear layer. with the development of better pre-trained embedding models, transfer-learning became a significant part of approaches that tackle ner. this technique implies using a pre-trained model in order to obtain an embedding for each word of the input. these representations are then used as input to a model that is able to solve the desired downstream task. in terms of pre-trained embeddings we highlight elmo embeddings @xcite , which are trained using a bidirectional language model, the flair embeddings @xcite , trained with a character-level language model, and finally embeddings retrieved from transformerbased models @xcite , such as bert @xcite , and roberta @xcite . any combination of these embeddings can be used as input to a model that is able to predict entity classes, such as a lstm-crf @xcite , a biaffine-classifier @xcite , or a linear layer @xcite . particularly relevant to this work is the possibility of using pre-trained cross-lingual embeddings, such as multilingual bert @xcite and xlm-roberta @xcite , both covering a high-number of languages with different scripts. such models have been shown to perform well on different tasks and languages @xcite @xcite . in the previous edition of the slavner shared task @xcite multiple submissions used multilingual bert to retrieve cross-lingual representations. in particular, tsygankova et al. (2019) used both word-and character-embeddings as input to a bilstm-crf, arkhipov et al. (2019) further pretrained multilingual bert on the four target slavic languages of last shared task's edition and combined its representations with a word-level crf, and the submission by iiuwr.pl used a combination of different embeddings, where bert and flair were included. ## approach our approach makes use of three key components: a multilingual contextual embedding model, a character-level embedding model, and a biaffine classifier model. in terms of multilingual contextual embedding models we have explored three options: besides finetuning the the top-layers of the contextual embedding model during training, we complement these representations with character-level embeddings, obtained with a single-layer cnn. the biaffine classifier model follows the work by @xcite . in particular, the token-and character-level embeddings are concatenated and fed into a highway bilstm @xcite which yields a representation for each token. these representations are given to two individual feed-forward neural networks (ffnns), responsible for creating a representation that models whether a token is the start/end of a span. finally, these are passed to a biaffine model, which returns a scores tensor with all possible start-end combinations with shape n × n × c, where n is the number of tokens and c is the number of ner classes plus one, corresponding to the no-entity prediction. this scores tensor masks non-valid spans, i.e., spans where the end position is lower than the start position. a series of heuristics is then applied to the scores tensor in order to predict spans. first, all the valid spans are retrieved and matched with the corresponding highest-scoring label. then, all spans whose highest-scoring label corresponds to an entity are sorted by score, from highest to lowest, and are evaluated sequentially. all predicted spans are kept, unless they clash with some of the spans already validated for that input, i.e., unless they overlap with entities that were given an higher score. one of the advantages of this model is that it can model both flat and nested entities, based upon the heuristics we apply. the model is optimized with the softmax crossentropy loss. ## conclusion we have proposed a multilingual approach to the 3 rd slavner shared task, where we can make use of a single model trained with multiple source languages to predict ner tags. in particular, we have shown that using a large model, such as xlm-r, coupled with character-level embeddings and a biaffine classifier is able to perform well when replicating the scenario of the 2 nd shared task on slavner, as well as for the languages added to this year's edition of the slavner shared task.