Using a Frustratingly Easy Domain and Tagset Adaptation for Creating S lavic Named Entity Recognition Systems (2021)

Abstract: We present a collection of Named Entity Recognition (NER) systems for six Slavic languages: Bulgarian, Czech, Polish, Slovenian, Russian and Ukrainian. These NER systems have been trained using different BERT models and a Frustratingly Easy Domain Adaptation (FEDA). FEDA allow us creating NER systems using multiple datasets without having to worry about whether the tagset (e.g. Location, Event, Miscellaneous, Time) in the source and target domains match, while increasing the amount of data available for training. Moreover, we boosted the prediction on named entities by marking uppercase words and predicting masked words. Participating in the 3rd Shared Task on SlavNER, our NER systems reached a strict match micro F-score of up to 0.908. The results demonstrate good generalization, even in named entities with weak regularity, such as book titles, or entities that were never seen during the training.

## introduction named entity recognition (ner) is a fundamental task in domain of natural language processing (nlp) that consists of extracting entities that semantically refer to aspects such as locations, people or organizations @xcite . since the creation of bert @xcite , multiple ner systems have brought the state of the art to new levels of performance. nonetheless, there are many challenges that still need to be faced, especially in the case of less-resources languages. in the 2 nd shared task on slavner @xcite , the top-two systems in the detection named entities (nes), @xcite and @xcite , managed to reach a relaxed partial micro f-score of 0.9, followed by 1 bsnlp.cs.helsinki.fi/shared-task.html, last visited on 9 march 2021 two other systems with values slightly better than 0.8 @xcite . for the 3 rd shared task on slavner, we consider that in order to improve the scores, in terms of the strict evaluation, and nes related to products and events, it is necessary to include additional data that could improve the generalization of the models to any kind of topic. while in the literature there are multiple techniques for training models over additional datasets, such as transfer learning and domain adaptation, using these techniques might pose additional questions. for example, to determine which layers to freeze, fine-tune or substitute. furthermore, different datasets might use dissimilar tagsets, which might be incompatible @xcite . in this paper, we present the participation of laboratory l3i in the 3 rd shared task on slavner. specifically, we participate with multiple ner systems for slavic languages using different bert models and training over diverse datasets through a frustratingly easy domain adaptation (feda) algorithm @xcite . 2 the feda algorithm has for objective to learn common and domain-specific patterns between multiple datasets, while keeping separately patterns belonging only to the domain-specific data @xcite . particularly, the use of feda allow us sharing the knowledge and patterns found in multiple datasets without having to worry about which different tagsets are used among them. apart from the feda algorithm, we explore some other techniques that might improve the performance of our ner system based on the ideas of @xcite . specifically, we analyze whether the marking and enrichment of uppercase tokens can improve the detection of nes. as well, we use the prediction of masked tokens as a way to improve ner systems' generalization. the rest of the paper is organized as follows. in section 2, we introduce the background for the proposed work. this is followed by the methodology in section 3. the data and the experimental settings are described in section 4 and section 5, respectively. in section 6, we present the results obtained. finally, the conclusions and future work are detailed in section 7. weak generalization: one of the most challenging aspects of ner systems is to deal with nes that have a weak or zero regularity, such as names of movies, and nes that were never seen during training @xcite . some methods found in the literature for improving generalization consists of learning manually defined triggers @xcite , but also permuting nes and reducing context such as in @xcite . feda: originally proposed by @xcite , the feda was firstly designed for sparse machine learning algorithms. later, @xcite , proposed a neural network version of this domain adaptation algorithm. while the former resides in duplicating input features, the latter consists of activating specific neural network layers. ## methodology consider @xmath0, d 2 , . . . , d n |@xmath1. furthermore, consider a classifier c a stack of two linear layers in which in between we set an activation layer relu and a dropout. the first linear layer has a size of 512, while the output h produced by c has a size of l, which is the number of different labels found in d. thus, the proposed model for doing the feda consists of adding on top of bert n + 1 classifiers such that we have @xmath2, c 1 , c 2 , . . . , c n }. the classifier c 0 represents a general classifier that will receive as input the sentences from all the datasets in d, while c @xmath3. for each sentence belonging to a dataset d k , we do the element-wise sum between h 0 and h k , i.e. h @xmath4. finally, h k is introduced it into a crf layer, which will determine the labels of each word in a sentence. figure 1 depicts the proposed architecture. for increasing the generalization of our ner systems, we explore the prediction of masked tokens during the training as proposed by @xcite . firstly, this method converts randomly selected tokens, within a sentence, into bert's special token @xcite . then, the ner system has to predict correctly the sentence's nes, despite the missing information, as well as predicting the masked tokens. the prediction of masked tokens is done by introducing bert's output into a linear layer, which has the same size of the pretrained vocabulary. during training, the loss produced by the prediction of masked tokens is added to the loss produced by the recognition of nes; during testing, this layer is inactive. although @xcite propose unicase, an architecture for training a language model that learns the casing of a word separately to the tokenization, in this work, we use a simpler method that does not require to retrain a language model. specifically, we use a marking and enrichment approach, where an uppercase word is tagged with two special bert's tokens, defined by us as [up] and @xcite , and where we include additional case versions. for instance, the word "rome" becomes "[up] [rom, ##e] [rome] [r,##ome] [up]". it is important to indicate that the prediction of the ne type is done uniquely over the first token, which correspond to the special token @xcite . in other words, the output produced by bert for the rest of the tokens is masked. the marking of the uppercase words is based on the ideas proposed by @xcite . ## datasets we use the data provided by the organizers for the 3 rd shared task on slavner. however, for the development of our internal models, we use the topics of nord stream and ryanair as testing partition, while the rest as training and development. for the final models, all the data provided is split into training and development sets. besides the data provided by slavner's organizers, we use the follwing ner corpora: slavner 2017 @xcite : slavic corpus annotated with 4 ne types: location, miscellaneous, organization and person. collection named entities 5 (cne5) @xcite 3 : russian ner corpus manually annotated with five ne types: geopolitical, location, media, person and organization. czech named entity corpus 2.0 (cnec) @xcite : czech corpus annotated with fine-grained ne. in this work, we have used 6 types of ne: location, organization, media, artifact, person and time. factrueval 4 : russian corpus annotated with three ne types: location, organization and person. finnish ner @xcite : although finnish is not a language to process in slavner, it has similar ne types to those used in the shared task: date, event, location, organization, person, product and time. we use this dataset to enrich the nes knowledge, specially on events and products. national corpus of polish (nkjp) 5 @xcite : polish corpus tagged with five ne types: person, organization, geopolitical, location, date and time. ner-uk 6 : collection of 264 ukrainian docuwikiann @xcite : it is a multilingual ner corpus based on wikipedia articles; it was annotated automatically using three types of nes: location, organization and person. we use of the corpus partitions used by @xcite . we use for all the additional corpora their training, development and testing partitions; if these are not provided, we create them using a stratified approach to ensure a proportional number of nes. ## experimental setup regarding bert, we use different pre-trained models: crosloengual (ulčar and robnik-šikonja, 2020), polish bert foot_2 , rubert @xcite and language-agnostic bert sentence embedding (labse) @xcite . all the files coming from slavner are tokenized and, those used for training and development are annotated at token-level. for bulgarian and slovene, we tokenize the documents using reldi-tokenizer foot_3 , while for the rest of languages, we use the neural parser proposed by @xcite . further-more, we over-tokenize all the files, i.e. we separate all the punctuation from tokens within a sentence, to solve some cases where abbreviation periods or dashes were not considered as part of a ne. for example, in slovene, roman numerals are followed by a period, such as in benedikt xvi. nevertheless, some ne annotations did not consider the period. some rules and manual corrections were applied to the tokenization where we determined the fix was critical. for instance, in polish, w. brytania (great britain) was being split into two sentences by the tokenizer. we automatically annotated the files by searching the longest match in the tokenized format and the annotation file. in case of ambiguity, the annotation tool requested a manual intervention. for the final submission, we converted the token-level output to a document-level one. all the nes types are encoded using bioes (beginning, inside, outside/other, end, single). as well, to reduce the number of entities types, we normalize those where the theoretical meaning is the same, i.e. pers into per or event into evt. for the models where masked tokens have to be predicted, we only affect sentences in the training partitions that are longer than 3 actual tokens, i.e. not bpe tokens. at each epoch, we select randomly 25% of each sentence's tokens and substitute them with @xcite . if a token after being processed by bert's tokenizer produces more than one bpe token, we mask only one of them. 10 regarding the models that are trained with marked uppercase tokens, at each training epoch, we randomly convert 5% of all the sentences into uppercase. this is done to provide some examples of uppercase sentences to datasets that do not present this phenomenon. in ## results in from a manual inspection, we have observed that multiple events were considered as products, such as miss usa, pizzagate and covid-19. some products were marked as organizations such as zoom, covax, apple tv+, although fewer organizations were tagged as products, such as pfizer/moderna and bbc. nonetheless, many of these nes could be both types depending on the context in which happen. in certain documents, organizations were marked as locations and viceversa, such as ostravské fakultní nemocnice (ostrava university hospital) and szpitala wojskowego w szczecinie (military hospital in szczecin). we have found interesting examples regarding products despite their irregularity. for example, the cyrillic and latin models managed to detect partially the 2020 book "nelojalen: resnična zgodba nekdanjega osebnega odvetnika predsednika donalda trumpa" (disloyal: a memoir: the true story of the former personal attorney to president donald j. trump). specifically, the entity was some models considered biontech as an organization and instagram as a product despite these nes were never seen during the training. as well, some medication-related products were correctly found such as azd1222, канакинумаб (canakinumab), remdesivir or zithromax, even if they did not exist on the training corpora. we observed, specially in cyrillic-scripted languages, that some named entities were incorrect because they were predicted without punctuation marks. for example: moderna inc vs moderna inc., гам-ковид-вак vs гам-ковид-вак and спутником vs "спутником". in latin-scripted languages, we observed the opposite although less frequently. for instance, roberta f. kennedyho jr. vs roberta f. kennedyho jr. in some documents the punctuation mark is included in certain nes but not in others, such as in korea płn. vs korea płn but korei płn.. ## conclusions and future work this work presented the participation of laboratory l3i in the 3 rd shared task on slavner. specifically, we proposed a collection of bert-based ner systems that were trained using multiple datasets through feda. the results showed us that our ner systems worked better on the u.s. elections topic (strict micro f-score between 0.762 and 0.908) than on the covid-19 topic (0.666 -0.775). overall, a competitive strength of our ner systems is that they managed to predict named entities occurring with weak regularity or that were never seen before. in the future, we will apply the proposed architecture on other languages and datasets.