Multilingual Named Entity Recognition and Matching Using BERT and Dedupe for S lavic Languages (2021)

Abstract: This paper describes the University of Ljubljana (UL FRI) Group’s submissions to the shared task at the Balto-Slavic Natural Language Processing (BSNLP) 2021 Workshop. We experiment with multiple BERT-based models, pre-trained in multi-lingual, Croatian-Slovene-English and Slovene-only data. We perform training iteratively and on the concatenated data of previously available NER datasets. For the normalization task we use Stanza lemmatizer, while for entity matching we implemented a baseline using the Dedupe library. The performance of evaluations suggests that multi-source settings outperform less-resourced approaches. The best NER models achieve 0.91 F-score on Slovene training data splits while the best official submission achieved F-scores of 0.84 and 0.78 for relaxed partial matching and strict settings, respectively. In multi-lingual NER setting we achieve F-scores of 0.82 and 0.74.

## introduction natural language processing (nlp) has recently become more popular, especially due to advancements in deep learning that helped to achieve superior results for various nlp tasks. one of the main issues -adaptation of models to low-resourced languages has been nicely addressed by multi-lingual and large pretrained models. languages such as english, german, spanish are very well developed, with a lot of resources, while on the other hand, under-developed languages, such as the slavic languages, have less annotated resources. still, research may show that highly inflected languages contain more semantic information that could be exploited using nlp in the future. in this paper we explore existing resources for slavic languages, more particularly, slovene. we use novel bert-based approaches @xcite and pretrained models, such as the croslo-engual bert @xcite . we especially focused into how well they perform for the ner task which takes part in bsnlp 2021, and how we can improve its performance with preexisting data sets, such as ssj500k @xcite and bsnlp 2017 datasets. additionally, we provide a simple strategy for training the dedupe method @xcite as part of the entity matching task. the strategy is based on choosing positive and negative examples for dedupe, which manages to outperform the baseline of assigning a single cluster to all entities on a language level. finally, we utilize nlp pipelines to perform pos tagging of the documents and bring the named entities in a normalized form. the paper is organized as follows. we first describe the shared task (section 2), then present our method variations for named entity recognition and matching (section 3). we conclude the paper with a subset of results and discussion (section 4). ## methods in this section we describe the methodology we used to obtain our submission results. to do so, we built a pipeline which first pre-processes the provided data in a format, appropriate for training. pre-processed data is then used to train ner models, normalize the ne occurrences, and find matches among them. in the last step we merge all the obtained results to create a bsnlp-compatible output. ## results and discussion the training of the ner models is divided into multiple training bundles (see appendix a). first we evaluated our models for all languages on a test set from each language - the slovene-based bundles try to identify the best model for the ner task. with the slovene-all bundle we establish a baseline for the performance of the chosen models on the slovene portion of the bsnlp 2021 data set. the results for the slovene all bundle are presented in the results from by comparing the results from for the em task, we obtained the results shown in for the participation in the shared task we submitted 5 system responses -one multi-language and others only for the slovene language. in the code to reproduce the experiment results and re-train the models is publicly available in our public github repository foot_8 . ## conclusion we have built a pipeline that addresses the bsnlp 2021 shared task. the pipeline normalizes the in-