Large Language Models for Multilingual S lavic Named Entity Linking (2023)

Abstract: This paper describes our submission for the 4th Shared Task on SlavNER on three Slavic languages - Czech, Polish and Russian. We use pre-trained multilingual XLM-R Language Model (Conneau et al., 2020) and fine-tune it for three Slavic languages using datasets provided by organizers. Our multilingual NER model achieves 0.896 F-score on all corpora, with the best result for Czech (0.914) and the worst for Russian (0.880). Our cross-language entity linking module achieves F-score of 0.669 in the official SlavNER 2023 evaluation.

## introduction the 4 th edition of shared task address three slavic languages: czech, polish, and russian, and five types of named entities (persons, locations, organizations, events, and products). all languages are highly inflective and have a rather free word order. thus named entity normalization task faces an additional challenge in the case of the normalization of multi-word expressions (mwe). in our submission, we continue experiments with xlm-r language model @xcite which has demonstrated the best result in previous shared task @xcite . we also elaborate on the normalization step for mwes by applying syntax-based noun phrase normalization tool to reach higher accuracy in named entity (ne) normalization and linking tasks. finally, we also improve entity linking by better algorithms for linking entity variants on a document level using string similarity, proximity, and type attributes. the paper is organized as follows. we start with an overview of the data preparation step (section 3) and the overall architecture of the system (section 4). then, we present each step in our workflowmention detection, entity normalization, and entity linking. we conclude the paper with a subset of results and a discussion (section 8). ## related work the shared task on slavic multilingual named entity recognition, normalization, and linking (slavner) has been organized since 2017 @xcite . only two systems were submitted for the first slavner. the best result for ner was achieved for polish (f-score of 66.6), while for cross-lingual entity matching only 9 f1 points were reached @xcite . authors of this system annotated parallel english-target language datasets using an english ner and projected annotations to the target language. a target language tagger was then trained using inferred datasets. seven teams submitted systems to the 2 nd slavner @xcite . the three best systems (ris @xcite , cogcomp @xcite and iiuwr.pl @xcite ) used bert for the ner task. the best model, cogcomp, yields an f-measure of 91% according to the shared task organizers. the cross-lingual entity linking results also have improved significantly: the best-performing model, iiuwr.pl yields the f-measure of 45%. six teams submitted their systems to the 3 rd slavner @xcite . overall ner task results were lower when compared to the 2 nd slavner. the best system, priberam @xcite , achieved f-measure of 85.7% for the relaxed partial evaluation. priberam used xlm-r large model, a character-level embedding model, and a biaffine classifier for ner task. for cross-lingual entity linking, the best-performing model, tld @xcite , achieved an f-measure of 50.4% using labse @xcite embeddings to align entities according to pre-defined thresholds. ## data preparation the data provided by the slavner task organizers contains annotations for five classes of entities: event (evt), location (loc), person (per), organization (org), and product (pro). for ner system training we convert data into a conll2003like format. we do not use the data from the bsnlp2017 shared task @xcite , as it has 4 named entity classes which are inconsistent with the rest of slavner data @xcite , and has shown to hurt the performance of ner models for this task @xcite . in addition to the dataset provided by the slavner task organizers @xcite , we use the following datasets in our experiments: kpwr @xcite contains polish texts labeled using 82 classes of entities, which we map to the 5 classes used in the slavner task. nkjp @xcite is national corpus of polish, tagged with fine-grained nes. we use entity types per ('forename', 'surname'), loc (placename, geogname), and org (orgname). finer (ruokolainen et al., 2019) is a finnish dataset that contains the same ne types as slavner, thus useful to train ner for evt and pro classes. cnec @xcite czech named entity corpus 2.0 is labeled according to a twolevel hierarchy of 46 named entities. it was mapped to the corresponding 4 classes of the slavner task: org, per, loc, and pro. factru @xcite is a russian dataset, labeled with 4 classes of entities (org, lo-corg, location, and person), which can be mapped to 3 classes of the slavner task: org, loc, per. ## architecture and systems the architecture of our solution is modular: the modules roughly correspond to the data processing steps necessary to reach the objectives of different slavner shared tasks: mention detection, lemmatization, and linking (figure 1 ). we submitted five systems to the slavner task. ## two approaches to mention detection: traditional and ensemble we consider the named entity mention detection and classification task as the ner task. we use the flair library @xcite to perform ner. flair library allows fine-tuning a transformer @xcite model with custom data. multilingual xlm-r has demonstrated the best result in previous shared task @xcite and is used as a basis for our ner models. the xlm-r is available in xlm-r base (@xmath0, @xmath1, @xmath2, 270m params) and xlm-r large(@xmath3, @xmath4, @xmath5, 550m params) variants. we use xlm-r large model fine-tuned on the dataset provided by the shared task organizers as a ner model for our system-2 and system-3. for system-4 we fine-tune a xlm-r large model on the dataset given by the shared task organizers combined together with kpwr-ner dataset @xcite . although multiple ner datasets for czech, polish, and russian are available, most of them could not be directly used due to differences in tagsets. however, even if the set of labeled classes is incompatible with the slavner labeling schema, it is still possible to use this data for training a ner system to recognize a single class that has compatible labeling. this is done by keeping only a single label in a dataset and deleting all other labels. using single-label datasets, we train a ner system by combing slavner dataset with this dataset and evaluate against the slavner test split. if a system achieves better results than the baseline system trained on slavner data, we consider this dataset as compatible with slavner and select to train the final ner model for a given label. datasets used to fine-tune each single-label ner model are summarised in due to performance and time restrictions, the xlm-r base model is used to fine-tune ensemble models. during the evaluation, all five ner models are run sequentially. the overlapping labels are resolved, first by selecting the longest labeled entity and then, if there is an exact overlap, by selecting the highest score returned by ner. this ensemble approach is used by our system-1. since the xlm-r models were created more than two years ago, and thus outdated with respect to current events, we crawled 2.6 gb of the latest czech, polish, and russian news articles foot_0 to perform additional pretraining of xlm-r base model. due to the time restrictions, additional pretraining was done using huggingface/transformers example script foot_1 with batch size 512, for 7000 steps. this additionally pre-trained xlm-r model was finetuned using the slavner dataset and the kpwr dataset for ner of system-5. ## entity normalization we use several strategies for entity normalization. in case of the czech language we apply a simple word-level lemmatization strategy. we use stanza @xcite czech language lemmatizer for this task. for entity normalization in polish and russian, we use a language-specific noun phrase generator. it allows us to transform the noun phrase into the corresponding base form taking into account the grammar rules of the specific language. the normalization workflow includes several steps: tokenization, morphological analysis, syntactic parsing, morphological transfer, and morphological synthesis of the base form. morphological analysis and synthesis are performed with help of a language-specific finite state transducer (fst). this fst solution was initially developed for the latvian language @xcite and recently extended to many other european languages -lithuanian, polish, finnish, swedish, spanish, french, german, and english. for the syntactic parsing cocke-younger-kasami (cyk) algorithm @xcite is employed by adapting the corresponding latvian tool @xcite . when analysing output of the normalisation tool, we identified several reasons for errors: ## entity linking the goal of the entity linking task is to associate entity mentions found in a text with corresponding entries in a knowledge base (kb) @xcite . traditional entity linking pipeline consists of three steps: mention detection, candidate selection, and disambiguation @xcite . the mention detection step is described in the section 5. due to the small expected size of our cross-lingual knowledge base (the actual maximum number of kb entries produced in this shared task by our systems was 939), we skip the candidate selection step. instead, a simple consistency check is applied to filter out mentions which do not have the same type @xcite . as result, the candidates are all entries in the knowledge base which have the same type as the entity mention, which we are attempting to link. the candidate selection and disambiguation usually include a three-step process of candidate generation, candidate ranking, and unlinkable mention prediction @xcite . in our submission, the candidates are ranked using a mention-ranking model @xcite to decide whether an active mention is co-referent with a candidate antecedent. we follow the algorithm proposed by @xcite : at first, we use labse to obtain entity mention embeddings and then we apply cosine similarity to calculate the similarity between obtained embeddings and those in the knowledge base. the similarity threshold for early stopping is set at 0.95 -if the similarity is above the threshold, the process links the entity mention to the candidate mention and returns the candidate mention id. if none of the candidate mentions has a similarity score above 0.6, the entity mention is considered not found in the knowledge base and is added as a new entry to the knowledge base. for entities with similarity scores between 0.6 and 0.95, the candidate with the highest score is selected for linking. usually, at the beginning of the text entities are introduced (named) carefully, e.g. with a full name (and acronym), while later in a text, when it is clear from the context what they refer to, entities are often used in the shortened form @xcite . for such cases, we introduce an additional linking step at the document level: for each entity mention, we check whether its name is part of another entity, e.g., encountering the name "asia", it could be matched as part of "asia bibi". we perform this step before attempting to link an entity to the knowledge base. we also check for organization and person name abbreviations and translations. at first, we identify entities that are surrounded by brackets (optionally, quoted). then, if the entity immediately preceding it belongs to the same type, both entities are linked together as aliases. ## results ## conclusions in this paper, we presented a modular architecture for the recognition, normalization, classification, and cross-lingual linking of named entities in slavic languages. each module (ner, normalization tool, and ne linker) is self-contained and could be improved independently from others. while none of the systems fine-tuned on additional datasets surpassed the xlm-r large system fine-tuned on slavner data, the ensemble system seems promising and could be retrained again using the xlm-r large model instead of xlm-r base in order to obtain better results.