Slav- NER : the 4th Cross-lingual Challenge on Recognition, Normalization, Classification, and Linking of Named Entities across S lavic languages (2023)

Abstract: This paper describes Slav-NER: the 4th Multilingual Named Entity Challenge in Slavic languages. The tasks involve recognizing mentions of named entities in Web documents, normalization of the names, and cross-lingual linking. This version of the Challenge covers three languages and five entity types. It is organized as part of the 9th Slavic Natural Language Processing Workshop, co-located with the EACL 2023 Conference.Seven teams registered and three participated actively in the competition. Performance for the named entity recognition and normalization tasks reached 90% F1 measure, much higher than reported in the first edition of the Challenge, but similar to the results reported in the latest edition. Performance for the entity linking task for individual language reached the range of 72-80% F1 measure. Detailed evaluation information is available on the Shared Task web page.

## introduction analyzing named entities (nes) in slavic languages poses a challenging problem, due to the rich inflection and derivation, free word order, and other morphological and syntactic phenomena exhibited in these languages @xcite . encouraging research on detection and normalization of nes-and on the closely related problem of cross-lingual, crossdocument entity linking-is of paramount importance for improving multilingual and cross-lingual information access in these languages. this paper describes the 4 th shared task on multilingual ne recognition (ner), which aims at addressing these problems in a systematic way. the shared task was organized in the context of the 9 th slav-nlp: workshop on natural language processing in slavic languages, co-located with the eacl 2023 conference. the task covers three languages-czech, polish and russian-and five types of ne: person, location, organization, product, and event. the data consists of documents collected from the web involving certain "focal" events. the rationale of such a setup is to foster the development of "end-to-end" ner and crosslingual entity linking solutions, which are not tailored to specific, narrow domains. this paper also serves as an introduction and guide for researchers wishing to explore these problems using the training and test data, which are released to the public. 1 the paper is organized as follows. section 2 reviews prior work. section 3 describes the task. section 4 describes the annotation of the dataset. the evaluation methodology is introduced in section 5. participant systems are described in section 6, and the results obtained by these systems are presented in section 7. conclusions and lessons learned are in section 8. ## prior work the work described here builds on a series of shared tasks on multilingual named entity recognition, normalization and cross-lingual matching for slavic languages, @xcite @xcite , which, to the best of our knowledge, are the first attempts at such shared tasks covering multiple slavic languages. high-quality recognition and analysis of nes is an essential step not only for information access, such as document retrieval and clustering, but it also constitutes a fundamental processing step in a wide range of nlp pipelines built for higher-level analysis of text, such as information extraction, see, e.g. @xcite . other ner-related shared tasks have been organized previously. the first non-english monolingual ner evaluations-covering chinese, japanese, spanish, and arabic-were held in the con-text of the message understanding conferences (mucs) @xcite and the ace programme @xcite . the first multilingual ner shared task, which covered several european languages, including spanish, german, and dutch, was organized in the context of the conll conferences @xcite . the ne types covered in these campaigns were similar to the ne types covered in our challenge. worth mentioning in this context is entity discovery and linking (edl) @xcite , a track of the nist text analysis conferences (tac). edl aimed to extract entity mentions from a collection of documents in multiple languages (english, chinese, and spanish), and to partition the entities into cross-document equivalence classes, by either linking mentions to a knowledge base or directly clustering them. an important difference between edl and our task is that edl required linking entities to a pre-existing knowledge base. related to cross-lingual ne recognition is ne transliteration, i.e., linking nes across languages that use different alphabets/writing systems. a series of ne transliteration shared tasks were organized as part of news-named entity workshops @xcite , focusing mostly on indian and asian languages. in 2010, the news workshop included a shared task on transliteration mining @xcite , i.e., mining of names from parallel corpora, in languages including english, chinese, tamil, russian, and arabic. research on ne focusing on slavic languages includes ne recognition for croatian @xcite , ne recognition in croatian tweets @xcite , a manually annotated ne corpus for croatian @xcite , ne recognition in slovene @xcite , a czech corpus of 11k annotated nes @xcite , ner for czech @xcite , tools and resources for fine-grained annotation of nes in the national corpus of polish @xcite , ner shared tasks for polish organized under the umbrella of poleval foot_1 (ogrodniczuk and łukasz kobyliński, 2018, 2020) and leszcze foot_2 campaigns, recent shared tasks on ne recognition in russian @xcite , the latter utilizing the nerel dataset (a russian dataset for named entity recognition and relation extraction, described in @xcite , and semeval 2022 task 11: multiconer multilingual complex named entity recognition foot_3 and semeval 2023 task 2: multiconer ii multilingual complex named entity recognition, foot_4 which included russian and ukrainian respectively. ## task description the data for this edition of the shared task consists of a set of documents in three slavic languages: czech, polish and russian. to facilitate entity linking, the set of documents is chosen to involve one specific event. the documents were obtained from the web, by posing keyword queries to search engines, or publicly available crawled data repositories, and extracting the textual content from the respective sources. the task is to recognize, classify, and "normalize" all named-entity mentions in each of the documents, and to link across languages all named mentions referring to the same real-world entity. formally, the multilingual named entity recognition task is subdivided into three sub-tasks: these tasks do not require positional information of the name entity mentions. thus, for all occurrences of the same form of a ne mention (e.g., an inflected variant, an acronym or abbreviation) within a given document, no more than one annotation should be produced. 6 furthermore, distinguishing typographical case is not necessary since the evaluation is case-insensitive. if the text includes lowercase, uppercase or mixed-case variants of the same entity, the system should produce only one annotation for all of these mentions. for instance, for "uefa" and "uefa" (provided that they refer to the same ne type foot_6 ), only one annotation should be produced. the recognition of common-noun or pronominal references to named entities is not included as part of the task. ## data in this edition of the challenge the annotated datasets from previous editions were used as training data. in particular, the training and test datasets annotated in bulgarian, czech, polish and russian from 2019 shared task @xcite and training and test datasets annotated in bulgarian, czech, polish, russian, slovene and ukrainian from 2021 shared task (piskorski et al., 2021) were used. the prior datasets annotated in six languages covered various major topics, including, i.a., the covid-19 pandemic, the 2020 usa presidential elections (usa 2020 elections), asia bibi, which relates to a pakistani woman involved in a blasphemy case, brexit, ryanair, which faced a massive strike, and nord stream, a controversial russian-european project. the test data for the current edition of the challenge involves the russia-ukraine war. each of the datasets, including the latest test data, was created as follows. for the focus entity/event, we posed a search query to google and/or publicly available crawled data repositories, in each of the target languages. the query returned documents in the target language. we removed duplicates, downloaded the htmlmainly news articles-and converted them into plain text. since the result of html parsing may include not only the main text of a web page, but also spurious text, some additional manual cleaning was applied when necessary. the resulting set of "cleaned" documents were used to manually select documents for each language and topic for the final datasets. documents were annotated using the inforex foot_8 web-based system for annotation of text corpora @xcite . inforex allows parallel access and resource sharing by multiple annotators. it let us share a common list of entities, and perform entity-linking semi-automatically: for a given entity, an annotator sees a list of entities of the same type inserted by all annotators and can select an entity id from the list. a snapshot of the inforex interface is in figure 1 . in addition, inforex keeps track of all lemmas and ids inserted for each surface form, and inserts them automatically, so in many cases the annotator only confirms the proposed values, which speeds up the annotation process a great deal. all annotations were made by native speakers. after annotation, we performed multiple phases of automatic and manual consistency checks, to reduce annotation errors, especially in entity linking. the training data statistics are shown in the participants received the test dataset russia-ukraine war-and were given circa 2 days to return up to 10 system responses. the topic was not announced in advance, and the annotations were not released. the rationale behind this decision was to motivate the participants to build a general solution for slavic ner, rather than to optimize their models toward particular scenarios or sets of names. ## evaluation methodology the ner task (exact case-insensitive matching) and name normalization (or "lemmatization") were evaluated in terms of precision, recall, and f 1 measure. for ner, two types of evaluations were carried out: in the relaxed evaluation we additionally distinguish between exact and partial matching: in the latter case, an entity mentioned in a given document is considered to be extracted correctly if the system response includes at least one partial match of a named mention of this entity. we evaluate the systems at several levels of granularity: we measure the performance (a) for all ne types and all languages, (b) for each given ne type and all languages, (c) for all ne types for each language, and (d) for each given ne type per language. in the name normalization task, we take into account only correctly recognized entity mentions and only those that were normalized (on both the annotation and the response system's sides). formally, let n correct denote the number of all correctly recognized entity mentions for which the system returned a correct base form. let n key denote the number of all normalized entity mentions in the gold-standard answer key and n response denote the number of all normalized entity mentions in the system's response. we define precision and recall for the name normalization task as: in evaluating document-level, single-language and cross-lingual entity linking we adopted the link-based entity-aware (lea) metric (moosavi and strube, 2016), which considers how important the entity is and how well it is resolved. lea is defined as follows. let @xmath0, k 2 , . . . , k |k| } denote the set of key entities and @xmath1, r 2 , . . . , r |r| } the set of response entities, i.e., k @xmath2. lea recall and precision are then defined as follows: lea brings several benefits. for example, lea considers resolved co-reference relations instead of resolved mentions and has more discriminative power than other metrics for co-reference resolution @xcite . the evaluation was carried out in "caseinsensitive" mode: all named mentions in system response and test corpora were lower-cased. ## participant systems out of the seven registered teams, we received results from three. further, two of these teams provided papers describing the details of their systems, presented in the 2023 slavic nlp workshop. we briefly review these systems here; for complete descriptions, please see the corresponding papers. the tilde system, (rinalds vīksna and rozis, 2023), utilizes the multilingual xlm-r model to perform all subtasks. they enhance their training set by incorporating diverse ner datasets, in addition to the slavic ner challenge training set. the authors fine-tune five different variants of the xlm-r large @xcite model that differ in the approach for the entity-linking subtask. for each variant, they use slightly different training datasets. in addition, one of the variants is an ensemble of five xlm-r base models, one for each of the five ner entity labels. the base model was initially pre-trained on 2.6 gb of recent czech, polish and russian news articles to integrate into the model new entities and events, which have emerged since the original model was trained. this process enables the model to embed the latest information and keep up-to-date with the evolving language usage. the amu system (pałka and nowakowski, 2023) combines a set of transformer-based models for named entity recognition, categorization, and lemmatization. they evaluated several monolingual (herbert, czert, and rubert) and multilingual (slavic-bert and xlm-roberta) bert-like models for entity recognition and categorization. for entity lemmatization, sequence-tosequence (seq2seq) models were applied, plt5 and mt5. the pre-trained models were fine-tuned on the dataset provided within the shared task and additional external resources, including datasets annotated with named entities: collection3, multi-nerd, polyglot-ner, wikineural; dictionaries of lemmatized named entities and multi-word expressions: sejf, sejfek, poleval 2019 task 2. the additional resources for lemmatization were only for polish. thus, the authors used opus-mt to translate the resources to other languages to overcome the language limitation. the third team-ctc, cognitive technologies center-submitted results, but did not provide a description paper; their approach was similar to the one employed by this team in the 2021 edition of the shared task @xcite . ## evaluation results only the tilde team submitted results for crosslingual entity linking, achieving 66.9% f 1 score. this is a great improvement compared to the third challenge, where the best results were around 50% of f 1 score. to date, the task of cross-lingual linking remains much more challenging than the task of entity extraction. note that in our setting, the performance of entity linking depends on the performance of name recognition : each system had to link entities that it had extracted from documents upstream rather than link a set of correct entities. in ## conclusion this paper reports on the 4 th multilingual named entity challenge focusing on recognizing mentions of nes in web documents in three slavic languages, normalization of the nes, and crosslingual entity linking. seven teams registered and three of them actively participated in the challenge and submitted system results with multiple variants. most systems use state-of-the-art transformer-based mod-els. overall, the results of the best-performing systems are quite strong for extraction and normalization of names, while entity linking-and in particular, cross-lingual entity linking-remains a very challenging task. we present the summary results for the main aspects of the challenge and the best-performing model from each team. to foster further research into nlp for slavic languages, including cross-lingual entity linking, our training and test datasets, the detailed annotations, and scripts used for the evaluations are made available to the research community on the shared task's web page. 10