Slav- NER : the 3rd Cross-lingual Challenge on Recognition, Normalization, Classification, and Linking of Named Entities across S lavic Languages (2021)

Abstract: This paper describes Slav-NER: the 3rd Multilingual Named Entity Challenge in Slavic languages. The tasks involve recognizing mentions of named entities in Web documents, normalization of the names, and cross-lingual linking. The Challenge covers six languages and five entity types, and is organized as part of the 8th Balto-Slavic Natural Language Processing Workshop, co-located with the EACL 2021 Conference. Ten teams participated in the competition. Performance for the named entity recognition task reached 90% F-measure, much higher than reported in the first edition of the Challenge. Seven teams covered all six languages, and five teams participated in the cross-lingual entity linking task. Detailed valuation information is available on the shared task web page.

## introduction analyzing named entities (nes) in slavic languages poses a challenging problem, due to the rich inflection and derivation, free word order, and other morphological and syntactic phenomena exhibited in these languages @xcite . encouraging research on detection and normalization of nes-and on the closely related problem of cross-lingual, crossdocument entity linking-is of paramount importance for improving multilingual and cross-lingual information access in these languages. this paper describes the 3 rd shared task on multilingual ne recognition (ner), which aims at addressing these problems in a systematic way. the shared task was organized in the context of the 8 th bsnlp: balto-slavic natural language processing workshop, co-located with the eacl 2021 conference. the task covers six languages-bulgarian, czech, polish, russian, slovene and ukrainian-and five types of ne: person, location, organization, product, and event. the input text collection consists of documents collected from the web, each collection centered on a certain "focal" event. the rationale of such a setup is to foster the development of "end-to-end" ner and cross-lingual entity linking solutions, which are not tailored to specific, narrow domains. this paper also serves as an introduction and a guide for researchers wishing to explore these problems using the training and test data, which are released to the public. 1 the paper is organized as follows. section 2 reviews prior work. section 3 describes the task; section 4 describes the annotation of the dataset. the evaluation methodology is introduced in section 5. participant systems are described in section 6, and the results obtained by these systems are presented in section 7. we present the conclusions and lessons learned in section 8. ## prior work the work described here builds on the 1 st and 2 nd shared task on multilingual named entity recognition, normalization and cross-lingual match-ing for slavic languages, @xcite , which, to the best of our knowledge, are the first attempts at such shared tasks covering multiple slavic languages. high-quality recognition and analysis of nes is an essential step not only for information access, such as document retrieval and clustering, but it also constitutes a fundamental processing step in a wide range of nlp pipelines built for higher-level analysis of text, such as information extraction, see, e.g. @xcite . other ner-related shared tasks have been organized previously. the first non-english monolingual ner evaluations-covering chinese, japanese, spanish, and arabic-were held in the context of the message understanding conferences (mucs) @xcite and the ace programme @xcite . the first multilingual ner shared task, which covered several european languages, including spanish, german, and dutch, was organized in the context of the conll conferences @xcite . the ne types covered in these campaigns were similar to the ne types covered in our challenge. worth mentioning in this context is entity discovery and linking (edl) @xcite , a track of the nist text analysis conferences (tac). edl aimed to extract entity mentions from a collection of documents in multiple languages (english, chinese, and spanish), and to partition the entities into cross-document equivalence classes, by either linking mentions to a knowledge base or directly clustering them. an important difference between edl and our task is that edl required linking entities to a pre-existing knowledge base. related to cross-lingual ne recognition is ne transliteration, i.e., linking nes across languages that use different scripts. a series of ne transliteration shared tasks were organized as a part of news-named entity workshops- @xcite , focusing mostly on indian and asian languages. in 2010, the news workshop included a shared task on transliteration mining @xcite , i.e., mining of names from parallel corpora. this task included corpora in english, chinese, tamil, russian, and arabic. research on ne focusing on slavic languages includes tools for ne recognition for croatian @xcite , ne recognition in croatian tweets @xcite , a manually annotated ne corpus for croatian @xcite , tools for ne recognition in slovene @xcite , a czech corpus of 11k annotated nes @xcite , ner tools for czech @xcite , tools and resources for fine-grained annotation of nes in the national corpus of polish @xcite , ner shared tasks for polish organized under the umbrella of poleval foot_1 evaluation campaigns @xcite . and a recent shared task on ne recognition in russian @xcite . ## task description the data for this edition of the shared task consists of sets of documents in six slavic languages: bulgarian, czech, polish, russian, slovene and ukrainian. to accommodate entity linking, each set of documents is chosen to revolve around one certain entity-e.g., a person, an organization or an event. the documents were obtained from the web, by posing a keyword query to a search engine or publicly available crawled data repositories, and extracting the textual content from the respective sources. the task is to recognize, classify, and "normalize" all named-entity mentions in each of the documents, and to link across languages all named mentions referring to the same real-world entity. formally, the multilingual named entity recognition task is subdivided into three sub-tasks: the task does not require positional information of the name entity mentions. thus, for all occurrences of the same form of a ne mention (e.g., an inflected variant, an acronym or abbreviation) within a given document, no more than one annotation should be produced. 3 furthermore, distinguishing typographical case is not necessary since the evaluation is case-insensitive. if the text includes lowercase, uppercase or mixed-case variants of the same entity, the system should produce only one annotation for all of these mentions. for instance, for "isis" and "isis" (provided that they refer to the same ne type), only one annotation should be produced. the recognition of commonnoun or pronominal references to named entities does not constitute part of the task. ## data for russian, polish, czech and bulgarian, the training and test data sets from the 2019 shared task were used as training data for 2021. for the new languages-ukrainian and slovene-new training sets were annotated. the test data in all six languages covered two major current topics: the covid-19 pandemic and the 2020 usa presidential elections (usa 2020 elections). the 2019 training data consist of four sets of documents extracted from the web, each related to a given focus entity. we tried to choose entities related to events in 2018 and 2019 covered in mainstream news in many languages. asia bibi, which relates to a pakistani woman involved in a blasphemy case, brexit, ryanair, which faced a massive strike, and nord stream, a controversial russian-european project. each dataset was created as follows. for the focus entity, we posed a search query to google and/or publicly available crawled data repositories, in each of the target languages. the query returned documents in the target language. we removed duplicates, downloaded the htmlmainly news articles-and converted them into plain text. since the result of html parsing may include not only the main text of a web page, but also spurious text, some additional manual cleaning was applied whenever necessary. the resulting set of "cleaned" documents were used to manually select documents for each language and topic, for the final datasets. documents were annotated using the inforex foot_4 web-based system for annotation of text corpora @xcite . inforex allows parallel access and resource sharing by multiple annotators. it let us share a common list of entities, and perform entity-linking semi-automatically: for a given entity, an annotator sees a list of entities of the same type inserted by all annotators and can select an entity id from the list. a snapshot of the inforex interface is in figure 1 . in addition, inforex keeps track of all lemmas and ids inserted for each surface form, and inserts them automatically, so in many cases the annotator only confirms the proposed values, which speeds up the annotation process a great deal. all annotations were made by native speakers. after annotation, we performed automatic and manual consistency checks, to reduce annotation errors, especially in entity linking. training and test data statistics are presented in the testing datasets-covid-19 and usa 2020 elections-were released to the participants who were given circa 2 days to return up to 5 system responses. the participants did not know the topics in advance, and did not receive the annotations. the main drive behind this decision was to push participants to build a general solution for slavic ner, rather than to optimize their models toward a particular set of names. ## evaluation methodology the ner task (exact case-insensitive matching) and name normalization (or "lemmatization") were evaluated in terms of precision, recall, and f1-measure. for ner, two types of evaluations were carried out: in relaxed evaluation we additionally distinguish between exact and partial matching: in the latter case, an entity mentioned in a given document is considered to be extracted correctly if the system response includes at least one partial match of a named mention of this entity. we evaluate systems at several levels of granularity: we measure performance for (a) all ne types and all languages, (b) each given ne type and all languages, (c) all ne types for each language, and (d) each given ne type per language. in the name normalization task, we take into account only correctly recognized entity mentions and only those that were normalized (on both the annotation and system's sides). formally, let n correct denote the number of all correctly recognized entity mentions for which the system returned a correct base form. let n key denote the number of all normalized entity mentions in the gold-standard answer key and n response denote the number of all normalized entity mentions in the system's response. we define precision and recall for the name normalization task as: in evaluating document-level, single-language and cross-lingual entity linking we adopted the link-based entity-aware metric (lea) (moosavi and strube, 2016), which considers how important the entity is and how well it is resolved. lea is defined as follows. let @xmath0, k 2 , . . . , k |k| } denote the set of key entities and @xmath1, r 2 , . . . , r |r| } the set of response entities, i.e., k @xmath2. lea recall and precision are then defined as follows: lea brings several benefits. for example, lea considers resolved co-reference relations instead of resolved mentions and has more discriminative power than other metrics for co-reference resolution @xcite . the evaluation was carried out in "caseinsensitive" mode: all named mentions in system response and test corpora were lower-cased. ## participant systems six teams submitted descriptions of their systems as bsnlp workshop papers. we briefly review these systems here; for complete descriptions, please see the corresponding papers. two additional teams submitted their results with short descriptions of their systems, which appear in this section. the ul fri system, @xcite , generated results for several settings, models and languages, although the team's main motivation is to develop effective ner tools for slovenian. the system uses contemporary bert and roberta multilingual pre-trained models, which include slovene among other languages. the system was further trained on the slavner dataset for the ner task and used the dedupe method for the entity matching task. the best performing models were pre-trained on slovene. the results also indicate that two-step prediction of ne could be beneficial. the team made their code publicly available. the priberam labs system, @xcite , focuses on the ner task. it uses three components: a multilingual contextual embedding model, a character-level embedding model, and a bi-affine classifier model. the paper reports results for different multilingual contextual embedding models, which included multilingual bert, xlm-roberta, or the slavic bert. for different languages the best-performing models where different, but having the same language within the large pre-trained model usually improved the results-e.g., slavic bert, which used additional resources for bulgarian, russian and polish, also performed best for these languages. the system uses heuristics to predict and resolve spans of nes, and in this way it is able to tag overlapping entities. the code for the system is made available. the tld system, @xcite , uses a staged approach. the first stage is identification of nes in context, which is treated as a sequence labeling problem and is performed by a multilingual bert model from google, modified by the team. entity linking is the second stage, which uses a list of labse embeddings; matched entries need to pass a pre-defined threshold of cosine similarity with existing entries; otherwise they are added as new values to the list. the third stage is normalisation of identified entities, which is performed using models provided with stanza. the l3i system, @xcite , combines bert models with the "frustratingly easy" domain adaptation algorithm. it also uses other techniques to improve system's ner performance, such as marking and enrichment of uppercase tokens, prediction of ne boundaries with a multitask approach, prediction of masked tokens, fine-tuning the language model to the domain of the document. the traspas system, @xcite , tests the assumption that the universal open-source nlp toolkits (such as spacy, stanza or trankit) could achieve competitive performance on the multilingual ner task, using large pretrained transformer-based language models available from huggingfacetransformers, which have not been available in previous editions of the shared task. the team tests the generalizability of the models to new low-resourced domains, and to languages such as slovene and ukrainian. the uwr-vl system, @xcite , utilizes large collections of unstructured and structured documents for unsupervised training of embedding of lexical units and for recog-nizing and linking multiple real-world nes. in particular, the team makes use of commoncrawl news articles, wikipedia, and its structured counterpart wikidata as knowledge sources, to address the problem of data scarcity, building neural gazetteer via collecting different embeddings from these knowledge sources. the system further uses standard neural approaches to the ner task, with a rnn classifier, in order to determine for every input word the probability of labelling it with various beginning and end ne tags. two more systems generated the results for the shared task-ctc-ner from the cognitive technologies center team, and paisc_wxd: ctc-ner is a baseline prototype of a ner component of an entity recognition system currently under development at the cognitive technologies center. the system has a hybrid architecture combining rule-based and ml techniques; the ml-component is loosely related to @xcite . the languages currently processed include russian, english and ukrainian. paisc_wxd uses the xlm-roberta model, followed by bilstm-crf on top. in addition, the system uses data enhancement based on machine translation. ## evaluation results figure 3 shows the performance of the systems averaged across all languages and both test corpora. for each team that provided a solution for all six languages (7 teams except ctc-ner), we present the best scores (f 1, p recision, and recall) obtained by the team in three evaluation modes. 6 as the plots show, the best performing model, priberam, yields f-measure 85.7% according to the relaxed partial evaluation, and 79.3% according to the strict evaluation. the priberam submission scores highest in precision -89,4% relaxed partial, and 85.1% strict -but much lower in recall -82.2% relaxed partial, and 74.3% strict. among the teams that submitted results for cross-lingual entity linking, only two achieved results comparable with the benchmarks achieved on the second challenge, and this year's results surpass those benchmarks by a substantial margin. the best results for each team, averaged across two corpora, are shown in note that in our setting the performance on entity linking depends on the performance on name recognition and normalization: each system had to link entities that it had extracted from documents upstream, rather than link a set of correct entities. tables 4 and 5 present the f1-measures separated by language, for all tasks for the covid-19 and usa 2020 elections data sets these tables show only the top-performing model for each team. for recognition, we show only the relaxed evaluation, since the results obtained on the three evaluation schemes are correlated, as can be seen from figure 3 . the tables indicate some variation in scores obtained on the test corpora this variation could be due to a number of factors, including actual differences in the test data, as well as differences in annotation across languages. this variation should and will be investigated in greater depth. in ## conclusion this paper reports on the 3 rd multilingual named entity challenge focusing on recognizing mentions of nes in web documents in six slavic languages, normalization of the nes, and crosslingual entity linking. the challenge has attracted substantial interest, following the prior challenges in 2017 and 2019, with 10 teams registering for the competition and eight teams submitting results from working systems, with multiple variants. most systems use state-of-the-art neural network models. overall, the results of the bestperforming systems are quite strong for extraction and normalization, while cross-lingual linking is the most challenging of the tasks. we show summary results for the main aspects of the challenge and the best-performing model for each team. for detailed, in-depth evaluations of all participating systems and their performance please consult the shared task's web page and the papers by the respective teams. to stimulate further research into nlp for slavic languages, including cross-lingual entity linking, our training and test datasets, the detailed annotations, and scripts used for evaluations are made available to the public on the shared task's web page. 7 the annotation interface is released by the inforex team, to support further annotation of additional data for future tests. this challenge covered six slavic languages. for future editions of the challenge, we plan to expand the data sets, covering a wider range of entity types, and supporting cross-lingual entity linking. we plan to expand the training and test data to include non-slavic languages. we will also undertake further refinement of the underlying annotation guidelines-a highly complex task in a real-world setting. more complex phenomena also need to be addressed, e.g., coordinated nes, contracted versions of multiple nes, etc. we believe that the reported results and the annotated datasets will help stimulate further research on robust, end-to-end analysis of real-world texts in slavic languages.