W iki NE u R al: C ombined Neural and Knowledge-based Silver Data Creation for Multilingual NER (2021)

Abstract: Multilingual Named Entity Recognition (NER) is a key intermediate task which is needed in many areas of NLP. In this paper, we address the well-known issue of data scarcity in NER, especially relevant when moving to a multilingual scenario, and go beyond current approaches to the creation of multilingual silver data for the task. We exploit the texts of Wikipedia and introduce a new methodology based on the effective combination of knowledge-based approaches and neural models, together with a novel domain adaptation technique, to produce high-quality training corpora for NER. We evaluate our datasets extensively on standard benchmarks for NER, yielding substantial improvements up to 6 span-based F1-score points over previous state-of-the-art systems for data creation.

## introduction named entity recognition (ner) is the task of identifying specific words as belonging to predefined semantic types, such as person, location, organization, etc. @xcite . ner is widely used in many downstream tasks, like question answering @xcite , machine translation @xcite , information retrieval @xcite ), text summarization @xcite ), text understanding @xcite and entity linking @xcite , among others. with recent advances in natural language processing, and in particular with the advent of pretrained language models such as bert @xcite , once a sufficient amount of training data is available for the task of interest, fine-tuning is often employed to address the task successfully. unfortunately, such training data are scarce and expensive to create, especially when labels are finegrained and many languages have to be covered, as is the case for ner. various works have been put forward which address data paucity by aiming at automatically producing multilingual silver-standard training data for ner @xcite @xcite . each of these leverages the link structure of wikipedia to generate named entity annotations. however, this strategy has two drawbacks: only small portions of text in wikipedia are linked, and mapping wikipedia links to the corresponding ner classes is not trivial and introduces errors. different methods have been investigated to cope with these problems, such as heuristics based on wikipedia redirects, surface form token matching, and category-based rules. although we also rely on wikipedia text and its hypertext organization, we depart from previous works in our exploration of new languageindependent techniques for silver data creation for ner by providing a general approach based on an effective combination of knowledge-based techniques and neural models. our contributions are as follows: we release data and software at https://github. com/babelscape/wikineural . ## related work since the first shared task on ner organized by @xcite , several tasks and human-annotated datasets have been proposed. the @xcite datasets @xcite were created in four different languages (spanish, dutch, english, and german) from newswire articles and focused on 4 entity types: per (person), org (organization), loc (location), and misc (miscellaneous, i.e., all other entity types). several other ner shared tasks were organized in the years which followed, covering further languages such as indic @xcite and balto-slavic languages @xcite . early ner systems were based on domainspecific features and rules, which require human engineering. starting from @xcite , neural ner architectures requiring minimal feature engineering have become enduringly popular @xcite . nevertheless, to fully benefit from these systems, large amounts of data for training are required. although various ner datasets have been created, they have remained a scarce resource, available only for a narrow set of languages. moreover, they have often been small in size, limited to a few domains, and genre-specific (e.g., news). for these reasons, over the last two decades, various works have been carried out to turn wikipedia texts into multilingual ner training corpora. @xcite introduced wikiner, a pipeline to automatically create multilingual training data for ner by exploiting the structure and the texts of wikipedia. first, they classified each wikipedia document into named entity types, training and evaluating on manually-labeled wikipedia articles across 9 languages. then, wikipedia links were converted into labels by classifying the target articles into entity types (per, org, loc, misc). finally, heuristics based on redirects were applied to infer more named entity mentions. interestingly, @xcite showed that, when testing on manually-annotated wikipedia sentences, models trained on gold-standard newswire datasets perform poorly compared to models trained on automatically-created wikipedia corpora. similarly, @xcite proposed wikiann, a language-independent framework to automatically extract name mentions from documents by leveraging wikipedia markups. specifically, they first classified english wikipedia entries into cer-tain entity types, and then they applied a crosslingual entity transfer to propagate these labels to other languages. other works relied on freebase @xcite , a sizeable collaborative graph database, either by using its association to english wikipedia as a training feature @xcite , or by mapping its attributes to entity types, in order to identify the ner classes @xcite . moreover, @xcite also tried to overcome the issue of missing annotations of non-anchored mentions in wikipedia by using a simple surface stringmatching heuristic, and resampled the datasets they produced to reduce the high class imbalance. in our work we follow this same direction but introduce several contributions based on a novel combination of knowledge-based and neural techniques that lead to considerable improvements. to the best of our knowledge, we are the first to exploit multilingual bert's power in a silver data creation process for ner: we use it to independently i) distinguish named entities from concepts, ii) validate annotations and, iii) discover annotations. further, to address the sparsity problem, rather than relying on often-noisy redirections, we exploit the synonymy information provided by a multilingual lexical knowledge base, i.e., babel-net foot_0 @xcite . ## wikineural we now describe our approach to producing multilingual silver-standard training data for named entity recognition. a graphical representation of the steps that characterize the wikineural annotation pipeline is depicted in figure 1 . ## domain adaptation thanks to the use of wikipedia, our automaticallycreated datasets cover a wide range of domains. however, in many cases tests are performed on a limited set of domains. to address this issue and enable the production of domain-fitting ner datasets, here we introduce our methodology for performing domain adaptation. this consists of a domain extraction technique which, later combined with the approach presented in section 3, enables the creation of domain-adapted training data for ner systems when given domain-specific texts. ## ablation study in order to show the effectiveness of the steps detailed in section 3, we disassembled our ner data creation pipeline. we conducted these experiments on the english wikineural corpus; results are shown in we can summarize this ablation study by pointing out an average gap of more than 21 f 1 -score points between the final wikineural version detailed in section 3 and the basic one, which only uses strings anchored in wikipedia articles. for completeness, we also constructed a baseline version of the wikineural dataset using just the neural model employed in section 3.5.2 to annotate wikipedia articles. the system trained on the resulting dataset achieved 69.46 ± 0.50 on conll, 77.46 ± 0.43 on wikigold and 64.53 ± 0.41 on ontonotes, showing how the combination of neural and knowledge-based approaches adopted by the final version of wikineural is essential in order to achieve higher performances. ## conclusion we presented wikineural, an automatic, language-independent approach for generating labeled datasets for ner in multiple languages. while we follow other silver-data creation approaches and exploit the hyperlinked texts of wikipedia articles, we depart from past works in three fundamental aspects which integrate knowledge-based and neural approaches: i) we automatically type tags by utilizing the structure of a multilingual lexical-semantic knowledge base, babelnet, ii) we exploit neural bert-based models to discern entity from non-entity tags and iii) as a complementary approach to confirm and augment sentences with entity tags, iv) we put forward a domain adaptation technique which can produce ner training data for arbitrary domains. we finally showed, through an extensive evaluation, that wikineural can be used to train competitive ner systems, providing substantial performance improvements over previous state-of-the-art approaches for silver-data creation. as future work, we plan to extend our study to produce named entity tags for a larger set of classes and languages.