Universal NER : A Gold-Standard Multilingual Named Entity Recognition Benchmark (2024)

Abstract: We introduce Universal NER (UNER), an open, community-driven project to develop gold-standard NER benchmarks in many languages. The overarching goal of UNER is to provide high-quality, cross-lingually consistent annotations to facilitate and standardize multilingual NER research. UNER v1 contains 19 datasets annotated with named entities in a cross-lingual consistent schema across 13 diverse languages. In this paper, we detail the dataset creation and composition of UNER; we also provide initial modeling baselines on both in-language and cross-lingual learning settings. We will release the data, code, and fitted models to the public.

## introduction high-quality data in many languages is necessary for broadly multilingual natural language processing. in named entity recognition (ner), the majority of annotation efforts are centered on english, and cross-lingual transfer performance remains brittle (e.g., @xcite . amongst non-english human-annotated ner datasets, while there have been multiple separate efforts in this front (e.g., @xcite @xcite , these either have disjoint annotation schemes and labels, cover a single language or small set of related languages, or are not widely accessible (e.g., @xcite . for most of the world's languages, the only readily available ner data is the automatically annotated wikiann dataset @xcite , though this annotation paradigm introduces data quality issues and limits its usefulness for evaluation @xcite . to address this data gap, we propose universal ner (uner), an open community effort to 1 https://www.universalner.org , uner v1 available at https://doi.org/10.7910/dvn/gq8hdl ## dataset design principles named entity recognition (ner) is the task of identifying text spans in a given context that uniquely refer to specific named entities. the task of ner has a long tradition @xcite and facilitates many downstream nlp applications, such as information retrieval @xcite and question answering @xcite . furthermore, successful ner tagging requires a model to reason about semantic and pragmatic world knowledge, which makes the task an informative evaluation setting for testing nlp model capabilities. as with universal dependencies, the goal of universal ner is to develop an annotation schema that can work in any language. traditionally, the ud @xcite and upos @xcite projects have chosen what amounts to the intersection of tags across all language-specific tagsets, keeping the resultant tagset broad and simple. we follow a similar strategy, picking tags that broadly cover the space of proper nouns. universal ner's annotation schema emphasizes three coarse-grained entity types: person (per), organization (org), and location (loc). we provide a short description and an example for each tag. annotation guidelines in preparation for annotation, we developed extensive annotation guidelines, foot_0 using the norne project guidelines (jørgensen et al., 2020) as a starting point. along with tag descriptions, our guidelines include many examples, as well as instructions for dealing with ambiguity and unclear constructions, such as email addresses, pet names, and typographical errors. we expect that the guidelines will be further refined and updated as annotation proceeds. to manage this, we track version numbers and changelogs for different iterations of the guidelines. each data release will include the corresponding annotation guidelines at the time of release. ## dataset annotation process having described the theoretical basis for the tagset, we now discuss the mechanics of annotation. ## universal ner: statistics and analysis this section presents an overview of the universal ner (uner) dataset. uner v1 adds a ner annotation layer to 19 datasets (primarily treebanks from ud). it covers 13 geneologically and typologically diverse languages: cebuano, danish, german, english, croatian, narabizi, portuguese, russian, slovak, serbian, swedish, tagalog, and chinese 4 . overall, uner v1 contains ten full datasets with training, development, and test splits over nine languages, three evaluation sets for lower-resource languages (tl and ceb), and a parallel evaluation benchmark spanning six languages. ## conclusion we introduce universal ner (uner), a goldstandard data initiative covering 13 languages for named entity recognition (ner). the datasets included in uner v1 cover a wide variety of domains and language families, and we establish initial performance metrics for these benchmarks. uner opens several opportunities for research in ner outside of english and for cross-lingual transfer; in particular, this project provides humanannotated and standardized evaluations for multilingual ner. after releasing the current version of the uner project, we plan to expand language coverage and diversity of this effort by both recruiting additional annotators and integrating existing ner datasets when possible. this will also allow us to obtain more robust agreement measures and verify the quality of existing annotations in uner. in the longer term, our aims for universal ner include rigorous quality checking of annotation results for robustness and further integration of finetuned models and data analysis tools into the project.