Fine-grained Named Entity Annotation for F innish (2021)

Abstract: We introduce a corpus with fine-grained named entity annotation for Finnish, following the OntoNotes guidelines to create a resource that is cross-lingually compatible with existing annotations for other languages. We combine and extend two NER corpora recently introduced for Finnish and revise their custom annotation scheme through a combination of automatic and manual processing steps. The resulting corpus consists of nearly 500,000 tokens annotated for over 50,000 mentions categorized into the 18 OntoNotes name and numeric entity types. We evaluate this resource and demonstrate its compatibility with the English OntoNotes annotations by training state-of-the-art mono-, bi- and multilingual deep learning models, finding both that the corpus allows highly accurate recognition of OntoNotes types at 93% F-score and that a comparable level of tagging accuracy can be achieved by a bilingual Finnish-English NER model.

## introduction named entity recognition (ner), the identification and typing of text spans referring to entities such as people and organizations in text, is a key task in natural language processing. state of the art ner approaches apply supervised machine learning methods trained on corpora that have been manually annotated for mentions of entity names of interest. while extensive corpora with fine-grained ner annotation have long been available for high-resource languages such as english, ner for many lesser-resourced languages has been limited by smaller, lower-coverage corpora with comparatively coarse annotation. a degree of language independence has long been a central goal in ner research. one notable example are the conll shared tasks on language-independent named entity @xcite @xcite . the spanish, dutch, english and german datasets introduced in these shared tasks were all annotated for the same types of entity mentions -persons, organizations, locations, and miscellaneous -and the datasets still remain key benchmarks for evaluating ner methods today (e.g. @xcite ). nevertheless, until recently most ner methods aimed for language independence only in that they supported training on corpora of more than one language, resulting in multiple separate monolingual models. in recent years, advances in deep learning have made it possible to create multilingual language models that achieve competitive levels of performance when trained and applied on texts representing more than one language (e.g. @xcite ). one notable model is the multilingual version of the influential bert model @xcite , mbert, trained on more than 100 languages. mbert performs well on zero-shot cross-lingual transfer experiments, including ner experiments @xcite . @xcite propose an mbert-based model trained simultaneously on multiple languages. training and validating on the ontonotes v5.0 corpus (see section 2.3) and the conll datasets, they show that multilingual models outperform models trained on one single language and have cross-lingual zero-shot ability. the zeroshot cross-lingual transfer ability of mbert also spikes interest in the study of multilingual representations, both on mbert @xcite , and on multilingual encoders in general @xcite @xcite . ## data in the following, we introduce the corpora used in this study, additional text sources for the new corpus, and the pre-trained models used in our experiments. the properties and key statistics of the corpora are presented in ## annotation we next summarize the primary steps performed to revise and extend the annotation of the two source corpora to conform with the ontonotes ner guidelines @xcite . figure 1 shows visualizations of the annotation for selected sentences. trivial mappings of the mentions annotated in the existing finnish ner corpora, effectively all annotations with the type per are valid ontonotes person annotations. similarly, most event and date annotations were valid as-is as ontonotes annotations of the same names. these annotations were carried over into the initial revised data, changing only the type name when required. conditional mappings by contrast to the types allowing trivial mapping from existing to revised annotation, loc, org and pro required more complex mapping rules. for example, the existing annotations mark both geo-political entities (gpes) and other locations with the type loc without distinguishing between the two. to create ontonotes-compatible annotation, source loc annotations were mapped to either loc or gpe annotations on the basis of the annotated text using manually created rules. for example, suomi/loc ("finland") was mapped to suomi/gpe and välimeri/loc ("mediterranean") to välimeri/loc. similar rules were implemented to distinguish e.g. fac from org and loc as well as work of art and law from pro. dictionary-based tagging not all mentions in scope of the ontonotes annotation guidelines are in scope of the finer annotation guidelines applied to mark the previously introduced finnish ner corpora. in addition to most ontonotes numeric types (see below), in particular nationalities, religious and political groups (norp in ontonotes) and languages (language) were not annotated in the source corpora. to create initial ontonotes annotation for these semiclosed categories of mentions, we performed dictionary-based tagging using lists compiled from sources such as wikipedia and manually translated ontonotes english terms tagged with the relevant types. 4 numeric types to annotate ontonotes numeric types (cardinal, ordinal, etc.) in the turku ner corpus section of the data, we mapped the manual part-of-speech and feature annotation of the source corpus (tdt) to initial annotations that were then manually revised to identify the more specific types such as percent, quantity and money based on context. for the finer texts, annotation for these types followed a similar process with the exception that automatic part-of-speech and feature annotation created by the turku neural parser @xcite was used as a starting point as no manual syntactic annotation was available for the texts. fine-grained tokenization the finer annotation guidelines specify that annotated name mensemi-automatic and manual revision after initial automatic revisions, a series of semiautomatic and manual revision rounds were performed using the brat annotation tool @xcite . in particular, the consistency of mention annotation and typing was checked using the search functionality of the tool foot_5 and all cases where a string was inconsistently marked or typed were revisited and manually corrected when in error. additionally, the automatically created pre-annotation for the newly added text (section 2.4) was revised and corrected in a full, manual annotation pass. all manual revisions of the data were performed by a single annotator familiar with the corpora as well as the finer and ontonotes guidelines. while the single-annotator setting regrettably precludes us from reporting inter-annotator agreement, our monolingual and cross-lingual results below suggest that the consistency of the annotation has not decreased from that of the source corpora. ## methods we next present the applied ner method and detail the experimental setup. ## results we next present summary statistics of the newly introduced corpus and then present the results of the machine learning experiments. ## discussion and conclusions we have introduced a new corpus for finnish ner created by combining and extending two previously released corpora, finer and the turku ner corpus, and by mapping their custom annotations into the fine-grained ontonotes representation through a combination of automatic and manual processing steps. the resulting corpus consists of over 50,000 annotations for nearly 500,000 tokens of text representing a broad selection of genres, topics and text types, and is not only the largest resource for finnish ner created to date, but also identifies three times as many distinct name and numeric entity mention types as the previously introduced finnish ner corpora. to assess the internal consistency of the newly created annotation and to provide a baseline for further experiments on the data, we evaluated the performance of a bert-based ner system initialized with the finbert model and fine-tuned on the new finnish data. these experiments indicated that the annotations of the new corpus can be automatically recognized at nearly 93% fscore, effectively matching previous results with much coarser-grained entity types. to further assess the compatibility of the newly introduced annotation with the original english ontonotes corpus v5.0 name annotation, we fine-tuned bi-and multi-lingual bert models on the combination of the finnish and english corpora, finding that bilingual models can effectively match or potentially even outperform monolingual ones, thus confirming the compatibility of the newly created annotation with existing ontonotes resources. all resources introduced in the paper are available under open licenses from https:// github.com/turkunlp/turku-one