E stonian Named Entity Recognition: New Datasets and Models (2023)

Abstract: This paper presents the annotation process of two Estonian named entity recognition (NER) datasets, involving the creation of annotation guidelines for labeling eleven different types of entities. In addition to the commonly annotated entities such as person names, organization names, and locations, the annotation scheme encompasses geopolitical entities, product names, titles/roles, events, dates, times, monetary values, and percents. The annotation was performed on two datasets, one involving reannotating an existing NER dataset primarily composed of news texts and the other incorporating new texts from news and social media domains. Transformer-based models were trained on these annotated datasets to establish baseline predictive performance. Our findings indicate that the best results were achieved by training a single model on the combined dataset, suggesting that the domain differences between the datasets are relatively small.

## introduction named entity recognition (ner) is a practical natural language processing (nlp) task that involves identifying and extracting named entities from texts, such as person names, organization names, locations, and other types of entities. ner is widely used in various downstream applications, such as document anonymisation and text categorisation. typically, modern ner systems are trained as supervised tagging models, where annotated training data is utilised for training models to identify and tag text spans that correspond to named entities. for the estonian language, prior endeavors to develop ner systems have involved the creation of an annotated dataset labelled with person, organisation, and location names @xcite . this dataset has been utilised for training crf-and transformer-based ner models @xcite @xcite . in addition to these efforts, a dataset in a different domain, 19th-century parish court records, was recently annotated with named entities @xcite . this paper describes the efforts to augment further the development of general-purpose named ner systems for the estonian language. the primary focus of this study is annotating additional estonian texts with named entities, utilising a newly developed rich annotation scheme. two annotated datasets were created as part of this effort. firstly, the existing ner dataset @xcite was reannotated using the new annotation scheme. secondly, approximately 130k tokens of new texts, predominantly sourced from news portals and social media, were annotated to create a new dataset. these annotations serve to expand the availability of annotated data for training and evaluating ner models in the estonian language. the second part of this paper delves into the experimental results obtained from training predictive bert-based models on the annotated datasets. the primary objectives of these experiments were to establish the baseline performance of various entity types of the newly developed annotation scheme and to explore the optimal utilisation of the two datasets, which stem from slightly distinct domains. the findings revealed that the baseline performance on the newly annotated dataset was slightly lower than the less richly annotated estonian ner dataset, indicating that the new annotations may possess some noise while also being richer and more intricate. moreover, the study revealed that the domains of the two datasets were similar enough such that a model trained on the combined dataset exhibited comparable or even superior performance compared to models trained on each dataset separately. in short, our paper makes two key contributions: ## dataset creation this section describes the process of creating the two labelled ner datasets for estonian. foot_0 ## experiments we had two primary goals when conducting the experiments. the first goal was to establish the baseline performance on both the main ner and new ner datasets. while several previous studies have reported results on the old annotations of the main ner dataset, the new annotations we used in our study are more comprehensive and were collected independently without reference to the old annotations. therefore, the baseline performance of the main ner dataset with the new annotations may differ. similarly, as the new ner dataset contains new material, it is crucial to evaluate its baseline performance as well. the second goal of our study was to investigate potential domain differences between the two datasets. specifically, the average document length in the new ner dataset was more than three times higher than that of the main ner dataset. also, the new ner dataset contains at least 30k tokens from the social media domain. moreover, the news part of the new ner dataset documents was not limited to formal news texts but also included less formal opinion pieces. hence, our objective was to determine the optimal approach for utilising these datasets, namely whether training separate models for each dataset would be more effective or if combining the data and training a single model would yield better results. we opted to utilise only the first-level annotations for training our models. this decision was based on the finding that much fewer entities were labelled at the second and third levels, as evidenced by the statistics presented in ## model we employed a transformer-based token classification model for our experiments, adopting the commonly-used bio format for entity labelling. in this format, the b-tag indicates the start of an entity, the i-tag denotes the continuation of an entity, and the o-tag is assigned to word tokens that do not belong to any named entity. the to-kenclassification implementation from the huggingface transformers library @xcite was utilised for this purpose. as our base model, we used the estbert model with a sequence length of 128 foot_2 (tanvir et al., 2021), which was fine-tuned on the ner datasets. in our experiments, we kept the batch size fixed at 16 and utilised the adam optimiser with betas set to 0.9 and 0.98 and an epsilon value of 1e-6. the models were trained for a maximum of 150 epochs, with early stopping implemented if the overall f1-score on the validation set did not improve for 20 consecutive epochs by more than 0.0001 f1-score points. we used the seqeval package @xcite for evaluations during training and final testing. the learning rate was optimised on the validation set using a grid of values 5e-6, 1e-5, 3e-5, 5e-5, 1e-4. each model was trained ten times with different random seeds to account for randomness, and the mean values with standard deviations are reported. ## results we first trained and evaluated models separately on both datasets to assess their overall modeling performance. then, we trained a joint model using data from both datasets and compared its performance on the evaluation sets of both datasets. this allowed us to evaluate the effectiveness of using a combined dataset compared to training on each dataset separately. reannoated main ner new ner # precision recall f1-score # precision recall f1-score per 642 .827 (.012) .871 (.009) .848 (.005) 109 .809 (.044) .816 (.023) .811 (.019) org 504 .654 (.016) .666 (.014) .660 (.013) 85 .580 (.027) .585 (.052) .581 (.024) loc 118 .643 (.036) .478 (.028) .547 (.016) 31 .600 (.065) .560 (.060) .576 (.044) gpe 714 .821 (.012) .831 (.021) .826 (.008) 129 .900 (.017) .879 (.030) .889 (.014) title 171 .676 (.023) .814 (.014) .739 (.011) 19 .750 (.062) .718 (.064) .731 (.048) prod 161 .572 (.033) .628 (.026) .598 (.024) 60 .509 (.043) .474 (.052) .488 (.029) event 13 .069 (.029) .077 (.034) .072 (.031) 16 .518 (.104) .558 (.104) .525 (.070) date 308 .682 (.020) .720 (.017) .700 (.007) 64 .816 (.027) .824 (.024) .820 (.021) time 39 .553 (.066) .555 (.045) .553 (.053) 6 .812 (.041) .788 (.108) .797 (.074) percent 62 .985 (.016) .867 (.032) .922 (.019) 11 .895 (.126) 1 (-) .940 (.074) money 76 .636 (.040) .568 (.030) .600 (.030) 12 .659 (.085) .742 (.126) .693 (.083) overall 2571 .737 (.010) .757 (.009) .747 (.004) 497 .736 (.014) .734 (.017) .735 (.006) ## discussion this study marks the first endeavor to annotate a more comprehensive set of entities beyond the commonly annotated person, organization, and location names in the estonian language. the interannotator agreement results indicate that the annotators consistently labelled certain entities, such as per, gpe, and percent, while the reliability was lower for other entities. in particular, the event entity had the lowest inter-annotator agreement. an in-depth analysis of inconsistencies in annotation, both in event and other en-tities, could be conducted as a follow-up work to identify the sources of confusion and enhance the annotation guidelines. in line with previous efforts in other languages, such as finnish, we opted to annotate nested entities by permitting up to three levels of nesting. however, upon analysing the data statistics, it was revealed that only a few entities were annotated on the third level. additionally, even though many entities were labelled on the second level, their reliability, as evidenced by inter-annotator agreements, was not deemed sufficiently high. hence, utilising these labels for training predictive models may not yield productive results. in this study, we obtained three sets of annotations for both datasets, enabling us to assess the variability in the annotations. however, it is essential to acknowledge that the choice of annotators may have introduced limitations to the annotation process. for the main ner dataset, all annotators were linguistic students, which provided expertise and interest in the annotation task, as intended. however, this uniformity in the background may have resulted in limitations in the recall of entity annotations, as noted in previous research @xcite . on the other hand, the annotators for the new ner dataset were more diverse, including computer science students. nevertheless, since the task was part of their coursework, their motivation and interest in the annotation task might not have been as high. our experimental results with the bert-based model indicate that although there may be a domain shift between the two datasets at the entity level for certain entities, training a single joint model on both datasets seems justified. it is important to note that our models based on estbert are only baselines, and as demonstrated in previous studies @xcite , utilising other base models such as estonian wik-ibert @xcite or xlm-roberta could potentially yield higher performance results. ## conclusions we provided a detailed overview of the annotation process for two estonian ner datasets, annotated with a comprehensive annotation scheme encompassing eleven distinct entity types. additionally, the datasets included nested annotations of up to three levels, although the reliability of the nested annotations was found to be less consistent compared to the first-level entities. in order to establish baseline predictive accuracy, we conducted experiments with two modeling scenarios on these newly annotated datasets. this involved training two separate models, one for each dataset and a joint model on the combined dataset. our findings revealed that the joint model outperformed the separate models, except for a few entity types, indicating that the domain differences between the datasets are relatively minimal. as such, we recommend utilising these two datasets jointly as a single, more diverse dataset for ner training purposes.