W iki G old SK : Annotated Dataset, Baselines and Few-Shot Learning Experiments for S lovak Named Entity Recognition (2023)

Abstract: Named Entity Recognition (NER) is a fundamental NLP tasks with a wide range of practical applications. The performance of state-of-the-art NER methods depends on high quality manually anotated datasets which still do not exist for some languages. In this work we aim to remedy this situation in Slovak by introducing WikiGoldSK, the first sizable human labelled Slovak NER dataset. We benchmark it by evaluating state-of-the-art multilingual Pretrained Language Models and comparing it to the existing silver-standard Slovak NER dataset. We also conduct few-shot experiments and show that training on a sliver-standard dataset yields better results. To enable future work that can be based on Slovak NER, we release the dataset, code, as well as the trained models publicly under permissible licensing terms at https://github.com/NaiveNeuron/WikiGoldSK

## introduction named entity recognition (ner) is a lower-level natural language processing (nlp) task in which the aim is to both identify and classify named entity expressions in text into a pre-defined set of semantic types, such as location, organization or person @xcite . it is a key component of many downstream nlp tasks, ranging from information extraction, machine translation, question answering to entity linking and co-reference resolution, among others. since its introduction at muc-6 @xcite , the task has been studied extensively, usually as a form of token classification. in recent years, the advent of pre-trained language models (plms) combined with the availability of sufficiently large high quality ner-annotated datasets has led to the introduction of ner systems with very high reported performance, sometimes nearing human annotation quality @xcite . as the predominant method for adapting plms to a specific task of interest is model fine-tuning using training data, the availability of annotated ner datasets for both the training as well as the evaluation part of the process of creating a ner system is critical. since their creation is expensive, many works have focused on extracting multilingual silver-standard ner datasets from publicly available corpora such as wikipedia, exploiting the link structure to locate and classify named entities @xcite @xcite . while these methods have yielded ner-annotated datasets of significant size, with the recent follow-up work reporting quality comparable to that of datasets created via manual annotation @xcite , their application has multiple limitations: only a limited amount of wikipedia text is inter-linked, mapping wikipedia links to the pre-defined ner classes is non-trivial and their application often depends on the existence of high quality knowledge bases which may not be available for some domains and languages. in this paper we focus on slovak, a language of the indo-european family, spoken by 5 million native speakers, which is still missing a manually annotated ner dataset of substantial size. to fill this gap, we propose the following contributions: to alleviate the need for sizeable datasets at low cost across multiple languages, various methods of automatically generated ner-annotated datasets have been introduced. in @xcite the authors introduce the wikiner datasets, which makes use of wikipedia articles and spans 9 languages but does not include slovak. utilizing a similar approach, @xcite first classified english wikipedia pages to specific entity types and then used the cross-lingual links to transfer the annotations to other languages. as not all entries are linked, the authors also utilize self-training and translation methods to match as many entries as possible. this pipeline generates a dataset that covers 282 languages and includes slovak as well. with roughly 50 thousand entities annotated with categories person, location and organization, it is the largest publicly available slovak ner dataset to date. another approach of resolving the need for a sizable training dataset is to utilize few-shot learning, in which only a couple of expertly annotated training examples are provided. recently, the methods based on the combination of cloze-style rephrasing with language models have been shown to perform comparably to gpt-3 @xcite while having significantly fewer parameters @xcite . we consider a variant of pattern-exploiting training (schick and schütze, 2020a) called peter @xcite and to the best of our knowledge for the first time evaluate its performance for a general-purpose ner system in a specific language. slovak ner the prior art in slovak ner is limited. in @xcite the authors identified potential named entities as words with capital letters and recognized new entities by finding the entity scope through wikipedia parsing. for the purposes of this work they also created a dataset annotated by 60 human experts totalling 1620 entities. the authors of @xcite and @xcite worked with datasets based on more than 5000 articles extracted from slovak wikipedia, containing more than 15 000 entities and used multiple well-established nlp toolkits and libraries (such as spacy) to train ner models on this dataset. utilizing a different datasource, @xcite have focused on the twitter account of one of the biggest slovak journal and created a dataset with 10 000 of its ner-annotated tweets and almost 16 000 entities, and used it to train a ner model which utilized both fasttext @xcite vectors and the bilstm neural network architecture. unfortunately, none of the datasets and models mentioned in the aforementioned works are publicly available. despite having 5 million native speakers and being one of the official languages of the european union, there are relatively few readily available nlp tools tailored specifically for slovak, which might be to some extent caused by its linguistic and historical closeness to the much better resourced czech. this creates a peculiar dichotomy: slovak has too many native speakers to be considered "low-resource" but at the same time lacks readily available labelled datasets that are a prerequisite for many standard nlp tools. the "language richness" taxonomies such as @xcite consider slovak among the "the rising stars" of languages, but it is, to the best of our knowledge, one of the few in this category that lacks a sizable, manually labelled ner dataset foot_1 . the introduction of slo-vakbert in @xcite does suggest, however, that there is interest in creating slovakspecific nlp tools and resources. our work aims to help push this trend further. ## dataset when creating the wikigoldsk dataset, our principal aim was to create a high quality, publicly available human annotated corpora that could be used to both evaluate and build slovak ner systems and that would be comparable to well-established benchmark datasets in other languages. to ensure the resulting dataset can be used in the future for research as well as commercial use, we sampled 412 articles from the skwiki-20210701 dump of slovak wikipedia, licensed under the terms of the creative commons attribution-share-alike license 3.0. in order for an article to be included, its last change date needed to be in 2021 and its length had to fit between 500 and 5 000 characters foot_2 . the raw text of the articles was tokenized by the generic english spacy tokenizer, with a manual pass over the dataset in which the slovak-specific tokenization mistakes were remedied. we use the same set of tags as the conll-2003 ner shared task @xcite , that is location (loc), person (per), organization (org) and miscellaneous (misc), and our annotation guidelines are inspired by the ones introduced by the bsnlp 2017 shared task @xcite . the annotation was done using prodigy foot_3 in which the whole dataset was preloaded with the labels predicted by the slovak-bert model finetuned on the training part of the slovak portion of the wikiann dataset. the dataset was annotated by three slovak native speakers who are also authors of this paper. two annotators provided annotations for the full dataset whereas one annotator corrected half of the dataset. the cohen's kappa coefficient between the first two annotators is 0.90 when compared on the token level and 0.81 for the tokens where both annotators agreed that they were not a part of a named entity. as per the benchmark established in @xcite , the coefficient values in both cases suggest "almost perfect" strength of agreement and a high quality of the annotation. to arrive at the final dataset, the ambiguities were resolved in a discussion between the annotators. the summary statistics of the resulting dataset, along with the existing slovak ner datasets, can be found in to make the dataset compatible with existing benchmarks, we also introduce a standard train/dev/test split in the 7:1:2 ratio, described in detail in ## experiments we conduct two types of experiments with the newly introduced dataset. first, we establish a set of baselines based on existing state-of-the-art plms that were pre-trained on slovak data. next, we emulate a low-resource setup by only using a small sample of the training set and use it to evaluate a few-shot learning approach as well. ## results the results of the evaluation of baselines can be seen in when it comes to the few-shot learning experiments, the results can be seen in ## conclusion in this work, we introduce wikigoldsk, the first sizable, manually annotated ner dataset in slovak. we have established first baseline benchmarks on the dataset using state-of-the-art models, including multilingual and slovak-specific models. the experiments with few-shot learning suggest that its performance does not reach that of supervised learning. the wikigoldsk dataset is publicly released under permissible licensing terms, enabling training and evaluation of future models as well as tracking the progress in slovak ner.