T hai Nested Named Entity Recognition Corpus (2022)

Abstract: This paper presents the first Thai Nested Named Entity Recognition (N-NER) dataset. Thai N-NER consists of 264,798 mentions, 104 classes, and a maximum depth of 8 layers obtained from 4,894 documents in the domains of news articles and restaurant reviews. Our work, to the best of our knowledge, presents the largest non-English N-NER dataset and the first non-English one with fine-grained classes. To understand the new challenges our proposed dataset brings to the field, we conduct an experimental study on (i) cutting edge N-NER models with the state-of-the-art accuracy in English and (ii) baseline methods based on well-known language model architectures. From the experimental results, we obtained two key findings. First, all models produced poor F1 scores in the tail region of the class distribution. There is little or no performance improvement provided by these models with respect to the baseline methods with our Thai dataset. These findings suggest that further investigation is required to make a multilingual N-NER solution that works well across different languages.

## introduction named entity recognition (ner) is a task of extracting named entities from given text. it identifies the span of each entity and categorizes the identified span into an entity category. ner is essential in many downstream tasks, e.g., entity linking, question answering, and knowledge graph. in addition, @xcite show that the contextualized representations that include entity information can improve many downstream tasks. the conventional ner paradigm can only label one entity type for each entity span. for example, the entity "chiang mai university" will be considered as a single span ignoring the nested structure of the term "chiang mai," which is the name of the city that the university is situated in. as a result, we may overlook critical information that may have an impact on the language understanding in a downstream task. to mitigate this drawback, one may introduce a nested structure into the ner problem. let us again consider the "chiang mai university" example. in addition to annotating the entire span as an organization, n-ner also identifies the sub-entity of "chiang mai" as a location. this feature can be useful in a downstream task that requires linking an entity to useful references, e.g., a university to its affiliated city. considerable research attention has been dedicated to formulating a technique to solve the n-ner problem @xcite . one can use an n-ner model to recursively decompose a complex entity into a tree structure of sub-entities and have them annotated accordingly. while n-ner has many potential benefits to downstream tasks that require deep language understanding, there is still a lack of datasets for low-resource languages to help develop reliable n-ner models. in order to train n-ner models, we need a dataset with hierarchical information of each named entity. n-ner datasets are available in several languages. especially, english, a high resource language, has a few n-ner datasets available for multiple domains @xcite including news, social media, and molecular biology. the diversity of n-ner corpora is only available in english. n-ner datasets are not as widely available for other languages, let alone the diversity of corpora. in german, another high-resource language, there is only one n-ner dataset available @xcite . for low-resource languages, such as vietnamese, the two available datasets @xcite are still small compared to a large n-ner dataset in english @xcite . in this paper, we address the scarcity of non-english n-ner resources by introducing a thai n-ner dataset. despite over 58 million internet users 1 , the thai language suffers from the lack of annotated resources to build nlp systems. we propose a thai n-ner dataset comprising 264,798 entity mentions obtained from 4,894 documents. in addition to the nested entity structure, we also have more than one hundred classes providing great fidelity in entity categorization as shown in figure 1 . the number of entity mentions and variety of entity classes are comparable to a large n-ner dataset in english @xcite . our dataset contains text samples, in both formal and colloquial settings, from news articles and restaurant reviews. additionally, our corpus allows for the multilingual evaluation of "language-agnostic" deep learning models, which is the current nlp research trend. to facilitate future n-ner research, we make the dataset, the annotation guideline, and the model weights publicly available. to summarize, our contributions are as follows: ## related work in this section, we discuss various attempts on n-ner corpora. as shown in in terms of the number of classes, it is also worthnoting that three out of six corpora has less than ten classes and only nne @xcite has more than 100 classes. the details of these corpora are given as follows. ace-2004 @xcite and ace-2005 @xcite are early examples of n-ner datasets. ace-2005 @xcite dataset comprises 30,966 mentions from 12,548 sentences with 7 coarse-grained entity types. in addition to n-ner annotations, ace-2005 also contains labels for other tasks such as recognition of relation and event extraction. genia @xcite introduces an n-ner data for bioinformatics. this project provides a high-quality corpus annotated for biological entity names. the dataset is composed of 2,000 abstracts, 92,681 mentions from 9,533 sentences with 32 entity types. nne @xcite is a recent large finegrained n-ner dataset composed of 114 classes. unlike previous n-ner corpora, the nne dataset annotates entities with more details. for example, "6 september 2019", a date named entity mention, in the nne dataset, each element in this mention is annotated with finer detail, "6" is annotated with day tag, "september" with month tag, and "2019" with year tag. nosta-d @xcite is the first and only german n-ner dataset. nosta-d is composed of 41,005 mentions, 12 entity types, and 31,300 sentences from the german wikipedia and online news. the previous german ner dataset, @xcite , shows that the performance of german is lower than english's foot_0 . however, the german conll-2003 dataset is known to be inconsistent because it was annotated by non-native speakers. hence, nosta-d aims to provide a high quality free public ner dataset by using native speakers as annotators. in contrast to previous n-ner corpora, nosta-d has a less restrictive copyright license. vlsp-2018 @xcite is a standard benchmark for vietnamese n-ner. it was designed for the vietnamese ner shared task to foster the development of high-quality open-source software. this dataset contains 35,817 mentions from 1,282 documents with 3 entity types. dan+ @xcite presents the first n-ner dataset for danish. this work investigates the possibility of transfer-learning between languages for the n-ner task. moreover, dan+ is a multi-domain dataset; they also study the challenges of domain-shift in their dataset. the dataset contains 6,425 mentions, 130,095 tokens, 4 classes from 6,867 sentences, obtained from multiple domains such as news and social media (reddit, twitter, and arto). nosta-d, vlsp-2018, and dan+ have a modest corpus size and a small number of entity types comparing to the nne dataset. this shows that there is still a resource gap for non-english corpora. on the other hand, for thai, there are only coarsegrained flatten-ner datasets which are publicly available @xcite . ## thai n-ner corpus in this section, we introduce thai n-ner-the first thai-nested named entity recognition dataset. our dataset is comparable to the nne corpus @xcite , which is the most elaborate english n-ner dataset in terms of the number of mentions, depth, and the number of classes. in particular, thai n-ner comprises 264,798 mentions organized into 104 classes and has a maximum depth of 8 layers. ## data statistics this section discusses the dataset statistics and analyzes the distribution of classes in the dataset. we compare our dataset with other n-ner datasets in other langauges. one of the challenges in this dataset is class imbalance. due to the number of classes, the scarcity of data for rare classes contribute to the severity of class imbalance. we visualize the distribution of classes in training set in figure 3 . the graph shows the distribution of mentions per class in training set sorted by frequency. to analyse the severity of class imbalance, we divided the classes into three groups follow pareto principle: head, body and tail with samples per classes are 80%, 15% and 5% respectively. more precisely, in body and tail parts, they contain only 20% of samples in training set, but consist of 84 classes from 104 classes. in conclusion, we introduce a dataset for thai n-ner that is comparable to the standard n-ner dataset in english. additionally, we point out a challenging long-tail distribution problem in n-ner that allows researchers to explore. ## experimental settings and results the objectives of the experimental studies are as follows: the first objective is to help researchers understand how existing techniques perform on our dataset and to help them choose the most appropriate baseline for future research. the second objective is concerned with the distribution of classes which follows the 80-20 pareto principle. as shown in figure 3 , the top 20% most frequent classes account for 80% of the mentions. we also study how these techniques perform differently at the head, body, and tail parts of the distribution. the third objective is to compare how existing models perform on our thai dataset with respect to results from existing studies conducted on english datasets. ## error analysis to understand the limitation of current n-ner solutions, we investigate reoccurring mistake patterns from the wangchanberta-sp models used in the experimental studies. we categorize the common prediction mistakes into four groups as follows: (1) incorrect span prediction: out of 5,334 prediction errors, 3,103 errors are from span length mismatch as shown in figure 5 . (2) ambiguous entity mentions: mentions with higher class distribution entropy have more error rates. (3) ambiguity between fine-grained classes: there are 1,160 fewer errors when evaluated with coarse-grained ground truths. (4) scarcity of training samples: the model only made 1,380 prediction attempts for mentions in tail classes. while 1,081 of the predictions are correct, there are 3,511 ground truths. the previous section also reveals this issue via the poor recall scores in the tail part of the long-tail dis-tribution. we provide the description of each error pattern along with examples in appendix a.8. ## summary we present the first thai n-ner corpus with 104 classes. it has 1,272,381 words, and 264,798 mentions. the size of our corpus is comparable to one of the large n-ner corpora in english. unlike other thai ner corpora, in addition to nested structure information, our dataset is annotated with fine-grained entity types to provide more detail of the named entities. this corpus addresses the data scarcity issue for thai nlp. in addition, it allows nlp researchers to benchmark their methods in a multilingual setting. moreover, this dataset allows researchers to explore the effect of long-tail distribution. we hope that our dataset will encourage researchers to include thai in their benchmark and reduce the disparity between thai and high resource languages.