A rabic Fine-Grained Entity Recognition (2023)

Abstract: Traditional NER systems are typically trained to recognize coarse-grained categories of entities, and less attention is given to classifying entities into a hierarchy of fine-grained lower-level sub-types. This article aims to advance Arabic NER with fine-grained entities. We chose to extend Wojood (an open-source Nested Arabic Named Entity Corpus) with sub-types. In particular, four main entity types in Wojood (geopolitical entity (GPE), location (LOC), organization (ORG), and facility (FAC) are extended with 31 sub-types of entities. To do this, we first revised Wojood’s annotations of GPE, LOC, ORG, and FAC to be compatible with the LDC’s ACE guidelines, which yielded 5, 614 changes. Second, all mentions of GPE, LOC, ORG, and FAC (~ 44K) in Wojood are manually annotated with the LDC’s ACE subtypes. This extended version of Wojood is called WojoodFine. To evaluate our annotations, we measured the inter-annotator agreement (IAA) using both Cohen’s Kappa and F1 score, resulting in 0.9861 and 0.9889, respectively. To compute the baselines of WojoodFine, we fine-tune three pre-trained Arabic BERT encoders in three settings: flat NER, nested NER and nested NER with sub-types and achieved F1 score of 0.920, 0.866, and 0.885, respectively. Our corpus and models are open source and available at https://sina.birzeit.edu/wojood/.

## introduction named entity recognition (ner) is the task of identifying and classifying named entities in unstructured text into predefined categories such as people, organizations, locations, disease names, drug mentions, among others @xcite . ner is widely used in various applications such as information extraction and retrieval @xcite , question answering @xcite , word sense disambiguation @xcite , machine translation @xcite , automatic summarization @xcite , interoperability @xcite and cybersecurity @xcite . traditional ner systems are typically trained to recognize coarse and high-level categories of enti-ties, such as person (pers), location (loc), geopolitical entity (gpe), or organization (org). however, less attention is given to classifying entities into a hierarchy of fine-grained lower-level subtypes @xcite . for example, locations (loc) like asia and red sea could be further classified into continent and water-body, respectively. similarly, organizations like amazon, cairo university, and sphinx cure can be classified into commercial, educational, and health entities, respectively. belgium, beirut, and brooklyn can be classified into country, town, and neighborhood instead of classifying them all as gpe. the importance of classifying named entities into subtypes is increasing in many application areas, especially in question answering, relation extraction, and ontology learning @xcite . as will be discussed in the following sub-section, the number of ner datasets that support subtypes is limited, particularly for the arabic language. the only available arabic ner corpus with subtypes is the ldc's ace2005 @xcite . however, this corpus is expensive. in addition, ace2005 was collected two decades ago and hence may not be representative of the current state of arabic language use. this is especially the case since language models are known to be sensitive to temporal and domain shifts (see section 5). to avoid starting from scratch, we chose to extend upon a previously published and open-source arabic ner corpus known as 'wojood' @xcite . wojood consists of 550k tokens manually annotated with 21 entity types. in particular, we manually classify four main entity types in wojood (gpe, loc, org, and fac) with 31 new finegrained subtypes. this extension is not straightforward as we have to change (5, 614 changes) the original annotation of these four types of entities to align with ldc guidelines before extending them with subtypes. the total number of tokens that are annotated with the 31 subtypes is 47.6k. our extended version of wojood is hereafter called wojood f ine . we measure inter-annotator agreement (iaa) using both cohen's kappa and f 1 , resulting in 0.9861 and 0.9889, respectively. to compute the baselines for wojood f ine , we fine-tune three pre-trained arabic bert encoders across three settings: (i) flat, (ii) nested without subtypes, and (iii) nested with subtypes, using multitask learning. our models achieve 0.920, 0.866, and 0.885 in f 1 , respectively. the remaining of the paper is organized as follows: section 2 overviews related work, and section 3 presents the wojood f ine corpus, the annotation process, and the inter-annotator-agreement measures. in section 4, we present the experiments and the fine-tuned ner models. in section 5 we present error analysis and out-of-domain performance and we conclude in section 6. ## related work most of the ner research is focused on coarsegrained named entities and typically targets a limited number of categories. for example, chinchor and robinson (1997) proposed three classes: person, location and organization. the miscellaneous class was added to @xcite . additional four classes (geopolitical entities, weapons, vehicles, and facilities) were also introduced in the ace project @xcite . the ontonotes corpus is more expressive as it covers 18 types of entities @xcite . coarse-grained ner is a good starting point for named entity recognition, but it is not sufficient for tasks that require a more detailed understanding of named entities @xcite . substantial research has been undertaken to identify historical entities. for instance, the hipe shared task @xcite focused on extracting named entities from historical newspapers written in french, german, and english. one of its subtasks was the recognition and classification of mentions according to finer-grained entity types. the corpus used in the shared task consists of tokens annotated with five main entity types and 12 subtypes, following the impresso guidelines @xcite . a similar corpus, called newseye, was collected from historical newspapers in four languages: french, german, finnish, and swedish @xcite . the corpus is annotated with four main types: per, loc, org, and prod. the loc entities were further classified into five subtypes, and the org entities into two subtypes. @xcite proposed a one million fine-grained ner corpus for dutch, which was annotated using six main entity types and 27 subtypes (10 subtypes for pers, three for org, nine for loc, three for prod, and two for events). @xcite noted that ner models cannot effectively process fine-grained labels with more than 100 types. thus, instead of having many finegrained entities at the top level, they propose a tagging strategy in which they use 15 main entity types and 131 subtypes. additionally, @xcite proposed a fine-grained set of 112 tags and formulated the tagging problem as multiclass multi-label classification. a recent shared task was organized by @xcite at semeval-2023 task 2, called multi-coner 2 (fine-grained multilingual named entity recognition). a multilingual corpus (mul-ticoner v2) was extracted from localized versions of wikipedia covering 12 languages -arabic is not included. the corpus was annotated with a ner taxonomy consisting of 6 coarse-grained types and 33 fine-grained subtypes (seven subtypes for person, seven for group, five for prod, five for creative work, and five for medical). most participating systems outperformed the baselines by about 35% f 1 . there are a few arabic ner corpora @xcite , but all of them are coarse-grained. the anercorp corpus covers four entity types @xcite , canercorpus covers 14 religion-specific types (salah and zakaria, 2018), and ontonotes covers 18 entities @xcite . the multilingual ace2005 corpus @xcite , which includes arabic, covers five coarse-grained entities and 35 fine-grained subtypes (3 subtypes for pers, 11 for gpe, seven for loc, nine for org, and five for fac). nevertheless, the ace2005 corpus is costly and covers only one domain (media articles) that was collected 20 years ago. the most recent arabic ner corpus is wojood @xcite , which covers 21 nested entity types covering multiple domains. however, wojood is a coarse-grained corpus and does not support entity subtypes. to build on previous research on arabic ner, we chose to extend the wojood corpus with finergrained subtypes. to ensure that our wojood exten-sion is compatible with other corpora, we chose to follow the ace annotation guidelines. ## analysis for all tasks, all models almost always converge in the first 10 epochs. for all models, there is a positive correlation between performance and the number of training samples. for example, for classes represented well in the training set (e.g., country, town and gov), models perform at 0.90 f 1 or above. the inverse is also true, with poor performance on classes such as sport, boundary and celes-tial. there are also some nuances. for example, we can see that the best model is struggling with the com subtype class even though the model has scored good results with classes with fewer samples such as cluster. the main reason for this is that types such as cluster are a closed set of classes (e.g., "european union", "african union") where the model can easily memorize them, while the com refers to an infinite group of commercial entities, that can not be limited. figure 5 is a plot of the number of samples in training data (x-axis) vs. performance (y-axis) that clearly shows the general pattern of good performance positively correlating with the number of training samples. ## conclusion and future work we presented wojood f ine , an extension to the wojood ner corpus with subtypes for the gpe, loc, org, and fac. wojood f ine corpus is the first finegrain corpus for msa and dialectal arabic with nested and subtyped ner. the gpe, org, fac and loc tags form more than 44k tokens of the corpus, which was manually annotated using subtypes entities. our inter-annotator agreement iaa evaluation of wojood f ine annotations achieved high levels of agreement among the annotators. the achieved evaluations are 0.9861 kappa and 0.9889 f 1 . we also fine-tune three pre-trained models ar-bertv2, marbertv2 and arabertv2 and tested their performance on different settings of wojood f ine . we find that arabertv2 achieved the best performance on nested and nested with subtypes tasks. in the future, we plan to test pretrained models on nested subtypes with hierarchical architecture. we also plan to link named entities with concepts in the arabic ontology @xcite to enable a richer semantic understanding of text. additionally, we will extend the wojood f ine corpus to include more dialects, especially the syrian nabra dialects @xcite as well as the four dialects in the lisan @xcite corpus.