W ojood NER 2023: The First A rabic Named Entity Recognition Shared Task (2023)

Abstract: We present WojoodNER-2023, the first Arabic Named Entity Recognition (NER) Shared Task. The primary focus of WojoodNER 2023 is on Arabic NER, offering a novel NER datasets (i.e., Wojood) and the definition of subtasks designed to facilitate meaningful comparisons between different NER approaches. WojoodNER-2023 encompassed two Subtasks: FlatNER and NestedNER. A total of 45 unique teams registered for this shared task, with 11 of them actively participating in the test phase. Specifically, 11 teams participated in FlatNER, while 8 teams tackled NestedNER. The winning team achieved F1 score of 91.96 and 93.73 in FlatNER and NestedNER respectively.

## introduction ner is a fundamental task in natural language processing (nlp), especially in information extraction and language understanding @xcite . the objective of ner is to identify and classify named entities in a given text into predefined categories, such as "person", "location", "organization", "event", and "occupation". ner is also a critical task for many nlp applications, such as question-answering systems @xcite , knowledge graphs @xcite , and semantic search @xcite , interoperability @xcite among others. named entities can either be flat or nested. for instance, in the sentence "cairo bank announces its profit in 2023", there are two flat entities: "cairo bank" is tagged as org (i.e., organization) and "2023" as date. in nested ner, entity mentions contained inside other entity mentions are also considered named entities. in this case, "cairo", is tagged as gpe (i.e., geopolitical entity). section 3 illustrates more examples. as will be discussed in section 2, research in arabic ner is currently limited, particularly in the context of nested entities. this limitation is not exclusive to modern standard arabic (msa) but extends to various arabic dialects across diverse domains and ner subtypes. the majority of existing research on arabic ner primarily emphasizes flat entities to cover a limited set of entity types, mainly "person", "organization", and "location". in this paper, we provide an overview of the wojoodner-2023 shared task foot_0 , which represents a significant step forward in advancing ner research in the arabic language. the shared task encompasses subtask1 (flatner) and subtask2 (nest-edner). for this competition, we grant participants access to the wojood corpus @xcite foot_1 , a substantial and diverse arabic ner dataset known as wojood. as shown in figure 1 , wojood is particularly notable for its scale, containing approximately 550k tokens. about 12% of the corpus was collected from social media in palestinian and lebanese dialects curras and baladi corpora @xcite . the remaining ∼ 88% is in msa, covering multiple domains, including health, finance, politics, ict, terrorism, migration, history and culture, and law and elections, making it a rich resource for various research purposes. wojood was annotated manually using 21 entity types, offering a rich arabic ner corpus. the primary objective of this shared task is to encourage participants to explore different ner methodologies. teams were invited to experiment with various approaches, ranging from classical machine learning to advanced deep learning and transformer-based techniques, among others. the shared task generated a remarkably diverse array of submissions. a total of 45 teams registered to participate in the shared task. among these, 11 teams successfully submitted their models for evaluation on the blind test set during the final phase of the competition. as a result, we received 11 papers that provide detailed insights into the results achieved by these teams for either one or both of the subtasks. the rest of the paper is organized as follows: section 2 provides a brief overview of arabic ner. we describe the two subtasks and wojoodner-2023 restrictions in section 3. section 4 introduces shared task datasets and evaluation setup. we present participating teams and shared task results and provide a high-level description of submitted systems in section 5. we conclude in section 6. ## literature review ner has been a long-standing research area, with significant advances made in recent years. as will be discussed in this section, early ner approaches focused on identifying and classifying flat named entities, and recent research has focused on nested ner. in this section, our primary focus is exclusively on arabic ner research, encompassing corpora, methodologies and shared tasks. corpora. most of the available arabic ner corpora are annotated as flat ner. anercorp @xcite , sourced from the news domain (msa text), comprises ∼ 150k tokens. its main emphasis is directed towards four distinct entity types. canercorpus (salah and zakaria, 2018) is dedicated to classical arabic (ca) and encompasses a dataset of 258k tokens. this corpus is annotated for a total of 14 entity types, all of which pertain to religious entities. ace2005 @xcite is a multilingual corpus that incorporates arabic text encompassing five distinct types of entities. ontonotes5 @xcite dataset con-sists of approximately 300k tokens, meticulously annotated with 18 distinct entity types. nevertheless, these corpora were collected a long time ago and mainly cover the media and politics domains; hence, may not be representative of the current state of arabic language use. this is especially the case since language models are known to be sensitive to temporal and domain shifts. recently, @xcite proposed wojood, the largest arabic ner corpus. it is distinctive for its support of both flat and nested entity annotations, making it a crucial resource utilized in this shared task. it comprises roughly 550k tokens encompassing a diverse range of 21 unique entity types, spanning both msa and two dialectal arabic forms (the palestinian curras2 and lebanese baladi corpora @xcite . ## task description to the best of our knowledge, wojoodner-2023 is recognized as the inaugural shared task in arabic ner. in this competition, we present two distinct subtasks-one for "flatner" and the other for "nestedner". these subtasks are of paramount importance in addressing the challenges inherent in arabic ner processing. we now describe each subtask in detail. ## shared task datasets and evaluation this section presents the dataset, evaluation metrics, and the submission process. datasets. wojoodner-2023 shared task employs the wojood corpus as its primary dataset @xcite . the wojood corpus encompasses approximately 550k tokens, spanning both msa and two arabic dialects, annotated using 21 entity evaluation metrics. the official evaluation metric for subtask1 and subtask2 is the macro-averaged f 1 score. in addition to this metric, we also report system performance in terms of precision, recall, and accuracy for submissions to both subtasks. submission roles. we allowed participant teams to submit up to four runs for each test set, for both subtasks. in each one, we strictly retain only the submission with the highest score from each participating team. although the official results were solely derived from the blind test set. to streamline the evaluation of participant systems, we have set up two separate codalab @xcite com-petitions for scoring each subtask. 4 we are keeping the codalab @xcite for each subtask active even after the official competition has concluded. this is aimed at facilitating researchers who wish to continue training models and evaluating systems with the shared task's blind test sets. as a result, we will not disclose the labels for the test sets in any of the subtasks. ## conclusion and future work in this paper, we present the outcomes of wojoodner-2023, the inaugural shared task dedicated to both flat and nested ner challenges in the arabic language. the results obtained from the participating teams underscore the persistent challenges associated with ner. however, it is promising to observe that various innovative approaches, often harnessing the capabilities of language models, have demonstrated their effectiveness in addressing this complex task. as we move forward, we remain committed to further advancing research in this domain. our vision includes ongoing efforts to enhance the field of arabic ner, incorporating the valuable insights gained from wojoodner-2023 and continuing to explore innovative solutions. we plan to extend the wojood corpus to include more dialects. we plan to include the syrian nabra dialects @xcite as well as the four dialects in the lisan @xcite corpus. specifically covering two dialects, palestinian and lebanese.