W ojood NER 2024: The Second A rabic Named Entity Recognition Shared Task (2024)

Abstract: We present WojoodNER-2024, the second Arabic Named Entity Recognition (NER) Shared Task. In WojoodNER-2024, we focus on fine-grained Arabic NER. We provided participants with a new Arabic fine-grained NER dataset called Wojoodfine, annotated with subtypes of entities. WojoodNER-2024 encompassed three subtasks: ( i ) Closed-Track Flat Fine-Grained NER, ( ii ) Closed-Track Nested Fine-Grained NER, and ( iii ) an Open-Track NER for the Israeli War on Gaza. A total of 43 unique teams registered for this shared task. Five teams participated in the Flat Fine-Grained Subtask, among which two teams tackled the Nested Fine-Grained Subtask and one team participated in the Open-Track NER Subtask. The winning teams achieved F_1 scores of 91% and 92% in the Flat Fine-Grained and Nested Fine-Grained Subtasks, respectively. The sole team in the Open-Track Subtask achieved an F_1 score of 73.7%.

## introduction ner plays a crucial role in various natural language processing (nlp) applications, such as question-answering systems @xcite , knowledge graphs @xcite , and semantic search @xcite , information extraction and retrieval @xcite , word sense disambiguation @xcite , machine translation @xcite , automatic summarization @xcite , interoperability @xcite and cybersecurity @xcite . ner involves identifying mentions of named entities in unstructured text and categorizing them into predefined classes, such as pers, org, gpe, loc, event, and date. given the relative scarcity of resources for arabic nlp, research in arabic ner has predominantly concentrated on "flat" entities and has been limited to a few "coarse-grained" entity types, namely pers, org, and loc. to address this limitation, the wojoodner shared task series was initiated @xcite . it aims to enrich arabic ner research by introducing wojood and wojood f ine @xcite , nested and fine-grained arabic ner corpora. in wojoodner-2024 we provide a new version of wojood, called wojood f ine . wojood f ine enhances the original wojood corpus by offering fine-grained entity types that are more granular than the data provided in wojoodner-2023. for instance, gpe is now divided into seven subtypes: country, state-or-province, town, neighborhood, camp, gpe_org, and sport. loc, org, and fac are also divided into subtypes as shown in figure 1 . wojood f ine contains approximately 550k tokens and annotated with 51 entity types and subtypes, covering 47k subtype entity mentions. it is worth mentioning that sinatools supports wojood and can be accessed via application programming interface (api) @xcite . teams were invited to experiment with various approaches, ranging from classical machine learning to advanced deep learning and transformerbased techniques, among others. the shared task generated a remarkably diverse array of submissions. a total of 43 teams registered to participate in the shared task. among these, five teams successfully submitted their models for evaluation on the blind test set during the final evaluation phase. the rest of the paper is organized as follows: section 2 provides a brief overview of arabic ner. we describe the three subtasks and the shared task restrictions in section 3. section 4 introduces shared task datasets and evaluation. we present the participating teams, submitted systems and shared task results in section 5. we conclude in section 6. ## literature review ner has been an area of active research for many years, witnessing notable progress recently. this section will cover the evolution from initial efforts in recognizing flat-named entities to the current focus on nested ner, with a particular emphasis on arabic ner, including discussions on corpora, methodologies, and shared tasks. corpora. the majority of arabic ner corpora are designed for flat ner annotation. aner-corp @xcite , derived from news sources, contains approximately 150k tokens and focuses on four specific entity types. caner-corpus (salah and zakaria, 2018) targets classical arabic (ca) and includes a dataset of 258k tokens annotated for 14 types of entities related to religious contexts. the ace2005 @xcite corpus is multilingual and includes arabic texts annotated with five distinct entity types. the ontonotes5 @xcite dataset features around 300k tokens annotated with 18 different entity types. however, these corpora are somewhat dated and primarily cover media and political domains, which may not accurately reflect contemporary arabic usage, particularly as language models are sensitive to changes over time and across domains. recently, @xcite introduced wojood, the largest arabic ner corpus to date, notable for supporting both flat and nested entity annotations. this corpus, essential for this shared task, includes about 550k tokens and covers 21 unique entity types across modern standard arabic (msa) and two arabic dialects (palestinian curras2 and lebanese baladi corpora @xcite ). wojood f ine @xcite , an extension of wojood adds support for entity sub-types, with a total of 51 entities organized in two-level hierarchy. it is important to note that wojood has been recently extended to include relationships @xcite . methodologies. research in arabic ner employs a variety of approaches, ranging from rulebased systems @xcite to machine learning techniques @xcite @xcite @xcite . recent studies have adopted deep learning strategies, utilizing character and word embeddings in conjunction with long-short term memory (lstm) @xcite , and bil-stm architectures paired with conditional random field (crf) layer @xcite . deep neural networks (dnn) are explored in @xcite , alongside pretrained language models (lm) @xcite . @xcite conducted a comprehensive review of various approaches to nested entity recognition, including rule-based, layered-based, region-based, hypergraph-based, and transition-based methods. fei et al. (2020) introduced a multi-task learning framework for nested ner using a dispatched attention mechanism. @xcite developed a method for nested ner that calculates all region representations from the contextual encoding sequence and assigns a category label to each. readers can also refer to the wojoodner-2023 shared task for dnn techniques used for flat and nested arabicner @xcite . ## task description wojoodner-2024 confronts the intricacies of arabic ner with three distinct subtasks: flat fine-grained ner, nested fine-grained ner, and open-track ner. these subtasks provide an evaluation environment, allowing researchers to experiment with diverse approaches for identifying and classifying named entities, along with their subtypes, under controlled (closed) and flexible (open) settings. remark: the wojood dataset used in wojoodner-2023 @xcite cannot be used in this shared task because the two datasets follow different annotation guidelines. ## datasets and evaluation metrics in this section, we will describe the dataset, evaluation metrics, and the submission procedure. ## conclusion in this paper, we present the outcomes of the second edition of wojoodner shared task. the results from the participating teams highlight the ongoing difficulties in ner, yet it is encouraging to see that various innovative approaches, particularly those leveraging the power of language models, have proven effective in tackling this complex task. as we progress, we are dedicated to advanc-ing research in this field. our vision includes continuous efforts to improve arabic ner, drawing on the valuable insights from wojoodner-2024 and exploring new solutions. additionally, we plan to expand the wojood f ine corpus to encompass more dialects.