A rabic NLU 2024: The First A rabic Natural Language Understanding Shared Task (2024)

Abstract: This paper presents an overview of the Arabic Natural Language Understanding (ArabicNLU 2024) shared task, focusing on two subtasks: Word Sense Disambiguation (WSD) and Location Mention Disambiguation (LMD). The task aimed to evaluate the ability of automated systems to resolve word ambiguity and identify locations mentioned in Arabic text. We provided participants with novel datasets, including a sense-annotated corpus for WSD, called SALMA with approximately 34k annotated tokens, and the dataset with 3,893 annotations and 763 unique location mentions. These are challenging tasks. Out of the 38 registered teams, only three teams participated in the final evaluation phase, with the highest accuracy being 77.8% for WSD and 95.0% for LMD. The shared task not only facilitated the evaluation and comparison of different techniques, but also provided valuable insights and resources for the continued advancement of Arabic NLU technologies.

## introduction natural language understanding (nlu) is a core aspect of natural language processing (nlp), facilitating semantics-based human-machine interactions @xcite . one of the key challenges in arabic is ambiguity, because arabic exhibits morphological richness, encompassing a complex interplay of roots, stems, and affixes, and rendering words susceptible to multiple interpretations based on their morphology @xcite . ambiguity in language can lead to misunderstandings, incorrect interpretations, and errors in nlp applications @xcite . a core nlu task is word sense disambiguation (wsd), and its special case location mention disambiguation (lmd). wsd aims to determine the correct sense of ambiguous words in context @xcite , while lmd focuses on disambiguating location mentions that are referred to with multiple toponyms, i.e., particular place or location @xcite . the arabic linguistic complexity, coupled with inherent polysemy, underscores the necessity for these lexical disambiguation tasks. they help decipher the intended sense of a word and the targeted entity of a mention within diverse contexts. for instance, in the sentence " / i drank from the springs of tripoli and ate the sweetness of cheese in them" one needs to disambiguate the meaning of two ambiguous words: " / springs," which has 11 related senses, and " / tripoli," which could refer to a location in either lebanon or libya. wsd is particularly important for tasks like machine translation @xcite , where it plays a pivotal role in improving accuracy by selecting contextually appropriate translations. information retrieval systems (abderrahim and abderrahim, 2022) also heavily rely on accurate wsd and lmd to ensure search queries yield relevant results, considering the appropriate senses/entities of words within queries. furthermore, applications such as question answering @xcite , sentiment analysis @xcite ), text summarization @xcite , news analysis @xcite , and semantic search (modi and jagtap, 2018) benefit from wsd and lmd. these tasks contribute to a nuanced understanding of arabic text, enhancing the accuracy and relevance of results across diverse nlp applications. recently, semantic disambiguation tasks have become integral to addressing hallucinations in large language models (llms) @xcite . while these tasks are extensively researched in well-resourced languages, there is a noticeable lack of focus on arabic, despite their pivotal role in nlp @xcite . this scarcity in arabic nlp research can largely be attributed to the lack of datasets supporting these essential tasks @xcite . without sufficient data, researchers face significant obstacles in developing and evaluating models tailored to the complexities of the arabic language. to address these challenges and draw attention to the issues faced in arabic nlu, we organized the first arabic natural language understanding (ara-bicnlu 2024) shared task, focusing on two subtasks: wsd and lmd. we have provided the participating teams with carefully annotated datasets, which are publicly accessible. specifically, we have provided two manually annotated high-quality datasets for arabic wsd and lmd. the teams were invited to experiment with diverse deep learning and machine learning methodologies, including, but not limited to, generative approaches, multitask learning, transfer learning, sequence classification, sequence-to-sequence modeling, and graph models. despite having 38 registered teams, we received only three submissions, highlighting the challenging and non-trivial nature of the shared task topics. the remainder of the paper is organized as follows: section 2 offers a brief literature of arabic wsd and lmd. section 3 details the intricacies of the shared task. section 4 discusses the wsd task, including definition, dataset, baselines, participants' systems, and results. section 5 discusses the lmd task, including definition, dataset, baselines, participants' systems, and results. finally, section 6 concludes the paper. ## related work nlu enables language models to accurately represent the knowledge embedded in words, which is crucial for core semantic tasks like wsd. wsd remains challenging despite the advancements in deep learning models like bidirectional encoder representations from transformers (bert) and generative pre-trained transformers (gpt). the challenge extends beyond wsd, involving various disambiguation tasks such as lmd, which aims at disambiguating multiple toponyms for a given location. more complex disambiguation tasks involve intents, anaphora, metaphors, and poetry. therefore, computational semantics must explore these areas in greater depth, beyond merely considering the zipfian distribution of words. ## conclusion and future work in this paper, we present the results of the ara-bicnlu 2024 shared task, focusing on the challenges of word sense disambiguation and location mention disambiguation in arabic natural language understanding. the findings from participated teams highlight the ongoing challenges and research gaps associated with these subtasks. we observe that generative models underperform traditional classification architectures trained on labeled data. this is specially the case for low resourced languages which are not well supported in llms. llms are mostly english-centric due to the imbalanced training corpora. this also extends to other systems and tools such as geopy, requiring machine translation to english to achieve the desired performance. the challenge of multilingual support becomes even more apparent when working with arabic dialectical data. to really democratize arabic nlp, it is essential to compile large datasets in various arabic dialects, as demonstrated by the work of @xcite @xcite . additionally, new techniques must be developed to address the scarcity of dialectical data, and llms specifically tailored for the arabic language need to be trained. our vision for this shared task is to create a collaborative environment that accelerates research and development in arabic nlu. by facilitating the evaluation and comparison of various models and techniques, we aim to uncover new insights, foster innovation, and build a strong foundation of resources. this effort seeks to overcome current obstacles and significantly advance the capabilities of arabic nlu technologies.