NADI 2024: The Fifth Nuanced A rabic Dialect Identification Shared Task (2024)

Abstract: We describe the findings of the fifth Nuanced Arabic Dialect Identification Shared Task (NADI 2024). NADI’s objective is to help advance SoTA Arabic NLP by providing guidance, datasets, modeling opportunities, and standardized evaluation conditions that allow researchers to collaboratively compete on prespecified tasks. NADI 2024 targeted both dialect identification cast as a multi-label task (Subtask 1), identification of the Arabic level of dialectness (Subtask 2), and dialect-to-MSA machine translation (Subtask 3). A total of 51 unique teams registered for the shared task, of whom 12 teams have participated (with 76 valid submissions during the test phase). Among these, three teams participated in Subtask 1, three in Subtask 2, and eight in Subtask 3. The winning teams achieved 50.57 F1 on Subtask 1, 0.1403 RMSE for Subtask 2, and 20.44 BLEU in Subtask 3, respectively. Results show that Arabic dialect processing tasks such as dialect identification and machine translation remain challenging. We describe the methods employed by the participating teams and briefly offer an outlook for NADI.

## introduction arabic is a collection of languages, language varieties, and dialects that can be classified into three main categories. classical arabic (ca) is the language of the qur'an, old literature, and old scientific writing. ca has played a significant role in the spread of islamic culture and continues to be crucial for scholarship in language and religious institutes. modern standard arabic (msa) is a simplified and 'modernized' descendent of ca that is employed in formal education and pan-arab media as well as governmental circles in most arab countries. dialectal arabic (da) refers to the various forms of arabic spoken in different parts of the arab world in informal settings, tv shows, and everyday life. these three main categories of arabic share lexica and grammatical structures to varying degrees, with some dialects being at one end of the continuum and ca at the other end. the nuanced arabic dialect identification (nadi) shared task series was launched in 2020 as a venue for creating resources, affording modeling opportunities, and building a research community around the processing of dialectal arabic. nadi 2024 is the fifth version of the shared task, hosted by the second arabic natural language processing conference (arabicnlp 2024). dialect identification is the task of automatically detecting the source variety of a given text or speech segment. in previous versions of nadi, dialect identification was cast as a single-label classification task. that is, the input text can be assigned only one dialectal category (usually at the country level). arabic dialects, however, can overlap significantly. this is especially the case for dialects spoken in geographically proximate areas where lexica, grammatical structures, as well as sound patterns are usually shared to notable degrees. while speech language identification models, e.g., @xcite @xcite , would typically have access to acoustic features that can help facilitate teasing apart these neighboring varieties, this is not the case for text-based systems as the input is intrinsically impoverished. aligning with this observation, recent work by @xcite analyzed the errors of a single-label dialect classification system and found that about 66% of these errors are not true errors. to accommodate these research findings and open up a space for further investigation of challenges faced by singlelabel models, we design a subtask in nadi 2024 (subtask 1) as a multi-label classification task in which the given text can belong to more than a single arabic dialect. since arabic dialects also overlap, sometimes significantly, with msa, we also introduce a new subtask for estimating the level of dialectness of text on a scale between zero and one (subtask 2). due to the challenges posed by dialects for machine translation (mt) systems, nadi 2024 continues to offer opportunities for advancing the translation of arabic dialects through subtask 3. we now review related literature. ## task description nadi 2024 maintains the focus on processing dialectal arabic. more concretely, we target both dialect identification (did) and dialectal machine translation through three subtasks. subtask 1 focuses on did, cast as a multi-label classification task, and subtask 2 aims at capturing the arabic level of dialectness in texts (aldi). as translation of arabic dialects remains particularly challenging, we devote subtask 3 to dialect mt. we now describe each subtask in detail. ## conclusion this year, we organized nadi 2024, the fifth edition of the shared task, having three subtasks: multilabel dialect identification (mdid), arabic level of dialectness (aldi) estimation, and da-to-msa machine translation. we had 51 registered teams, out of which 12 submitted their systems' predictions with eight accepted system description papers. the results indicate that there is still room for improvement across the various tasks. in the future, we intend to cover more arabic dialects in nadi and propose novel ways of modeling that involve the use of large language models.