NADI 2023: The Fourth Nuanced A rabic Dialect Identification Shared Task (2023)

Abstract: We describe the findings of the fourth Nuanced Arabic Dialect Identification Shared Task (NADI 2023). The objective of NADI is to help advance state-of-the-art Arabic NLP by creating opportunities for teams of researchers to collaboratively compete under standardized conditions. It does so with a focus on Arabic dialects, offering novel datasets and defining subtasks that allow for meaningful comparisons between different approaches. NADI 2023 targeted both dialect identification (Subtask1) and dialect-to-MSA machine translation (Subtask 2 and Subtask 3). A total of 58 unique teams registered for the shared task, of whom 18 teams have participated (with 76 valid submissions during test phase). Among these, 16 teams participated in Subtask 1, 5 participated in Subtask 2, and 3 participated in Subtask 3. The winning teams achieved 87.27 F1 on Subtask 1, 14.76 Bleu in Subtask 2, and 21.10 Bleu in Subtask 3, respectively. Results show that all three subtasks remain challenging, thereby motivating future work in this area. We describe the methods employed by the participating teams and briefly offer an outlook for NADI.

## introduction arabic is a term usually used to collectively refer to a host of languages and language varieties, rather than a single language. while most of these languages and varieties are similar to one another, there can be significant differences between some of them. for example, egyptian arabic and moroccan arabic are not mutually intelligible. arabic can also be classified into three broad categories, classical, modern standard, and dialectal. of these, classical arabic (ca) represents the variety used in old forms of literature such as poetry and the qur'an, the holy book of islam. association with religion and literary expression endows ca with prestige, and it continues to be used to date side by side with other varieties. modern standard arabic (msa) @xcite is a modern-day variety that is more familiar to native speakers and is usually employed by pan-arab media organizations, government, and in education. the third category, dialectal arabic (da), is itself a superclass that is collectively assigned to a host of varieties that are sometimes defined regionally (e.g., gulf, levantine, nile basin, and north african @xcite ), but are increasingly recognized at the more nuanced levels of country or even sub-country @xcite ). nlp treatment of arabic dialects has thus far focused more on dialect identification @xcite @xcite , machine translation (mt) @xcite , morphosyntax @xcite . dialect identification is the task of automatically detecting the source variety of a given text or speech segment, and is the main focus of the current work where we introduce the findings and results of the fourth nuanced arabic dialect identification shared task (nadi 2024). the main objective of nadi is to encourage research on ara-bic dialect processing by offering datasets and facilitating diverse modeling opportunities under a common evaluation setup. the first instance of the shared task, nadi 2020 @xcite , focused on province-level dialects. nadi 2021 @xcite , the second iteration of nadi, focused on distinguishing both msa and da according to their geographical origin at the country level. the third instance, nadi 2022 @xcite , investigated both arabic dialect identification and dialectal sentiment analysis. nadi 2023, the current edition, continues this tradition of extending to tasks beyond dialect identification. namely, we propose new subtasks focused at machine translation from arabic dialects into msa. more concretely, nadi 2023 shared task is comprised of three subtasks: subtask 1 on dialect identification, while subtask 2 and subtask 3 are on dialect mt. the difference between subtask 2 and subtask 3 is that the former is a closed track where participants are allowed to use only our provided training data, whereas the latter is open track and so allows participants to train their systems on any additional datasets so long as these additional training datasets are public at the time of submission. while we invited participation in any of the three subtasks, we encouraged teams to submit systems to all subtasks. by offering three subtasks, our hope was to receive systems that exploit different methods and architectures. many of the submitted systems investigated diverse approaches, thus fulfilling our objective. a total of 58 unique teams registered for nadi 2023. of these, 18 unique teams actually made submissions to our leaderboard (@xmath0. we received 14 papers from 14 teams, of which we accepted 13 for publication. results from participating teams show that both dialect identification at the country level and dialectal mt remain challenging even to complex neural methods. these findings clearly motivate future work on all tasks. the rest of the paper is organized as follows: section 2 provides a brief overview of arabic dialect identification and sentiment analysis. we describe the two subtasks and nadi 2023 restrictions in section 3. section 4 introduces shared task datasets and evaluation setup. we present participating teams and shared task results and provide a high-level description of submitted systems in section 5. we conclude in section 6. ## task description in nadi-2023, we place our emphasis on two nlp tasks, both crucial to processing of dialectal arabic. dialect identification remains an important step in any pipeline for processing dialects, for which reason nadi-2023 subtask 1 maintains the focus on identification of arabic dialects. in particular, subtask 1 targets dialect at the country level. another important nlp task that has not particularly witnessed accelerated progress over the past few years is machine translation of arabic dialects. for this reason, we take as our second focal point mt of dialects through subtask 2 and subtask 3. we now describe each subtask in detail. ## shared task datasets and evaluation in this section, we describe the datasets we make available to participants, introduce the chosen evaluation metrics, and outline the clear instructions we provided for the submission process. ## conclusion we presented findings and results of nadi-2023, the fourth edition of the nadi shared task focused on fine-grained arabic dialect identification. this edition also introduced two subtasks centered on machine translation from four arabic dialects into msa. results acquired by participant teams show that dialect identification remains a challenging task but that various types of approaches, many of which involve exploiting language models, can be used to handle the task. similarly, translating arabic dialects is unsurprisingly very challenging due to lack of training data. in the future, we plan to continue supporting both dialect identification and machine translation through nadi.