U ni M anc at NADI 2023 Shared Task: A Comparison of Various T5-based Models for Translating A rabic Dialectical Text to M odern S tandard A rabic (2023)

Abstract: This paper presents the methods we developed for the Nuanced Arabic Dialect Identification (NADI) 2023 shared task, specifically targeting the two subtasks focussed on sentence-level machine translation (MT) of text written in any of four Arabic dialects (Egyptian, Emirati, Jordanian and Palestinian) to Modern Standard Arabic (MSA). Our team, UniManc, employed models based on T5: multilingual T5 (mT5), multi-task fine-tuned mT5 (mT0) and AraT5. These models were trained based on two configurations: joint model training for all regional dialects (J-R) and independent model training for every regional dialect (I-R). Based on the results of the official NADI 2023 evaluation, our I-R AraT5 model obtained an overall BLEU score of 14.76, ranking first in the Closed Dialect-to-MSA MT subtask. Moreover, in the Open Dialect-to-MSA MT subtask, our J-R AraT5 model also ranked first, obtaining an overall BLEU score of 21.10.

## introduction the arabic language serves as a linguistic umbrella for approximately 420 million speakers, predominantly dispersed across 22 countries in the middle east and north africa (mena) region. a defining characteristic of the language is its diglossic nature, where modern standard arabic (msa) coexists with a myriad of dialects, commonly referred to as dialectal arabic (da) @xcite . msa is the formal version of the arabic language, employed in educational settings, official documents and written literature. it serves as a standardised communication medium across arabicspeaking countries. in contrast, da represents the colloquial forms of arabic, which are more region-specific and employed in day-to-day verbal interactions @xcite . notably, dialects can vary significantly based on geographic location and socio-economic factors, ranging from subtle differences to being nearly mutually unintelligible. this linguistic variation presents considerable challenges for machine translation (mt) models trained on msa. these models often fail to capture the nuanced differences in dialects, resulting in poor translation performance when applied to da. compounding this issue is the scarcity of parallel corpora containing msa translations of text written in da, limiting resources for model training and evaluation @xcite . in the context of these challenges, this paper aims to explore the extent to which various sequence-to-sequence models based on the textto-text transfer transformer, popularly known as t5 @xcite , can translate a source text written in an arabic dialect to a target text that is written in msa. we participated in the nuanced arabic dialect identification (nadi) 2023 shared task @xcite , specifically in subtasks 2 and 3, described below. subtask 2: dialect-to-msa mt -closed task. the objective of this subtask is sentence-level machine translation from four dialects (egyptian, emirati, jordanian and palestinian) to msa. participants were restricted to using the madar parallel corpus @xcite for training and were asked to evaluate their models on newly released development and test sets. subtask 3: dialect-to-msa mt -open task. this subtask is similar to subtask 2, except for the fact that participants were allowed to utilise additional datasets for model training. one of the goals of this subtask is to encourage the creation of new parallel corpora to facilitate future research. apart from investigating the performance of various t5-based models on the above-mentioned tasks, our work makes an additional contribution by developing a new dataset, emi-nadi, which contains msa translations of sentences written in emirati, one of the most under-resourced dialects. the emi-nadi dataset and the code for developing and evaluating our models for subtasks 2 and 3 have been made publicly available. foot_0 ## datasets this section describes the datasets that were utilised in training our models. ## methodology in this section, we introduce the t5-based models that we built upon, explain how they were finetuned and discuss hyperparameter optimisation. ## evaluation and results all models for subtasks 2 and 3 were evaluated using the bilingual evaluation understudy (bleu) metric @xcite , which estimates the similarity between a machine-translated text and a reference translation based on overlapping tokens. the results of our joint regional (j-r) and independent regional (i-r) models for subtasks 2 and 3, without using beam search (i.e., beam @xmath0, are shown in tables 3 and 4 , respectively. one can observe in in the comparison of the j-r and i-r models (without beam search) for subtask 3 shown in tables 5 and 6 present the results of our chosen models on the nadi test sets for subtasks 2 and 3, respectively. as shown in ## discussion in tables 3 and 4 , it can be observed that for both subtasks the independent regional (i-r) models performed better compared to the joint regional (j-r) models, with arat5 performing the best overall. this can be explained by the fact that arat5 was trained with a specific focus on arabic whereas the others (mt0 and mt5) were trained on many other languages apart from arabic. this implies that for the dialect-to-msa translation task, a model that was trained solely on the arabic language is superior over multilingual models. given that the i-r models performed better, multiple beam sizes were explored. our results show that increasing the beam size leads to an improvement in overall performance. however, it is worth noting that the optimal beam size could vary between the development and test sets (e.g., beam @xmath4, although the difference in performance is very marginal. error analysis was conducted to qualitatively evaluate our best-performing model for subtask 3. specifically, we analysed cases where the model obtained low bleu scores and manually assessed the quality of the translations produced by the model. an example for each dialect is shown in ## conclusion and future work in this paper, we describe the approaches we developed for nadi 2023 subtask 2 (closed dialect-to-msa mt) and subtask 3 (open dialect-to-msa mt). our results reveal that fine-tuning arat5 and incorporating beam search during translation lead to top-ranking performance. possible future directions include the development of a multilingual model focussed on arabic dialects and msa, and the creation of further parallel corpora covering low-resourced arabic dialects.