rematchka at NADI 2023 shared task: Parameter Efficient tuning for Dialect Identification and Dialect Machine Translation (2023)

Abstract: Dialect identification systems play a significant role in various fields and applications as in speech and language technologies, facilitating language education, supporting sociolinguistic research, preserving linguistic diversity, enhancing text-to-speech systems. In this paper, we provide our findings and results in NADI 2023 shared task for country-level dialect identification and machine translation (MT) from dialect to MSA. The proposed models achieved an F1-score of 86.18 at the dialect identification task, securing second place in first subtask. Whereas for the machine translation task, the submitted model achieved a BLEU score of 11.37 securing fourth and third place in second and third subtask. The proposed model utilizes parameter efficient training methods which achieves better performance when compared to conventional fine-tuning during the experimentation phase.

## introduction dialect identification plays a crucial role in understanding and analyzing linguistic variation within a language. this importance extends to the arabic language, which encompasses a wide range of dialects spoken across various regions. with the advancements in natural language processing and language models, dialect identification systems have become increasingly valuable in accurately identifying and distinguishing arabic dialects. by accurately identifying arabic dialects, language models contribute to fields such as speech recognition, language learning, and even cultural preservation. however, dialect identification in the arabic language presents unique challenges due to the extensive linguistic diversity and complexity of arabic dialects. language models, while powerful tools for natural language processing, face inherent difficulties when applied to arabic dialect identification. these challenges arise from dialectal variations, limited training data, and data scarcity for certain dialects. the nadi shared task series @xcite @xcite is a wellknown competition that offers datasets and modeling opportunities in order to improve research work developed for dialect identification. in previous versions of the competitions, various teams have participated. @xcite finetuned marbert using two different approaches. the first approach uses model embedding along with a cnn classifier. the other approach is to use model embedding with quasi-recurrent neural networks. (abdel-salam, 2022) used is an ensemble between fine-tuned bert-based models and various approaches of parameter efficient tuning including p-tuning and prompt-tuning. @xcite used general pre-training as a first step followed by fine-tuning. @xcite added an adapter layer on top of the marbert model. this paper presents our work and findings in the nadi 2023 shared task @xcite . the nadi 2023 shared task consists of three subtasks. the first subtask is a country-level dialect identification, while the second and third subtasks are a sentence-level machine translation from four dialects to msa, given that a key challenge is the hard nature of the problem. we use best practices from recent research on improving model generalization and robustness by using different parameter-efficient techniques (peft). parameterefficient fine-tuning (peft) is an alternative to full model fine-tuning, where a small number of taskspecific parameters are updated and the majority of language model parameters are frozen. in this way, only one general language model alongside the modified parameters for each task is saved or transferred. peft techniques include prefix-tuning (li and liang, 2021), lora @xcite , prompttuning @xcite and soft-prompting @xcite . the rest of the paper is structured as follows: section 3 discusses the proposed methods, section 4 shows experimental results, and section 5 concludes the paper. ## methodology this section presents the various approaches used while developing the final models. for subtask 1, the final model is a weighted ensemble of peft bert-based models and fine-tuned models. for subtasks 2&3, a single model is used. ## results and discussion in this section, the performance of the models is reported based on the official metric during devphase and test-phase. moreover, error analysis is conducted to identify weaknesses of the proposed models. for subtask 1 the official metric is the micro average f1-score, while for subtask 2&3 the official metric is the bleu score. ## conclusion we presented our attempts for the nadi shared task in this article. our solution is an ensemble of many bert-based models. these models are created in a variety of ways, including prefix-based models, fine-tuned models, and prompt-based models. the findings reveal that our suggested models perform well in the three subtasks, taking second place in subtask 1 and fourth and third places in subtask 2&3. future work will concentrate on developing a robust model to improve dialect recognition. furthermore, to research and identify traits that better distinguish dialects.