Dialect & Sentiment Identification in Nuanced A rabic Tweets Using an Ensemble of Prompt-based, Fine-tuned, and Multitask BERT -Based Models (2022)

Abstract: Dialect Identification is important to improve the performance of various application as translation, speech recognition, etc. In this paper, we present our findings and results in the Nuanced Arabic Dialect Identification Shared Task (NADI 2022) for country-level dialect identification and sentiment identification for dialectical Arabic. The proposed model is an ensemble between fine-tuned BERT-based models and various approaches of prompt-tuning. Our model secured first place on the leaderboard for subtask 1 with an 27.06 F1-macro score, and subtask 2 secured first place with 75.15 F1-PN score. Our findings show that prompt-tuning-based models achieved better performance when compared to fine-tuning and Multi-task based methods. Moreover, using an ensemble of different loss functions might improve model performance.

## introduction arabic, spoken by over 500 million people worldwide, is the most populous member of the semitic language family. in general, arabic can be divided into three categories: (1) classical arabic, the language of early literature; (2) modern standard arabic (msa), which is commonly used in school and formal settings; and (3) dialectal arabic (da), a collection of geopolitically defined varieties. the existence of several dialects and complicated morphology are two distinguishing features of the arabic language. furthermore, the casual nature of social media chats, as well as the variations between msa and da, add to the complexity. arabic dialects are not standardized. there are no formal grammar rules or formalism to guide the speakers. this makes various tasks such as machine translation and speech recognition challenging. several works have been proposed to improve dialect identification as the recent shared-task nadi series (2020 and 2021) @xcite . several teams have used traditional methods as svm with tf-idf @xcite , others customized bert-based models. alkhamissi et al. ( 2021 ) added an adapter layer on top of marbert model. the authors of @xcite used multi-task learning to predict dialect on provenance and country level. this paper presents our work in the nuanced arabic dialect identification (nadi) shared task @xcite . the nadi shared task (2022) consists of two subtasks. the first subtask is a country-level dialect identification, while the second subtask is sentiment analysis based on different arabic dialects. given that a key challenge in this task is the unbalanced distribution and the hard nature of the problem. we follow best practices from recent work on enhancing model generalization and robustness. the rest of the papers goes as follow: section 3 discusses the proposed methods, section 4 shows experimental results, and section 5 concludes the paper. the nadi dataset provided by the organizers consists of 2 datasets for each subtask. ## system description this section presents the various approaches used while developing the final models: a voting classifier, a weighted ensemble of bert-based models, and a prompt-bert-based model. experimental setup for the fine-tuned models the learning rate was set to 4e-5 or 4e-6, cosineannealing learning rate scheduler was used, the model's weight decay was set to 1e-8 and the length of the sentence for tokenization was set to 128 or 256. during training, batch size was set to 32, and at the end of each epoch, the model was evaluated on dev-set. the best-performing model in terms of f1-macro is saved. ## results in this section, the performance of the model is reported based on the official metric during dev-phase and test-phase. moreover, error analysis is conducted to identify weaknesses of the proposed models. for subtask 1 the official metric is the macro average f1-score, while for subtask 2 the official metric is the macro-f1-pn score (macro f1-score for the negative and positive classes only). ## conclusion in this paper, we have presented our work submitted to nadi shared task. our proposed solution is an ensemble of different bert-base models. these models are developed differently, some are mtl models, fine-tuned models, or prompt-based models. the obtained results have shown that our proposed models achieve good results in both subtasks, by achieving first place in subtask 1 and first place in subtask 2. future work will focus more on building a robust model to improve recognition of some dialects. furthermore to investigate and find features that best discriminate dialects.