Domain Adaptation for A rabic Cross-Domain and Cross-Dialect Sentiment Analysis from Contextualized Word Embedding (2021)

Abstract: Finetuning deep pre-trained language models has shown state-of-the-art performances on a wide range of Natural Language Processing (NLP) applications. Nevertheless, their generalization performance drops under domain shift. In the case of Arabic language, diglossia makes building and annotating corpora for each dialect and/or domain a more challenging task. Unsupervised Domain Adaptation tackles this issue by transferring the learned knowledge from labeled source domain data to unlabeled target domain data. In this paper, we propose a new unsupervised domain adaptation method for Arabic cross-domain and cross-dialect sentiment analysis from Contextualized Word Embedding. Several experiments are performed adopting the coarse-grained and the fine-grained taxonomies of Arabic dialects. The obtained results show that our method yields very promising results and outperforms several domain adaptation methods for most of the evaluated datasets. On average, our method increases the performance by an improvement rate of 20.8% over the zero-shot transfer learning from BERT.

## introduction the arabic language is characterized by two main language varieties: modern standard arabic (msa) and arabic dialect. msa has a standard written form and acquires an official status across the arab countries, while dialectal arabic refers to the informal spoken dialects in the arab world @xcite . these dialects are used in daily life but have no standard written form @xcite @xcite . geographically and according to @xcite , arabic dialects can be classified into five coarse-grained regional dialects: egyptian, levantine, gulf, iraqi, and maghrebi. recent studies have categorized dialectal arabic into more fine-grained levels, including countries and cities @xcite . these dialects differ from one another and from msa, to a varying degree, at different linguistic levels @xcite . with the unprecedented reach of social media platforms, sentiment analysis (sa) has become a fundamental task for many applications. most research work in this area has been devoted to english and other european languages, while some research studies have addressed the question of transfer learning from msa to dialectal arabic. however, @xcite and @xcite have shown that zero-shot transfer learning, from models trained on msa data, does not perform well for sa on dialectal arabic data. so, existing works have focused on building resources and annotating corpora for a few dialects where most of them were collected from social media @xcite @xcite @xcite . nevertheless, dealing with arabic dialects as standalone languages is challenging since building manually such resources is costly and timeconsuming. it is well known that the generalization performance of machine learning (ml) models drops in the case of domain shift (out of distribution data). hence, there is an imperative need to leverage existing labeled data from other related domains, in order to address this challenge. the aim is to accurately transfer the learned knowledge from a source domain labeled data to a new target domain data. on the one hand, adaptive pretraining of contextualized word embedding models has shown an effective transfer learning performance under domain shift @xcite . it consists of finetuning the pre-trained language models on large unlabeled corpus from the target domain using the mlm objective. on the other hand, self-training and domain-adversarial learning have been applied successfully to many nlp applications @xcite @xcite . an effective method that combines domain-adversarial training and self-training is the adversarial-learned loss for domain adaptation (alda) @xcite . the domainadversarial training aligns both domains' output distributions, while self-training captures the discriminative features of the target domain data. in this paper, we introduce a new unsupervised domain adaptation method for arabic crossdomain and cross-dialect sentiment analysis based on arabert language model (antoun et al., 2020) and the adversarial-learned loss for domain adaptation (alda) @xcite . due to limited amount of unlabeled data for most target domains-dialects, we do not rely on the adaptive pre-training of arabert model. our method leverages the potentials of: i) contextualized word embeddings to learn high-level text representation, ii) adversarial domain training to match the output distributions of domains and dialects, and iii) selftraining to capture the discriminative features of the target domain data. to summarize, our main contributions are as follows: to the best of our knowledge, this is the first study that investigates domain adaptation for crossdomain, cross-dialect and cross-domain & crossdialect sentiment analysis, adopting the coarsegrained and the fine-grained taxonomies of arabic dialects. the proposed method outperforms several state-of-the-art methods on most test datasets. the rest of this paper is organized as follows. section 2 presents related work. in section 3, we introduce our method. section 4 illustrates the conducted experiments, and discusses the obtained results. finally, in section 6, we conclude the paper and outline a few directions for future work. ## method in this section, we present our model architecture. the noise-correcting discriminator, the classifier and the generator losses, employed in our model, are those of alda model @xcite . ## experiments in this section, we present the experiments carried out to investigate the performance of our proposed method for arabic cross-domain and cross-dialect sentiment analysis. we describe the used datasets and present the compared methods as well as the obtained results. we provide the experiments settings and implementation details of our method in section a. the source code for reproducing the experimentations can be found in our github repository foot_0 . ## error analysis to understand why our proposed method outperforms the previous methods, we perform an error analysis. in this error analysis we focus on two aspects: the misclassified instances by our system and the instances correctly predicted by our method which the other approaches fail. for the first aspect, the majority of misclassified samples correspond to very short sentences in the target dialect. most of them are either idiomatic, offensive or sarcastic expressions that are specific to the target dialect and contains words that are distant from msa : /waerp/, /gar allh yefw wsafy/, /mla than/ and /crf xarf/ 2 . it is worth mentionning that the other evaluated methods also misclassify these samples. for the second aspect, we have checked the cases where our method correctly predicts the instances labels while the other methods fail. overall, we notice that the zero-shot predictions were biased to the distribution of the source data, as example the arsas dataset contains 63% of negative instances. mmd, coral and dann overcome this issue by aligning the distribution of source and target features, which improves the results on the target domain. meanwhile, they tend to misclassify reviews that convey multiple sentiment polarities, as the case for hotel reviews or books reviews, where users tend to express their negative and positive sentiments in the same review. ## conclusion in this work, we have introduced an unsupervised domain adaptation method for arabic cross-2 transliteration is performed using safe buckwlater scheme domain and cross-dialect sentiment analysis based on the pretrained arabert language model and the adversarial-learned loss for domain adaptation (alda). we have performed several experiments to investigate the performance of our method as well as several state-of-the-art methods, adopting both the coarse-grained and the fine-grained taxonomies of arabic dialects. moreover, we have studied the performance of domain adaptation from the msa to arabic dialects using social media data. the overall obtained results showed that domain adaptation methods outperform zero-shot transfer from bert model by a large margin. furthermore, our method achieved a very promising performance and surpassed the evaluated methods on most test datasets. in future work, we plan to investigate domain adaptive pre-training by collecting unlabeled data for target domains and fine-tuning arabert using the mlm objective. the aim is to study the performance of our method using domain aware language model. since the zero-shot transfer performance using bert model drops significantly in cross-domain sentiment analysis experiments, we believe that training domain adaptation methods on top of domain aware bert model will lead to improved performance. we also plan to study domain adaptation from rich-resource languages such as english to arabic language and its dialects.