A Benchmark Study of Contrastive Learning for A rabic Social Meaning (2022)

Abstract: Contrastive learning (CL) has brought significant progress to various NLP tasks. Despite such a progress, CL has not been applied to Arabic NLP. Nor is it clear how much benefits it could bring to particular classes of tasks such as social meaning (e.g., sentiment analysis, dialect identification, hate speech detection). In this work, we present a comprehensive benchmark study of state-of-the-art supervised CL methods on a wide array of Arabic social meaning tasks. Through an extensive empirical analysis, we show that CL methods outperform vanilla finetuning on most of the tasks. We also show that CL can be data efficient and quantify this efficiency, demonstrating the promise of these methods in low-resource settings vis-a-vis the particular downstream tasks (especially label granularity).

## introduction proliferation of social media resulted in unprecedented online user engagement. people around the world share their emotions, fears, hopes, opinions, etc. online on a daily basis @xcite on platforms such as facebook and twitter. hence, these platforms offer excellent resources for social meaning tasks such as emotion recognition @xcite , irony detection @xcite , sarcasm detection @xcite , hate speech identification @xcite , stance identification @xcite , among others. while the majority of previous social meaning studies were carried out on english, a fast-growing number of investigations focus on other languages. in this paper, we focus on arabic. several works have been conducted on different arabic social meaning tasks. some of these focus on modern standard arabic (msa) @xcite , while others take arabic dialects as their target (elsahar and el-beltagy 2015; al @xcite . while many works have focused on sentiment analysis, e.g., @xcite @xcite @xcite @xcite and dialect identification @xcite @xcite @xcite @xcite , others focused on detection of user demographics such as age and gender @xcite , irony detection @xcite , and emotion analysis @xcite . our interest in the current work is improving arabic social meaning through representation learning. in spite of recent progress in representation learning, most work in arabic social meaning mostly focuses on finetuning language models such as arat5 @xcite , camelbert @xcite , marbert @xcite , qarib @xcite , among others. in particular, arabic social media processing has to date ignored the emerging sub-area of contrastive learning (cl) @xcite . given a labeled dataset, cl @xcite attempts to pull representations of the same class close to each other while pushing representations of different classes further apart (figure 1 ). in this work, we investigate five different supervised contrastive learning methods in the context of arabic social meaning. to the best of our knowledge, this is the first work that provides a comprehensive study of supervised contrastive learning on a wide range of arabic social meanings. we show that performance of cl methods can be task-dependent. we attempt to explain this performance from the perspective of task specificity (i.e., how fine-grained the labels of a given task are). we also show that contrastive learning methods generally perform better than vanilla finetuning based on cross entropy (ce). through an extensive experimental study, we also demonstrate that cl methods outperform ce finetuning under resource-limited constraints. our work allows us to demonstrate the promise of cl methods in general, and in low-resource settings in particular. to summarize, we offer the following contributions: ## methods given a set of training examples {x i , y i } @xmath0,...,n and an encoder based on a pre-trained language model (plm), f outputs contextualized token representation of x i , where h is the hidden representation of the final layer of the encoder. the standard practice of finetuning plms passes the pooled representation h @xcite of @xcite to a softmax classifier to obtain the probability distribution for the set of classes c (figure 2a ). ## datasets in this section, we present the arabic social meaning tasks and datasets used in our study. a summary of the datasets is presented in abusive and adult content. for the abusive and adult content detection tasks, we use datasets from @xcite and @xcite . these datasets consist of 1.1k and 43k tweets, respectively. for these datasets, the goal is to classify an arabic tweet into one of the two classes in the set, i.e., {obscene, clean} for the abusive task, and {adult, not-adult} for the adult content detection task. age and gender. for both tasks, we use the arap-tweet dataset (zaghouani and charfi, 2018) which consists of 1.3m, 160k, 160k for the train, dev, and test respecctively. the dataset covers 11 arab regions. zaghouani and charfi (2018) assign age group labels from the set {under-25, 25-to-34, above-35} and gender from the set {male, female}. dangerous. we use the dangerous speech dataset from alshehri et al. (2020). this dataset consists of 4, 445 manually annotated tweets labelled as either safe or dangerous. dialect identification: six datasets are used for this task: arsarcasm dia (farha and magdy, 2020), the arabic online commentary (aoc) (zaidan and callison-burch, 2014), nadi-2020 (abdul-mageed et al., 2020a), madar (bouamor et al., 2019), qadi (abdelali et al., 2020), and habibi (el-haj, 2020). the dialect identification task involves three dialect classification levels: (1) binary-level (msa vs. dia), (2) region-level (4 regions), and (3) country-level (21 countries). emotion. for this task, we use two datasets: aranet emo and semeval emo . the first one is proposed by abdul-mageed et al. (2020c). the dataset consists of 192k tweets labeled with the eight emotion classes from the set {anger, anticipation, disgust, fear, joy, sadness, surprise, trust}. semeval emo (mohammad et al., 2018) consists of 5, 603 tweets labeled with four emotions from the set {anger, fear, joy, sadness}. offensive language and hate speech. we use the dataset released by mubarak et al. (2020) during an offensive and hate speech shared task. foot_1 this dataset consists of 10k manually annotated tweets with four tags {offensive, not-offensive, hate, nothate irony. we use the irony identification dataset for arabic tweets (idat) developed by ghanem et al. (2019). this dataset contains 5, 030 msa and dialectal tweets. it is labeled with ironic and nonironic tags. sarcasm. we use the arsarcasm dataset released by (farha and magdy, 2020). arsarcasm contains 10, 547 tweets. the tweets are labeled with sarcasm and not-sarcasm tags. sentiment analysis this task includes 19 sentiment datasets. we merge the 17 datasets benchmarked by abdul-mageed et al. (2021a) with two new datasets: arabizi sentiment analysis dataset (fourati et al., 2020) and aracust (almuqren and cristea, 2021), a saudi telecom tweets corpus for sentiment analysis. the data contains 190k, 6.5k, 44.2k samples for train, dev and test. the dataset is labeled with three tags from the set {positive, negative, neutral}. ## experimental setup we implement all the methods using marbert @xcite (ubc-nlp/marbert) from huggingface's transformers library @xcite , as the backbone architecture. we use marbert as it is reported to achieve sota on a wide range of arabic language understanding tasks in @xcite . our methods, however, can be applied to any other model. we use the same hyperparameters for all the methods to ensure fair comparisons. we set the maximum sequence length to 128 and use a batch size of 16 to train the models using adam optimizer with a learning rate 5e -5. the initial number of training epochs is set to 25 with an early stopping threshold of 5. for cl-based models, we set λ to 0.5 and τ to 0.3. for all the experiments, we consider the checkpoint with the best macro f 1 score on the development sets to evaluate performance on the respective test sets. to limit gpu usage during our experiments, we normalize all datasets considered by limiting the size of train, dev, and test splits to 50k, 5k, 5k samples respectively. 3 6 results as explained, we compare different methods on 15 different arabic social media datasets involving binary and multiclass classification. we present performance of the methods in it is important to note that there is no unique superior method across the tasks. this shows that cl-based methods can be task-specific, depending on the nature of how they are formulated. for example, lcl performs well on multiclass datasets such as abusive and aranet emo , while tlcl performs well on semeval emo . lcl and tlcl adopt more fine-grained representations with the incorporation of the weighting network which consequently helps them distinguish confused classes. however, for dialect at regionlevel, we speculate that since the labels are already fine-grained, it is more important to improve the robustness rather than inter-label relationship. therefore, cat achieves best performance on this task, followed by tlcl. similarly, on binary classification tasks such as hate speech and offensive language detection, where a subtle semantic change in meaning can alter the labels, robust methods are expected to outperform others. therefore, adversarial methods like cat and tact achieve better f 1 -score. for most of the tasks, f 1 -scores obtained from different cl-methods are close to each other and the vanilla scl achieves similar average score to the other models. this proves that although taskspecific formulation may help the models to improve on a certain task, the most important factor evolves around the fundamental minmax nature of contrastive learning which is minimizing the distance among the representations of the same class while maximizing the distance among the representations of the different classes. ## analysis 7.1 data efficiency to investigate how the methods perform with limited data, we train the models under different size constraints using three datasets (one binary and we randomly pick 50k, 5k, and 5k samples respectively. ## limitations an inherent limitation of cl methods is their reliance on hyperparameters. in particular, they are sensitive to batch size. larger batch sizes usually yield better performance. other hyperparameters like τ and λ can also impact performance given a specific task. lastly, the accommodation of larger batch size comes at the cost of higher computational resources. ## conclusion in this work, we study various supervised contrastive learning methods for a wide range of arabic social meaning tasks. we show that cl-based methods outperform generic cross entropy finetuning for majority of the tasks. through empirical investigations, we find that improvements resulting from applying cl methods are task-specific. we interpret these results vis-a-vis different downstream tasks, with a special attention to the number of classes involved in each task. finally, we demonstrate that cl methods can achieve better performance with limited training data and hence can be employed for low-resource settings. in the future, we plan to extend our work beyond sentence classification by experimenting on tasks such as token-classification and questionanswering. our work stands as a comprehensive investigation of applying contrastive learning to arabic social meaning. we hope this work will trigger further investigations of cl in arabic nlp in general.