A lex UNLP - BH at S tance E val2024: Multiple Contrastive Losses Ensemble Strategy with Multi-Task Learning For Stance Detection in A rabic (2024)

Abstract: Stance detection, an evolving task in natural language processing, involves understanding a writer’s perspective on certain topics by analyzing his written text and interactions online, especially on social media platforms. In this paper, we outline our submission to the StanceEval task, leveraging the Mawqif dataset featured in The Second Arabic Natural Language Processing Conference. Our task is to detect writers’ stances (Favor, Against, or None) towards three selected topics (COVID-19 vaccine, digital transformation, and women empowerment). We present our approach primarily relying on a contrastive loss ensemble strategy. Our proposed approach achieved an F1-score of 0.8438 and ranked first in the stanceEval 2024 task. The code and checkpoints are availableat https://github.com/MBadran2000/Mawqif.git

## introduction the increase in user-generated content is evident across news portals, user forums, blogs, and social media platforms such as twitter, facebook, and instagram. this surge has led to the accumulation of vast amounts of textual data. these texts are readily available for analysis to serve diverse practical purposes @xcite . one such purpose is stance detection, also known as stance classification or stance prediction. for an input in the form of a piece of text and a target pair, stance detection is a classification problem where the stance of the author of the text is sought in the form of a category label from this set: favor, against, neither @xcite . stance detection applications vary from trend and market analysis, obtaining user reviews for products, opinion surveys, targeted advertising, polling, predictions for elections and referendums, automatic media monitoring, and filtering out unconfirmed content for better user experience, to online public health surveillance @xcite . earlier research in stance detection primarily concentrated on analyzing debates within online forums. recent efforts, however, have shifted towards analyzing social media platforms, notably twitter. unlike forum debates that are typically centralized, social media discussions on a given topic tend to be dispersed, although sometimes linked by hashtags. despite this dispersion, the wealth of additional features inherent to social media renders it a valuable source of data for stance detection endeavors @xcite . while a significant portion of stance detection studies has focused on the english language, recent endeavors have aimed at developing stance detection datasets for other languages. however, no comparable initiatives appear to have addressed the challenges within the arabic language @xcite . detecting stance in arabic presents formidable challenges owing to its morphological intricacies and the diverse range of dialects present, which deviate from modern standard arabic @xcite . in this paper, we present our submission to shared task 4: stanceeval2024 @xcite in the second arabic natural language processing conference (arabicnlp2024). the task focuses on stance detection in the arabic language using the mawqif dataset @xcite . we tackled the problem by using multitask model architecture, enhanced by multiple contrastive losses ensemble strategy. our proposed approach achieves an overall f1 score of 0.8438, securing the first rank in the stanceeval 2024 task. ## data for our experiments, we used mawqif dataset only. it consists of 4,121 samples collected from twitter platform. each sample is annotated for three the dataset is distributed as follows across the 3 targets 33.3% (1373 tweets) for covid-19 vaccine, 32.7% (1348 tweets) for digital transformation, and 33.97% (1400 tweets) for women empowerment the distribution of stance categories for each target is presented in we split the dataset into training, validation, and testing datasets with 70% for training, 15% for validation, and 15% for testing. to preprocess the data, we implemented several steps to ensure uniformity and enhance model performance. firstly, we replaced hindi numerals with their english counterparts to standardize numerical representations across the dataset. additionally, we substituted mentions, urls, and email addresses with special tokens to eliminate noise. moreover, for models utilizing arabert v2, we removed emojis entirely as the model was pretrained without them. ## system in this section, we provide a detailed description of our system developed for stance detection in arabic. the multitask model architecture including its associated losses is illustrated in figure 1 . ## results for our experimental settings, all experiments were conducted using a single rtx 3070 gpu. each model was trained for 50 epochs with an early stopping patience of 5 epochs based on validation loss. the batch size was set to 32, and the maximum input sequence length was fixed at 128 tokens. models incorporating dropout had a dropout probability of 0.1. the macro f1 score is used to measure our results. this score is the average of the f1 scores for the "favor" and "against" categories. this score is calculated for each target separately. then, the overall macro f1 score is found by averaging these scores across all targets. since each target in our approach has its own specific model, the f1 score is calculated individually for each model. subsequently, the overall macro f1 score is computed by averaging these scores across all targets. the results obtained from our comprehensive approach and modifications on the official test dataset are presented in ## discussion our results on the official test set for the stanceeval task, as presented in the results from our test set, as presented in in model 3, multitask learning and weighted loss further improved our results, achieving an overall f1 score of 0.8008. the multitask approach enabled the model to simultaneously tackle related tasks, fostering more robust and generalized representations of the data. addressing data imbalance, the weighted loss mechanism adjusted class contributions during training, prioritizing underrepresented classes. this combination of techniques proved highly effective in enhancing the model's performance. expanding our analysis to models 4 to 8, where various contrastive loss functions were introduced, we observed mixed performance improvements. while some contrastive losses contributed positively to the overall performance, not all yielded significant gains. nevertheless, the true efficacy of contrastive loss became evident in its impact on our ensemble strategy. by employing an ensemble of models trained with different contrastive losses, we achieved a notable overall f1 score of 0.8432. however, it's important to note that identifying the optimal combination of contrastive losses is a non-trivial task, requiring extensive experimentation and validation to determine the most effective ensemble configuration. ## conclusion in this paper, we propose a multiple contrastive loss ensemble strategy for arabic stance detection, which enabled us to achieve first place in the stanceeval 2024 task. through our approach, we demonstrated that incorporating dropout and data augmentation mitigates overfitting while using a weighted loss effectively addresses data imbalance, and multi-task learning improves the generalization of our models. by integrating various contrastive losses, we enhanced the distinctiveness of hidden representations, contributing to a more effective ensemble. this work highlights the potential of ensemble techniques and contrastive learning to improve stance detection in arabic texts.