S tance C rafters at S tance E val2024: Multi-task Stance Detection using BERT Ensemble with Attention Based Aggregation (2024)

Abstract: Stance detection is a key NLP problem that classifies a writer’s viewpoint on a topic based on their writing. This paper outlines our approach for Stance Detection in Arabic Language Shared Task (StanceEval2024), focusing on attitudes towards the COVID-19 vaccine, digital transformation, and women’s empowerment. The proposed model uses parallel multi-task learning with two fine-tuned BERT-based models combined via an attention module. Results indicate this ensemble outperforms a single BERT model, demonstrating the benefits of using BERT architectures trained on diverse datasets. Specifically, Arabert-Twitterv2, trained on tweets, and Camel-Lab, trained on Modern Standard Arabic (MSA), Dialectal Arabic (DA), and Classical Arabic (CA), allowed us to leverage diverse Arabic dialects and styles.

## introduction stance detection or identification is a growing research topic in the fields of natural language processing (nlp), social media analysis, and information retrieval (ir). it seeks to classify a writer's viewpoint on a certain topic based on their writing towards a target (e.g., notion, idea, event) @xcite . it is known to have several applications, including predicting election/referendum results, retrieving information, and classifying rumors @xcite . stance detection is known to be closely related to sentiment analysis, with the target classes favor, against, and none @xcite . it is also related to other tasks, such as sarcasm detection and irony detection. despite its relevance, it has gotten little attention in the arabic language. data-driven neural networks have demonstrated outstanding performance for a wide variety of tasks. traditionally, neural networks are trained separately for each task. however, neural networks frequently require extensive labeled training samples to achieve superior results @xcite . given the limited data for arabic stance identification and the availability of related tasks, it is necessary to employ novel techniques to construct robust models. multi-task learning (mtl) is an approach that handles numerous tasks concurrently and can leverage knowledge from related tasks to address data shortages @xcite . it assumes that all learning processes, or at least a portion of them, are related to one another. learning from multiple common tasks simultaneously allows models to collect generic information beyond task-specific characteristics @xcite . mtl has been used in the field of arabic stance identification with various weighting strategies, yielding better results than training a model using only the stance detection task @xcite . this paper details our work on the stance detection in arabic language shared task (stanceeval2024) @xcite . the stanceeval2024 shared task identifies writers' attitudes (favour, against, or none) on covid-19 vaccination, digital transformation, and women empowerment. the key challenge in this task is imbalance of importance between the tasks. therefore, we apply parallel mtl combining sentiment, sarcasm, and stance tasks to share knowledge across multiple tasks. moreover, we utilize diverse weighting loss techniques to give more importance to the primary task. the remaining sections of the paper are organized as follows: section 2 describes the stanceeval2024 dataset, section 3 describes the methodology, section 4 demonstrates the experimental results, section 5 discusses the key findings, and section 6 concludes the work. the code has been made open-source and available on github on the link the mawqif dataset is a multi-label arabic dataset designed for stance identification @xcite . it includes 4121 tweets around three topics: covid-19 vaccine, digital transformation, and women's empowerment. each tweet is annotated with stance, sentiment, and sarcasm. the dataset is separated into two sets: training and testing. training accounts for 85% of the data, while testing accounts for 15%. in this study, we only considered the stance labels: against and favour. the dataset is preprocessed using the arabertpreprocessor using the bert-base-arabertv02-twitter model. the dataset was further pre-processed by removing all the hashtags, urls, mentions, punctuation, and repeating characters, as well as applying normalization. ## system this study explored using multiple bert models for stance detection in arabic text. we began with a single bert model in the shared layers and then, inspired by the work in @xcite and @xcite , we experimented with combining two models (arabert, arabert-twitter, camel-bert) to leverage their diverse training data. we evaluated different weighting and aggregation techniques (averaging, attention) to find the optimal combination (figure 1 ). finally, we explored various loss weighting methods (static, relative) to further improve performance. during training, the following configurations were used: 20 epochs, learning rate of 2e-5, weight decay of 1e-5, dropout of 0.1, batch size of 4, seed value of 42, and max token length of 128. the models were validated based on the validation f1score. ## results in this section, the results of the developed models are reported during the development and test phases. ## discussion our suggested model exceeded the findings from the mawqif dataset by 2.79%. we discovered that pairing arabert-twitter with camelbert yielded the greatest results. arabert's success can be attributed to the mawqif dataset, which was acquired via twitter. arabert-twitter was trained on tweets that were not modern standard arabic (msa), whereas camelbert was trained on msa, dialectal arabic (da), and classical arabic (ca), consequently combining them enhanced the results. we also observed that aggregating the models using the attention mechanism performed better than the mean mechanism. the attention mechanism can outperform the mean mechanism in mtl since it dynamically weights model contributions based on their relevance. moreover, we found that using static weighting leads to attaining the best performance. we assume that the reason behind this is the low confidence present in the dataset. static weighting is less sensitive to noise and outliers in the loss values. the proposed framework performed relatively better on the women's empowerment target and relatively worse on the covid vaccine target. we believe the cause is that we fine-tuned the models using against and favour labels. woman empowerment has the fewest none labels, whereas covid vaccine has most none labels. encouraged by the success of mtl with attention-based combination, a future direction can be in regards to exploring graph neural networks (gnns). this would involve constructing a graph where each node represents a bert model's output, and then combining them using techniques like graph convolution, similar to work in @xcite , @xcite . moreover, incorpo-rating more weighting techniques, such as hierarchical and uncertainty weighting could be a direction to improve the results. employing uncertainty sampling is another way to improve generalization with less data as suggested in @xcite . ## conclusion this study explored an mtl approach for stance detection in arabic text, focusing on attitudes towards covid-19 vaccines, digital transformation, and women's empowerment. the proposed model architecture included a shared task layer comprising arabert-twitter and camelbert, which were combined via an attention mechanism, followed by a task-specific layer for each task. this ensemble approach, combined with static loss weighting, achieved a strong validation f1-score of 91.3668, outperforming architectures with a single bert model in the shared layers. the results suggest that leveraging diverse training data through multitask learning with attention-based combinations can improve the model's ability to capture various features and generalize better. on the test set, while the model excelled at the women's empowerment task (85.06 f1 score), it highlights the need for further development to improve performance on topics like covid-19 vaccination (79.84 f1 score). overall, the model achieved a promising macro f1 score of 81.68 on the blind test set.