PICT at S tance E val2024: Stance Detection in A rabic using Ensemble of Large Language Models (2024)

Abstract: This paper outlines our approach to the StanceEval 2024- Arabic Stance Evaluation shared task. The goal of the task was to identify the stance, one out of three (Favor, Against or None) towards tweets based on three topics, namely- COVID-19 Vaccine, Digital Transformation and Women Empowerment. Our approach consists of fine-tuning BERT-based models efficiently for both, Single-Task Learning as well as Multi-Task Learning, the details of which are discussed. Finally, an ensemble was implemented on the best-performing models to maximize overall performance. We achieved a macro F1 score of 78.02% in this shared task. Our codebase is available publicly.

## introduction in recent years, the exponential growth of social media platforms, online news outlets and digital communication has also led to a surge of usergenerated content. effectively analyzing the opinions and attitudes expressed within such content to understand the perspective of the user regarding the topic is called stance detection, which is a critical task in the field of natural language processing (nlp). previous works like @xcite have focused on multiple definitions of the task. the most widely used one (topic-phrase stance) is the stance of a text -favor, against, neutral is detected concerning a topic like 'women empowerment'. arabic is the fifth most spoken language in the world with over 420 million speakers worldwide. stance detection has been extensively studied for english and other languages, but the lack of research on this topic for arabic language exposes the gap in research. reasons for this are challenges due to unique scripting, morphology and * first author, equal contribution 1 http://github.com/ishaan-shukla10/ stanceeval2024 dialects, which increase the complexity of texts to be processed by machine learning models. one other reason is the limited data available in arabic for stance detection. there are extremely limited datasets available like @xcite which covers topics like the war in syria and has very few data points and @xcite which deals with news titles. the stanceeval shared task @xcite at the arabicnlp conference aims to detect writers' stances towards three selected topics like covid-19 vaccine, digital transformation and women empowerment on the mawqif dataset @xcite . earlier approaches like @xcite use character-level lstms and a multilingual variant of bert (mbert) @xcite . other approaches include the usage of bert-based models for zero shot stance detection @xcite , using gradient boosting classifiers @xcite or using end-to-end memory networks @xcite . we built upon the approach used by @xcite using language specific bert based models like camelbert @xcite , marbert @xcite and arabert @xcite and some of their variants. we ranked 7th in this shared task, with a net macro-f1 of 78.02%. individual macro-f1 scores for categories of 'women empowerment', 'covid vaccine' and 'digital transformation' were 75.52%, 79.85% and 78.69% respectively. the subsequent sections describe our approach used for the task. we also analyze the performance of our pipeline and present ablations after rigorous experiments. ## data the dataset contains 3502 examples belonging to 3 distinct topics -covid-19 vaccine, women empowerment and digital transformation. the distribution of examples across these topics is evenly figure 2: multi-task learning architecture distributed. the possible values of stance are -favor, against and none. there is a slight class imbalance in this category. the class 'favor' has the highest number of examples (2148) whereas the class 'none' has the least (333). the data distribution is illustrated in figure 1. the dataset was a multi-label dataset with each data point having an annotation for stance, sarcasm and sentiment. we used a train-validation split of 80:20 on the given dataset. the training set had 2805 examples and the validation set had 697 examples. the test dataset contained 619 examples on which the system was finally evaluated. we observed a significant imbalance in the distribution of class labels. to mitigate that we used methods like weighted random sampling, however, there was no performance improvement even after using such methods. hence, we did not use any data sampling methods while training the final system. ## system description the problem of stance detection was a multi-class classification problem, in which the stance of given tweets was to be evaluated from a total of 3 classes. both, single-task learning (stl) and multi-task learning (mtl) models were implemented in the system. we decided to use multi-task learning because it can improve performance on the primary task while aiding in generalization due to the inductive bias introduced by the additional tasks @xcite . to aid in mtl we used a focal loss in order to provide the right gradient signals to aid the primary task. the equation of the loss used is shown in equation 1. the sum of the individual task-wise losses was divided by 6 for normalization. the primary task was stance detection and the auxiliary tasks were sarcasm detection and sentiment detection. it can be clearly observed from then, depending on their performance on testing data, the best models were selected without any discrimination between stl and mtl models. we used models like marbert, marbertv2, arabert, camelbert, qarib @xcite their variants as the backbones in our system in the various experiments conducted. the pretrained checkpoints for all the models were used through the huggingface @xcite . we used basic pre-processing like the removal of links, mentions and english words. also, the tweets contained emojis. to effectively deal with them we decided to replace each emoji with its description in arabic. the models were then fine-tuned by connecting a linear layer to the pooler output was to the pooler output. hyperparameter tuning was carried out to obtain the best possible model performance, by considering learning rate, weight decay, number of epochs, batch size and context length. for the top 3 models, the best set of hyperparameters we found are mentioned in a total of 13 models were used for experimentation. the models were the variants of the models mentioned above post-trained or fine-tuned on domain-specific data like tweets or dialect identification. the results of the top 5 models are depicted in the test dataset can be seen in we explored both single-task learning (stl) and multi-task learning (mtl) setups, finding that the mtl approach, which incorporated auxiliary tasks of sarcasm detection and sentiment analysis, outperformed the stl models. we also show that an ensemble of the best performing stl and mtl models achieved a strong macro f1-score of 78.02% on the evaluation dataset, with the models performing best on the covid-19 vaccine topic, followed by digital transformation and women empowerment. we also show that multi-task learning benefits the models and its drawbacks. we foresee several future directions for the work done. one direction can be to develop and use more sophisticated methods for ensembling and multi-task learning. another direction is the development of systems that can achieve similar performance in low-resource settings either using synthetic data or by using efficient architectures.