StanceEval 2024: The First Arabic Stance Detection Shared Task (2024)

Abstract: Recently, there has been a growing interest in analyzing user-generated text to understand opinions expressed on social media. In NLP, this task is known as stance detection, where the goal is to predict whether the writer is in favor, against, or has no opinion on a given topic. Stance detection is crucial for applications such as sentiment analysis, opinion mining, and social media monitoring, as it helps in capturing the nuanced perspectives of users on various subjects. As part of the ArabicNLP 2024 program, we organized the first shared task on Arabic Stance Detection, StanceEval 2024. This initiative aimed to foster advancements in stance detection for the Arabic language, a relatively underrepresented area in Arabic NLP research. This overview paper provides a detailed description of the shared task, covering the dataset, the methodologies used by various teams, and a summary of the results from all participants. We received 28 unique team registrations, and during the testing phase, 16 teams submitted valid entries. The highest classification F-score obtained was 84.38.

## introduction the rapid expansion of social media platforms, online news sources, and digital communication has significantly increased user-generated content in recent years. this surge in online interactions has created a growing need for automated tools and techniques to analyze the opinions and attitudes expressed in these vast text streams effectively. stance detection, a crucial task in natural language processing (nlp), aims to identify a writer's position or perspective on a specific topic or entity by analyzing their written text and social media activity, including preferences and connections. it has various applications in marketing, politics, and social media analysis. stance detection is closely related to sentiment analysis. while sentiment analysis focuses on identifying the explicit sentiment polarity of a text-categorized as positive, negative, or neutral-stance detection classifies the viewpoint of a text towards a specific target as favor, against, or none. the target in stance detection is often abstract, like ideological topics, and may not be explicitly mentioned in the text, whereas sentiment analysis usually deals with non-ideological subjects. additionally, the alignment between sentiment and stance in a text can vary; a text may have a positive sentiment while opposing the target, or vice versa. in this paper, we summarize the results of the stanceeval 2024 foot_0 shared task where participants were asked to develop models that detect writers' stances (favor, against, or none) towards three topics: covid-19 vaccine, digital transformation, and women empowerment. the mawqif dataset, comprising arabic tweets collected from twitter, was used for training and evaluating the proposed models. each tweet in this dataset has been manually annotated with three labels: stance, sentiment, and sarcasm. ## literature review stance detection involves classifying a writer's stance on a given subject (target) based on written text (input). the output is typically categorized into one of the following: favor, against, none @xcite . the majority of stance detection research has concentrated on supervised machine learning models developed using manual human annotations @xcite . this includes approaches using support vector machines (svm) @xcite , convolutional neural networks @xcite , and recurrent neural networks @xcite . unsupervised and weakly supervised learning approaches have also been used for stance detection, including the use of graph neural networks, inferring user's stance based on their past tweets and the tweets of their interaction graph neighbors @xcite and label propagation on user interaction networks @xcite . while unsupervised and weakly supervised techniques may not achieve the same level of accuracy as supervised learning methods, they address the challenge of obtaining a sufficient amount of labeled data @xcite . transfer learning is another powerful paradigm utilized for stance detection. it leverages a pretrained model, initially trained on a broad, general task, and adapts it for the specific target task of stance detection. this approach significantly reduces the labeled data required for the target task, enhancing efficiency and effectiveness in model training @xcite . transductive transfer learning was mainly used for domain adaptation @xcite and cross-lingual learning @xcite . as for inductive transfer learning approaches, sequential transfer learning is the most commonly used type in stance detection, which uses a source task model to improve the target model's performance in a sequential manner @xcite . multitask transfer learning is another type of inductive transfer learning in which several input attributes are learned sequentially or in parallel. in the context of stance detection, these attributes include rumor veracity, sarcasm, and sentiment @xcite @xcite . with the advent of large language models (llms), researchers have started exploring their potential in stance detection @xcite . a recent benchmarking study by @xcite on multiple stance detection datasets (e.g., semeval 2016 (mohammad et al., 2016a), covid-lies @xcite , and phemerumors @xcite ) demonstrated that few-shot llm prompting results can be competitive with supervised models. however, these results often show inconsistent performance @xcite . due to the inconsistency, marginal performance differences, and the high inference cost, it remains an open research question whether llms should be employed for stance detection @xcite . in this shared task, we invite participants to develop and evaluate stance detection models specifically tailored to the arabic language. alternatively, participants can explore various prompting styles of llms to perform stance detection on the mawqif dataset @xcite , the first dataset designed for arabic stance detection. ## task description for the stanceeval-2024 task, participants are invited to showcase their approaches to stance detection. solutions may utilize machine learning techniques tailored specifically for stance detection or explore alternative methodologies. participants have the freedom to choose between single-task or multi-task learning (mtl) paradigms. in singletask learning, the focus lies solely on stance data for model development and training. conversely, mtl-based models offer the flexibility to incorporate additional information, such as sentiment and sarcasm cues from each tweet, to enhance the performance of the stance detection system. the provided dataset encompasses annotations for stance, sentiment, and sarcasm for each tweet. specifically, participants are tasked with developing models capable of detecting writers' stances towards three specific topics: covid-19 vaccine, digital transformation, and women empowerment. the possible stance labels are as follows: examining the data annotations unveils a disparity between stance and sentiment in texts. it is evident that a tweet can convey a negative sentiment despite holding a favorable stance, and vice versa. ## conclusion in conclusion, the stanceeval shared task, organized as part of the arabicnlp 2024 program, has made significant strides in advancing arabic stance detection. with the participation of 28 teams, 16 of which submitted valid entries and 13 provided detailed description papers, it reflects the growing interest and engagement in this research area. the comprehensive analysis presented in this paper underscores the diverse approaches and methodologies employed by the teams, offering valuable insights into the current state of stance detection in arabic texts. the results and comparative analysis not only highlight the strengths and limitations of various models but also lay the groundwork for future improvements and innovations. this shared task represents a crucial step towards the development of more effective and robust tools for analyzing user-generated text, with broad applications across politics, marketing, and social media analysis.