dz S tance at S tance E val2024: A rabic Stance Detection based on Sentence Transformers (2024)

Abstract: This study compares Term Frequency-Inverse Document Frequency (TF-IDF) features with Sentence Transformers for detecting writers’ stances—favorable, opposing, or neutral—towards three significant topics: COVID-19 vaccine, digital transformation, and women empowerment. Through empirical evaluation, we demonstrate that Sentence Transformers outperform TF-IDF features across various experimental setups. Our team, dzStance, participated in a stance detection competition, achieving the 13th position (74.91%) among 15 teams in Women Empowerment, 10th (73.43%) in COVID Vaccine, and 12th (66.97%) in Digital Transformation. Overall, our team’s performance ranked 13th (71.77%) among all participants. Notably, our approach achieved promising F1-scores, highlighting its effectiveness in identifying writers’ stances on diverse topics. These results underscore the potential of Sentence Transformers to enhance stance detection models for addressing critical societal issues.

## introduction stance detection is a pivotal task in natural language processing (nlp) that involves determining the position or attitude expressed in a text regarding a specific topic or entity. this task is crucial for a variety of applications, including sentiment analysis, opinion mining, and social media monitoring, where understanding public sentiment and opinion is essential @xcite . with the exponential growth of user-generated content on social media and online news platforms, there is a pressing need to develop sophisticated tools that can analyze and interpret the myriad of perspectives present in these texts. the mawqif 2022 shared task, an initiative focused on arabic stance detection, addresses this need by challenging participants to detect stances towards three contemporary topics: covid-19 vaccine, digital transformation, and women empowerment. this task is significant not only because it addresses critical societal issues but also because it highlights the complexities of processing arabic text, which is characterized by its rich morphology, diverse dialects, and intricate syntax @xcite . traditional methods for stance detection have largely relied on feature extraction techniques such as term frequency-inverse document frequency (tf-idf). these methods transform textual data into a numerical format, allowing machine learning models to process and analyze the text. our initial experiments leverage tf-idf features due to their simplicity and proven effectiveness in various text classification tasks @xcite . despite their utility, tf-idf-based methods have limitations, particularly in capturing the deeper semantic relationships and contextual nuances within text. to address these limitations, recent research has increasingly focused on deep learning techniques. long short-term memory (lstm) networks have demonstrated their ability to handle sequential data and capture dependencies in text, which are crucial for understanding stances @xcite . the advent of transformer-based models, such as bert and sentence transformers, has revolutionized the field of nlp by offering robust methods to capture semantic and contextual information in text. these models utilize self-attention mechanisms to understand the relationships within text, making them particularly effective for nuanced tasks like stance detection @xcite . transformers have set new benchmarks in various nlp tasks by leveraging their ability to generate dense, context-aware representations of text, facilitating a deeper understanding of the underlying meaning and intent. several studies have explored the use of bert and other transformers for stance detection, demonstrating their superiority over traditional methods in capturing subtle and complex stances in text. for example, @xcite showed that bert-based models outperform traditional approaches in detecting stances in english social media texts. however, these models often require extensive computational resources and fine-tuning, posing challenges for their application in resource-limited settings @xcite . building upon these advancements, our study explores a comparative analysis between traditional feature extraction methods and modern deep learning approaches for arabic stance detection. in our experiments, we employ both tf-idf and sentence transformers to detect stances towards the selected topics. our participation in the mawqif 2022 shared task allowed us to evaluate these methodologies rigorously, where our team, dzstance, achieved competitive results across the three topics, demonstrating the effectiveness of our approaches. in the following sections, we will delve deeper into our methodology and findings. section 2 provides an overview of the mawqif dataset used in our study. section 3 describes our approach to stance detection, including the feature extraction techniques and model architectures we employed. section 4 presents our experimental results and discusses their implications. finally, section 5 concludes the paper by summarizing the key insights and contributions of our work. ## dataset description the mawqif dataset @xcite , utilized in the stanceeval 2024 shared task @xcite , serves as a crucial resource for advancing natural language processing (nlp) in the domain of stance detection. this dataset comprises over 4,000 annotated text samples that encapsulate diverse stances-favorable, opposing, or neutral-on pertinent topics such as covid-19 vaccine, digital transformation, and women empowerment. the significance of the mawqif dataset lies in its ability to provide a comprehensive view of how different opinions and attitudes are expressed in ara-bic text. this makes it invaluable for researchers who aim to evaluate and enhance stance detection models. by leveraging such a dataset, one can explore and refine models to better understand and process nuanced stances within varied contexts. a closer look at the data statistics reveals notable class imbalances. for instance, the covid-19 vaccine category includes nearly equal proportions of favorable (43.53%) and opposing (43.48%) tweets, with a smaller fraction being neutral (12.85%). in contrast, the digital transformation topic shows a predominance of favorable stances (76.77%), with fewer opposing (12.41%) and neutral (1.92%) tweets. similarly, the women empowerment category also leans heavily towards favorable stances (63.95%), followed by opposing (31.18%) and neutral (4.96%) tweets. such imbalances can pose significant challenges for model training and evaluation. models trained on datasets with skewed class distributions may become biased towards the majority classes, leading to suboptimal performance on minority classes. therefore, addressing these imbalances is critical to ensure the development of robust and fair models. techniques like data resampling, class weighting, and the use of advanced algorithms capable of handling imbalance are essential strategies to mitigate these effects and enhance overall model performance. by understanding and leveraging the characteristics of the mawqif dataset, researchers can effectively tackle the complexities of stance detection, contributing to the broader field of arabic nlp and enabling more accurate and nuanced analysis of opinions and attitudes expressed in text. ## proposed system in our proposed system foot_0 , we explore two distinct methodologies for feature extraction: a weighted union of tf-idf features @xcite and sentence transformers. these techniques offer complementary advantages, leveraging the strengths of traditional feature representation and cutting-edge deep learning architectures. ## results and discussion this section evaluates the performance of our stance detection system on the test set using the f1-score as the primary metric. we conducted two sets of experiments to compare the effectiveness of different feature extraction techniques and model configurations. following these, we analyze our competitive performance in the stance detection challenge. ## conclusion in this paper, we introduced dzstance, our solution to the stanceeval 2024 shared task on arabic stance detection. leveraging sentence transformers in conjunction with logistic regression, our approach achieved competitive results with an overall average f1-score of 71.77%. this performance positioned us 13th among all participating teams, highlighting the effectiveness of advanced embedding techniques and robust classification algorithms in handling the complexities of arabic stance detection. the success of our approach underscores the significance of pre-trained models like sentence transformers for capturing nuanced semantic relationships within arabic text across diverse topics. looking ahead, further investigation into why the logistic regression model outperformed traditional methods such as linear support vector classification (lsvc) could yield insights through deeper hyperparameter tuning and broader evaluation on varied datasets. additionally, exploring hybrid approaches that integrate tf-idf features with advanced embedding models may offer enhanced model robustness and accuracy in arabic natural language processing tasks. by making our code and methodologies openly accessible, we aim to foster reproducibility and encourage ongoing advancements in arabic stance detection research, paving the way for more sophisticated and effective models in the future.