TAO at S tance E val2024 Shared Task: A rabic Stance Detection using A ra BERT (2024)

Abstract: In this paper, we present a high-performing model for Arabic stance detection on the STANCEEVAL2024 shared task part ofARABICNLP2024. Our model leverages ARABERTV1; a pre-trained Arabic language model, within a single-task learning framework. We fine-tuned the model on stance detection data for three specific topics: COVID19 vaccine, digital transformation, and women empowerment, extracted from the MAWQIF corpus. In terms of performance, our model achieves 73.30 macro-F1 score for women empowerment, 70.51 for digital transformation, and 64.55 for COVID-19 vaccine detection.

## introduction stance detection is a critical domain within natural language processing (nlp) that focuses on the automatic identification of attitudes or opinions expressed toward specific entities or propositions within textual data. with the ubiquity of the internet in our everyday lives, there has been a substantial increase in user-generated content online, where stances are primarily articulated through written comments or texts on forums, blogs, and social media platforms such as youtube, instagram, facebook, and twitter. while significant research has yielded powerful stance detection models for languages like english, arabic stance detection research still requires attention. although arabic language is one of the top ten global languages with over 420 million native speakers @xcite @xcite , arabic presents unique challenges for nlp applications due to its rich and complex morphology and the variety of its dialects @xcite . unlike formal media that utilize modern standard arabic (msa), social media platforms primarily feature dialectal arabic with an informal writing style, often replete with spelling errors, abbreviations, irregular grammar, emojis, and symbols. this inherent disparity creates a significant obstacle to automated stance detection on social media. mawaqif dataset (alturayeif et al., 2022) is considered the first arabic dataset for stance detection. this dataset, which forms the foundation of the stanceeval () shared task, stands out for its extensive reach, encompassing more than 4000 tweets in multi-dialectal arabic targeting three topics: "covid-19 vaccine", "digital transformation", and "women empowerment". these are carefully annotated with stance, sentiment, and sarcasm polarities. the shared task () highlights two principal stance detection challenges: although mtl models may outperform stl models. this paper addresses the problem of building a robust arabic stance detection model through the single-task learning approach. it allows a focused analysis of the linguistic and contextual features specific to the arabic language without the potential complexities and resource demands associated with multi-task learning frameworks. this approach ensures that the model is specifically tailored to recognize nuances and subtleties in arabic discourse, leveraging a rich dataset curated for stance detection to optimize accuracy and performance. our paper offers the following contributions: • we present an stl-based model that utilizes arabert v1 @xcite for detecting stances on the topics of the covid-19 vaccine, digital transformation, and women's empowerment. the organization of this paper is as follows. in section 2, we present prior and recent research on arabic stance detection. section 3 presents a comprehensive analysis of the dataset. section 4 describes our proposed system and experimental setup. section 5 presents and discusses our experimental results. in section 6, we conclude and point out ideas for future search. ## related works addressing the problem of arabic stance detection using stl approach has drawn the attention of researchers recently. in (alrowais and al-saeed, 2023), four stance detection models were built and compared using stl approach leveraging the transformers araelectra @xcite , marbert @xcite , arabert v1 (antoun et al., 2020) and qarib @xcite . in @xcite , more than 1.8 million tweets were collected to gauge public opinions on distance education in ksa during the 2020 academic year. various machine-learning and deep-learning models were trained and compared using stl approach. @xcite constructed and released austr, a smallsized dataset to address the problem of detecting the stance of authorities towards rumors in tweets. arabert (antoun et al., 2020) is trained using stl approach to detect the stances in their work. mtl approach has been used lately to address the problem of arabic stance detection in @xcite . ## data this section provides a comprehensive description of the dataset released by the stanceeval organizers. the dataset comprises over 4000 arabic tweets, encompassing diverse tweets representing different stances: in favor, against, or neutral. the tweets are targeted towards one of three topics: "covid-19 vaccine", "digital transformation", and "women empowerment". each tweet is not only labeled for stance but also for sentiment and sarcasm. this comprehensive labeling allows for a more in-depth study of how different opinions interact and for evaluating a model trained on multiple opinion dimensions in a multi-task paradigm. ## results and discussion we divided the dataset into training, validation, and testing sets, allocating 70%, 20%, and 10% of the data, respectively. our model, tao, was then evaluated on the test set, achieving a macro-f1 score of 92.36 for digital transformation, 91.34 for women empowerment, and 90.36 for covid-19 vaccine detection. note that we performed our evaluation and calculated the macro f1-score using the python script provided by the organizers. thus, the macro f1-score is calculated as the average of the f1-scores for the "favor" and "against" categories. the stance detection performance of tao on our testing set, as compared with benchmark models reported by @xcite , is presented in the performance of tao declined when evaluated on the blind dataset provided by the shared task organizers. it achieved a macro-f1 score of 64.55 for digital transformation, 73.30 for women empowerment, and 70.51 for covid-19 vaccine detection. several factors may account for this performance decline. one major issue is the imbalanced distribution of 'favor' and 'against' instances within the digital transformation and women empowerment targets, as well as across the entire dataset. additionally, the underrepresentation of the 'none' class posed a challenge for the model to learn and accurately predict these instances. furthermore, the larger size of the blind test set compared to our own test set likely contributed to an increased prediction error. ## conclusion in this paper, we present our bert-based system for arabic stance detection, developed for the ara-bicnlp2024 stanceeval shared task. our model utilizes arabertv1, a pre-trained arabic language model, within a single-task learning framework. the model achieves macro-f1 scores of 73.30 for women empowerment, 70.51 for digital transformation, and 64.55 for covid-19 vaccine detection. for future work, we plan to explore the training of different systems, such as marbert, arabert-twitter, and xlm-r. additionally, we aim to employ a multi-task learning (mtl) approach to further enhance the performance of our stance detection models.