Rasid at S tance E val: Fine-tuning MARBERT for A rabic Stance Detection (2024)

Abstract: As social media usage continues to rise, the demand for systems to analyze opinions and sentiments expressed in textual data has become more critical. This paper presents our submission to the Stance Detection in Arabic Language Shared Task, in which we evaluated three models: the fine-tuned MARBERT Transformer, the fine-tuned AraBERT Transformer, and an Ensemble of Machine learning Classifiers. Our findings indicate that the MARBERT Transformer outperformed the other models in performance across all targets. In contrast, the Ensemble Classifier, which combines traditional machine learning techniques, demonstrated relatively lower effectiveness.

## introduction in recent years, the rapid expansion of social media platforms, online news outlets, and digital communication has resulted in a significant rise in usergenerated content. this unprecedented increase in online discourse necessitates automated tools and techniques to effectively analyze the opinions and attitudes expressed within these extensive streams of text. stance detection, a crucial task in natural language processing (nlp), seeks to identify a writer's position or perspective on a specific topic or entity by examining their written text and social media activity, such as preferences and connections @xcite . stance detection differs from sentiment analysis in both focus and application. while sentiment analysis is concerned with identifying the emotional tone of a text, classifying it as positive, negative, or neutral, stance detection aims to determine the author's precise viewpoint or attitude toward a particular target or topic, identifying whether they are in favor, against, or neutral @xcite . in this paper, we present our contributions to the shared task focused on developing models for detecting writers' stances (favor, against, or none) towards three selected topics: covid-19 vaccine, digital transformation, and women empowerment @xcite . we evaluate three models for stance detection in the context of arabic micro-blogs, namely fine-tuning mar-bert @xcite , fine-tuning arabert transformer @xcite , and an ensemble classifier. the task is challenging due to the unique nature of the arabic language, the informal writing styles prevalent on social media platforms, and the class imbalance in the dataset. this paper is organized as follows: section 2 provides a detailed description of the dataset used. section 3 introduces the proposed models for stance detection in the arabic language. section 4 discusses the results obtained from our experiments. finally, section 5 presents the conclusion. ## data this shared task is based on the mawqif @xcite dataset, an arabic targetspecific stance detection dataset collected from twitter (now x). given three specific targets, it contains multiple labels for each tweet, describing the polarities of stance, sentiment and sarcasm. figure 1 shows word clouds for each of the targets. two data splits are provided for this task, train and blind test. train contains 3,502 samples for all targets, with 1167 for the covid vaccine target, 1145 for the digital transformation target and 1190 for the women empowerment target. figure 2 shows the division of stance labels by target. we notice that the stance for covid vaccine is equally divided between in favor and against, whereas digital transformation has tweets mostly in favor. we further split train with proportions 0.8 and 0.2 to generate a validation set. we used the raw text for fine-tuning and testing our models, without any preprocessing, since normalizing and cleaning the text did not improve the the models' performance on this task. ## system our submission involved fine-tuning a transformerbased bert language model with a 3-way classification layer on top. we used a model that has been trained on both dialectical arabic (da) and modern standard arabic (msa), marbert @xcite . we chose this model because its pretraining data included tweets, which contain informal and dialectical arabic, similar to the stanceeval task. the marbert model has the same architecture of bert base , which contains approximately 163m parameters, namely, 12 layers, 768 hidden size, and 12 heads @xcite . we used the huggingface library to fine-tune and evaluate our models @xcite . we trained against nothing favor stance 0 200 400 600 800 count target women empowerment covid vaccine digital transformation the model for 5 epochs with a 0.00001 learning rate, and 64 sample batch size. the maximum length for each sample was set to 256 tokens. moreover, we trained two additional models as baselines: a fine-tuned arabert (antoun et al., 2020) and an ensemble classifier. the ensemble classifier combines logistic regression, support vector machine, and multinomial naive bayes, augmented with a tf-idf vectorizer utilizing both word n-grams (@xmath0,2) and character n-grams (@xmath1,5). ## conclusion in this paper, we presented our submitted models for the stance detection in arabic language shared task, where we evaluated three distinct models: a fine-tuned marbert model, a finetuned arabert model, and an ensemble classifier. our evaluation revealed that the marbert transformer consistently outperformed the others across various targets. the arabert transformer also showed competitive performance but fell slightly short of mar-bert's overall efficacy. the ensemble classifier, which integrated various traditional machine learning techniques, achieved lower performance. this result underscores the challenges traditional machine learning models face in handling the complexity of stance detection tasks compared to deep learning models with contextual embeddings. future research should focus on enhancing these models, integrating additional linguistic features, enlarging the dataset, and addressing the existing class imbalance to further improve stance detection accuracy in arabic language.