T eam_ Z ero at S tance E val2024: Frozen PLM s for A rabic Stance Detection (2024)

Abstract: This research explores the effectiveness of using pre-trained language models (PLMs) as feature extractors for Arabic stance detection on social media, focusing on topics like women empowerment, COVID-19 vaccination, and digital transformation. By leveraging sentence transformers to extract embeddings and incorporating aggregation architectures on top of BERT, we aim to achieve high performance without the computational expense of fine-tuning. Our approach demonstrates significant resource and time savings while maintaining competitive performance, scoring an F1-score of 78.62 on the test set. This study highlights the potential of PLMs in enhancing stance detection in Arabic social media analysis, offering a resource-efficient alternative to traditional fine-tuning methods.

## introduction social media has become a pivotal platform for public discourse, providing a dynamic space where individuals express opinions and engage in debates on a multitude of topics. stance detection, a branch of natural language processing, plays a crucial role in analyzing these interactions by identifying the positions individuals hold towards specific issues. stance detection is particularly significant in understanding public sentiment and the social dynamics around key topics such as women empowerment, covid-19 vaccination, and digital transformation. accurate stance detection enables researchers and policymakers to gauge the prevailing attitudes and respond appropriately, fostering informed decisionmaking and promoting constructive dialogue. in the context of arabic social media, stance detection is essential for capturing the unique cultural and linguistic nuances that shape public opinion in the arab world, thereby enhancing the relevance and impact of social media analytics. in the era of large language models (llms) such as llama @xcite and pre-trained lan-guage models such as bidirectional encoder representations from transformers (bert) @xcite , the nlp community goes more towards using such language models to tackle several downstream tasks such as stance detection. plms such as bert are usually pre-trained on a general language modeling task (e.g. masked language modeling) to give the model a pretty good understanding of the language in general. to use the model then on a downstream task, the model weights are updated on a dataset annotated on that task. such approaches of pre-training language models and then fine-tuning them on downstream tasks have proved to work exceptionally well and boosted the performance on most nlp common tasks. they also have even been successfully applied to less common tasks like depression detection from social media @xcite . although boosting nlp tasks' performance through fine-tuning, the fine-tuning operation is costly in terms of the needed resources and time. fine-tuning plms usually requires powerful gpus to train all the plm weights on the downstream task and relatively high training time compared to simpler machine learning algorithms. in this context, some approaches proposed parameter-efficient finetuning approaches @xcite that mainly aim to minimize the number of parameters being updated on the downstream task. other approaches @xcite showed that plms can be used as feature extractors only without the need to fine-tune them. this is done by adding an extra aggregation architecture on top of the plm with a relatively small number of parameters compared to the plm. these aggregation architectures help extract sentence embeddings that are more powerful for text classification tasks. this work aims to explore the effectiveness of using plms as feature extractors -without any fine-tuning -on arabic stance detection. the major goal of using plms as feature extractors is eliminating the need to fine-tune them and hence saving computational and time resources. firstly, we explore the sentence transformers @xcite to extract sentence embeddings for the arabic tweets. secondly, we follow some of the approaches proposed by @xcite ) by adding aggregation architectures on top of bert to get a more powerful sentence embedding for stance detection. our work was submitted to the stanceeval @xcite competition and ranked fifth out of 16 teams. this shows how competitive plms can be used as feature extractors and gain a competitive performance besides saving the fine-tuning costs. ## data to train our models on arabic stance detection, we used the mawqif dataset @xcite . the dataset consists of 3502 arabic tweets for training and 619 for testing spanning three topics: women empowerment, covid vaccine, and digital transformation. the dataset annotations for the stance are either "favor", "against", or "none". the tweets are also annotated for sentiment and sarcasm to allow the capabilities of multi-task learning approaches. ## methodology this section illustrates the approaches for stance detection. the approaches we followed can be classified into three main approaches: sentence transformers-based approaches, bert-based embeddings, and parallel sum architecture on top of bert. ## discussion a major challenge we faced while tackling the problem is the overfitting. the relatively small size of the training set is the major cause of this problem. it can also be seen that the dataset is already distributed across three targets which makes the size per each target very small. to overcome this issue, we followed several approaches such as early stopping. another challenge was the choice of either training on the whole dataset or training several models per target. we tried both paths for sentence transformers and found that training on the whole dataset is better. our intuition for that is firstly training per target makes the training set much smaller for the model to learn from and hence increases the overfitting effect. the second intuition is that how people show their stance has a lot of similarities in the written tweet between the different targets and hence training on the whole dataset helps the model learn from all targets together. for future work in this direction, we could increase the dataset size by making use of other published datasets for stance detection such as @xcite . applying data augmentation to balance the dataset distributions across labels and to increase the overall dataset size is also believed to increase our models' performance. additionally, employing active learning techniques to select more informative and diverse data samples can further enhance the model's accuracy and efficiency from the new dataset @xcite . ## conclusion this research underscores the vital role of social media in public discourse and the importance of stance detection in understanding public sentiment towards critical issues such as women empowerment, covid-19 vaccination, and digital transformation. our study demonstrated that pre-trained language models (plms) when used as feature extractors, can effectively enhance stance detection without the resource-intensive process of finetuning. by leveraging sentence transformers and incorporating aggregation architectures on top of bert, we achieved competitive performance, evidenced by our strong showing in the stanceeval competition with an f1-score of 78.62. this approach not only saves significant computational resources and time but also opens new avenues for efficient and scalable nlp applications in social media analytics.