MGKM at S tance E val2024 Fine-Tuning Large Language Models for A rabic Stance Detection (2024)

Abstract: Social media platforms have become essential in daily life, enabling users to express their opinions and stances on various topics. Stance detection, which identifies the viewpoint expressed in text toward a target, has predominantly focused on English. MAWQIF is the pioneering Arabic dataset for target-specific stance detection, consisting of 4,121 tweets annotated with stance, sentiment, and sarcasm. The original dataset, benchmarked on four BERT-based models, achieved a best macro-F1 score of 78.89, indicating significant room for improvement. This study evaluates the effectiveness of three Large Language Models (LLMs) in detecting target-specific stances in MAWQIF. The LLMs assessed are ChatGPT-3.5-turbo, Meta-Llama-3-8B-Instruct, and Falcon-7B-Instruct. Performance was measured using both zero-shot and full fine-tuning approaches. Our findings demonstrate that fine-tuning substantially enhances the stance detection capabilities of LLMs in Arabic tweets. Notably, GPT-3.5-Turbo achieved the highest performance with a macro-F1 score of 82.93, underscoring the potential of fine-tuned LLMs for language-specific applications.

## introduction in the age of digital communication, social media platforms have become the norm in people's daily lives, not just as a medium for social interaction but also as a platform for expressing opinions and stances towards a wide array of topics, from politics and events to services and controversial issues. this amplified the need for advanced stance detection technologies, aimed at detecting whether the author of a text is in favor, against, or neutral towards a specific subject, which is a task that is becoming increasingly important for decision-making processes across various sectors such as businesses and public authorities @xcite . stance detection operates primarily on analyzing textual input to predict the author's viewpoint, which may be explicitly or implicitly conveyed. other author's social activities such as retweets and likes can be used to enhance model performance. the task is further categorized into target-specific, cross-target, and target-independent detection, each with its unique challenges and requirements, emphasizing the complexity of this field. initially, identifying viewpoints relied mostly on rules and traditional machine-learning methods. for instance, support vector machines (svm) were highly regarded in the early stages @xcite @xcite . however, the emergence of deep learning models transformed this landscape significantly. these models excelled in processing vast amounts of data and uncovering intricate patterns within it @xcite @xcite @xcite . subsequently, pre-trained language models, such as bert emerged, aiding in a richer comprehension of text through contextual analysis @xcite @xcite . nowadays, llms such as openai's chatgpt and meta ai's llama-2 are revolutionizing natural language processing (nlp) @xcite . these models, trained on extensive datasets, demonstrate a remarkable ability to mimic human language nuances with high accuracy @xcite . unlike older models, they can tackle different types of questions and grasp language nuances better, making them valuable for nlp tasks, including stance detection. refining llms such as gpt, llama3, and falcon for particular tasks through fine-tuning improves their accuracy and applicability for contextually sensitive stance detection on social media channels @xcite . by tailoring to the unique linguistic patterns and expressions prevalent in social media discourse, these models are empowered to exceed traditional methods, demonstrating remarkable proficiency in understanding sentiments and viewpoints. this research examines the improved perfor-mance of fine-tuned llms with the mawqaf dataset to analyze diverse user opinions. we demonstrate that fine-tuning greatly enhances the models' comprehension of user perspectives, providing a more profound understanding of online discussions. our study emphasizes the benefits of fine-tuning in natural language processing (nlp), particularly for stance detection, and showcases its advantages over conventional, less customized approaches. ## experimentation we fine-tuned gpt-3.5-turbo-0125 through ope-nai's web services, and the two open-source models locally. to reduce the model sizes to a manageable scale for the local training, we applied low-rank adaptation (lora) @xcite using the litgpt (ai, 2023) framework. the dataset was divided into an 85-15% train-validation split with stratification based on stance. all models were fine-tuned over 3 epochs using a single nvidia a100 pcie with 40gb ram. the hyperparameters for fine-tuning include lora rank of 32, α of 16, dropout rate of 0.05, batch size of 8, 10 warm-up steps, and a learning rate of 2 × 10 -4 . meta-llama-3-8b-instruct and falcon-7b-instruct required around 20 and 15 minutes to train respectively, whereas gpt-3.5-turbo-0125 took around 30 minutes. ## results our results, shown in meta-llama-3-8b-instruct showed moderate performance with a macro f1-score of 74.33%. it performed well in detecting 'favor' stances but struggled with 'against' stances, particularly in the digital transformation category. this suggests limitations in the model's comprehension and classification capabilities. falcon-7b-instruct had the lowest performance, with a macro f1-score of 40.73%. the model's performance varied significantly across different categories: women empowerment (42.03%), covid-19 vaccine (37.04%), and digital transformation (43.12%). the model struggled particularly with the 'against' class, achieving scores of 3.92% for women empowerment, 15.69% for covid-19 vaccine, and 0% for digital transformation. the exact reasons for falcon-7b-instruct's poor performance remain uncertain, but it seems the model struggles significantly with accurately classifying 'against' stances, which heavily impacted its overall effectiveness. ## conclusion in this paper, we conducted a detailed analysis of stance detection on the mawqif dataset, an arabic-language corpus annotated for multiple opinion dimensions across various dialects. our experiments demonstrate the effectiveness of llms in enhancing the accuracy of stance detection. we evaluated the performance of three different llms, gpt-3.5-turbo-0125, meta-llama-3-8b-instruct, and falcon-7b-instruct and observed notable dif-ferences in their ability to handle the complexities of multi-dialectal arabic in social media texts. the results indicate that fine-tuning llms significantly improves their ability to understand and detect stances in arabic tweets. notably, gpt-3.5-turbo-0125 emerged as the top performer, achieving remarkable precision in identifying both 'favor' and 'against' stances, underscoring the potential of fine-tuned llms for language-specific applications. the effectiveness of fine-tuning is further validated by the significant improvement over baseline models. additionally, the research highlights the challenges associated with fine-tuning llms, such as the substantial computational resources required and the complexities of adapting these models to specialized tasks. however, techniques like lora have proven effective in mitigating these challenges, facilitating more accessible and efficient fine-tuning processes. as we move forward, the insights gained from this study can guide future research towards enhancing model robustness, exploring more diverse datasets, and refining computational techniques to better meet the evolving needs of natural language processing applications. the integration of stance detection models into practical applications promises to improve decision-making processes, social media monitoring, and public sentiment analysis, making significant strides towards more informed and responsive digital communication platforms.