SMASH at S tance E val 2024: Prompt Engineering LLM s for A rabic Stance Detection (2024)

Abstract: This paper presents our submission for the Stance Detection in Arabic Language (StanceEval) 2024 shared task conducted by Team SMASH of the University of Edinburgh. We evaluated the performance of various BERT-based and large language models (LLMs). MARBERT demonstrates superior performance among the BERT-based models, achieving F1 and macro-F1 scores of 0.570 and 0.770, respectively. In contrast, Command R model outperforms all models with the highest overall F1 score of 0.661 and macro F1 score of 0.820.

## introduction stance detection, also known as stance classification, prediction, or identification, is an nlp task that aims to determine the viewpoint of a text's author (favor, against, or neutral) towards a specific target topic @xcite . furthermore, it has gained significance both as a standalone problem and in conjunction with applications such as sentiment analysis, argument mining, sarcasm detection, rumor detection, fact-checking, and fake news detection @xcite . arabic nlp presents unique processing challenges due to its diverse dialects and complex orthographic features @xcite . furthermore, the limited resources, such as annotated corpora for arabic compared to english, intensify these challenges @xcite @xcite . hence, the task of stance detection is gaining increasing attention in underrepresented languages such as arabic. stanceeval 2024 shared task @xcite , co-organized with the arabicnlp 2024 conference, is motivated by the aforementioned factors, aiming to adapt and overcome these challenges by confronting the distinct aspects of arabic language processing. this paper describes our participation in the stanceeval shared task, where we achieved 4th place. it also provides a detailed overview of the models we used to tackle the task. we compared the performance of bert-based models and prompt engineering on large language models (llms). our official submission was based on a prompt engineering approach utilizing command-r model, which achieved an f1 score of 0.670 and macro f1 score of 0.821. for bert-based models, marbert was the best, with an f1 of 0.568 and macro f1 score of 0.761 the rest of this paper is organized as follows: section 2 provides a detailed description of the dataset used for this work. section 3 outlines the experimental setup and briefly describes the models we utilized. subsequently, the results, discussions, and findings are presented in section 4, and the paper concludes with a summary and implications of the research in section 5. ## data the dataset used in the stanceeval 2024 shared task is mawqif, a multi-label arabic dataset for targetspecific stance detection @xcite . mawqif dataset contains 4,121 sentences covering three topics, 'covid-19 vaccine', 'digital transformation', and 'women empowerment' as shown in ## conclusion this study explores various models for the stance detection in the arabic language (stanceeval) 2024 shared task, focusing on both bert-based and new large language models (llms). our experiments demonstrated that while marbert excelled among the bert-based models, new larger models such as command-r achieved the highest overall performance. these findings significantly highlight the potential of llms in capturing the nuanced semantic and contextual elements necessary for stance detection in arabic. however, it is worth noting that all of these models are multilingual, which explains why some of them achieved relatively low performance. this signifies the importance of developing arabic-specific llms, which would help improve the performance on arabic nlp tasks.