CUFE at S tance E val2024: A rabic Stance Detection with Fine-Tuned Llama-3 Model (2024)

Abstract: In NLP, stance detection identifies a writer’s position or viewpoint on a particular topic or entity from their text and social media activity, which includes preferences and relationships.Researchers have been exploring techniques and approaches to develop effective stance detection systems.Large language models’ latest advancements offer a more effective solution to the stance detection problem. This paper proposes fine-tuning the newly released 8B-parameter Llama 3 model from Meta GenAI for Arabic text stance detection.The proposed method was ranked ninth in the StanceEval 2024 Task on stance detection in Arabic language achieving a Macro average F_1 score of 0.7647.

## introduction the transformer architecture has led to the creation of vast self-supervised neural networks with tens to hundreds of billions of parameters, trained on trillions of tokens, revolutionizing natural language processing. recent models based on the transformer architecture include gpt series @xcite , llama @xcite , and others. as of late 2024, the release of llama-3 sparked the growth of a thriving open-source community, resulting in the development of near-stateof-the-art models like mistral @xcite and @xcite . with minimal finetuning, these llms can adjust to new tasks and instructions. the nlp community utilizes peft techniques for cost-effective finetuning instead of full-parameter tuning. these techniques in peft can significantly reduce a model's computational footprint, as per the study @xcite . lora (low-rank adaptation) @xcite efficiently balances fine-tuning performance and efficiency using low-rank matrix approximations. through quantization techniques, llm model weights compression reduces storage and memory usage during training and inference. using normalfloat quantization (chen et al., 2023), these methods can maintain model performance while significantly decreasing llms' resource needs, enabling resource-effective finetuning via integration with peft methods such as lora. llms are capable of stance detection and various nlp tasks through their adaptable prompting methods. their advanced language skills allow them to accurately align textual context with target subjects, mirroring the author's intended stance @xcite . traditional nlp model training methods differ significantly from prompting strategies. using these strategies, llms can make predictions with less fine-tuning, demonstrating their adaptability for various tasks. this change in approach simplifies the application of llms and broadens their usefulness, excelling at tasks involving intricate language subtleties like stance detection @xcite . in this paper, a 8-billion parameter llama-3 model was fine-tuned for arabic stance detection, the rest of the paper is organized as follows. the related work is summarized in section 2. the dataset used for training and validation was detailed in section 3. in section 4, the system is presented. section 5 summarizes the study's key findings. ## mawqif dataset the mawqif dataset @xcite was for fine-tuning the llama 3 model, this dataset differs from previously published datasets as it covers middle eastern hot topics beyond, unlike other previously published datasets that prioritize political topics like elections and referendums over a broader range. the initial considerations. mawqif explores three hot topics: covid-19 vaccine, digital transformation, and women empowerment. the twitter platform was the source of the collected dataset. three to seven annotators were responsible for annotating each tweet-target pair. annotation of a row stops when its confidence score exceeds 0.7 or when it already has seven annotations. the mawqif dataset consists of 4,121 tweets classified under three categories: covid-19 vaccine (1,373), digital transformation (1,348), and women empowerment (1,400). each tweet in this dataset is tagged for its stance, sentiment, and sarcasm. the distribution of labels differs among the three targets. digital transformation discussions on twitter usually express a positive outlook. women empowerment and digital transformation topics exhibit a higher frequency of positive sentiment compared to the covid-19 vaccine topic, which only has 25% positive tweets. in covid19 vaccine tweets, sarcasm is used more often. 34% of seemingly positive tweets are actually sarcastic, while 31% of negative tweets exhibit a non-negative sentiment. ## methodology the llama-3 model was finetuned using formatted prompt-response triplets: target the learning rate for finetuning the llama-3 model was set to 1e-4 with the adam optimizer used during training. the evaluation is conducted every 50 steps with a batch size of 1. the low-rank approximation rank is set to 64, and its scaling factor for adaptation is set to 16 in the lora configurations. the model trainable parameters are all linear layers: "q_proj", "up_proj", "o_proj", "k_proj", "down_proj", "gate_proj", and "v_proj". a 0.05 dropout is applied in the lora layer. the model's weights are quantized to 4-bit precision and mixedprecision training with float16 and float32 is enabled to reduce memory requirements and accelerate the training process. this model was trained on google colab with a single nvidia a100 gpu with 40gb of memory. the finetuning of the model and the generation of the test results took almost one hour. the code used for training and generating the results can be found on the following github repository. 1 the proposed system achieved an average marcro f 1 score of 0.7647 on the testing dataset, and it attained an f 1 score of 0.8363 for detecting stance in women empowerment arabic tweets, an f 1 score of 0.8038 for detecting stance in covid-19 vaccine arabic tweets, and an f 1 score of 0.6539 for detecting stance in digital transformation arabic tweets. ## conclusion stance detection reveals valuable insights into public opinion, trends, and behaviors across various topics. llms' text processing and analysis skills make them essential in social media navigation. through fine-tuning, these models become more precise and adaptable, essential for understanding and engaging with societal discourse in meaningful ways. in this paper, peft methods like quantization and lora were used to finetune a llama-3 model with 8 billion parameters model, this model was adapted to detect stances in arabic tweets. the proposed method is efficient and it achieved an average macro f 1 score in the stanceeval 2024 task on stance detection in arabic language.