Bryndza at C limate A ctivism 2024: Stance, Target and Hate Event Detection via Retrieval-Augmented GPT -4 and LL a MA (2024)

Abstract: This study details our approach for the CASE 2024 Shared Task on Climate Activism Stance and Hate Event Detection, focusing on Hate Speech Detection, Hate Speech Target Identification, and Stance Detection as classification challenges. We explored the capability of Large Language Models (LLMs), particularly GPT-4, in zero- or few-shot settings enhanced by retrieval augmentation and re-ranking for Tweet classification. Our goal was to determine if LLMs could match or surpass traditional methods in this context. We conducted an ablation study with LLaMA for comparison, and our results indicate that our models significantly outperformed the baselines, securing second place in the Target Detection task. The code for our submission is available at https://github.com/NaiveNeuron/bryndza-case-2024

## introduction the climate activism stance and hate event detection @xcite aims to extend the growing body of work on stance, target and hate event detection @xcite by exploring these tasks in the context of climate activism. it does so by utilizing a novel climaconvo dataset @xcite , which is one of the first multi-aspect datasets of its kind. while traditional approaches to stance, target, and hate event detection rely on finetuned classifiers, our study takes a different route. we explore how a data scientist or analyst, with only api access to a large language model (llm) and without the option to finetune or alter the model, can still develop effective solutions. by creatively adjusting the prompts given to the llm and using external tools like vector databases and pretrained ranking models for enhancement, we've found this simple method to be surprisingly competitive. despite its simplicity, it secured the second-highest performance in the target detection subtask. ## related work for the past couple of years, the progress of natural language processing has been driven largely by existence and availability of datasets and data resources. in the context of climate, some notable examples include climatebert: a pretrained language model for climate-related text @xcite , a dataset for detecting real-world environmental claims @xcite as well as the newly introduced climaconvo dataset @xcite , which forms the basis of the shared task on stance and hate event detection in tweets related to climate activism. all of the subtasks of this shared task can be modeled as classification problems and as such there exists an extensive body of academic work on this topic. in particular, methods like svm @xcite , lstm @xcite as well as custom architectures such as deephate @xcite have been proposed and evaluated. inspired by outstanding generalizational ability of large language models -including chatgpt foot_0 , gpt-4 @xcite , llama @xcite and othersand their performance in classification tasks, especially in zero-and few-shot settings, we investigate their adaptability and effectiveness for the tasks of stance, target and hate event detection. although works whose aim would be similar do exist, such as for instance @xcite and @xcite , a shared task provides a unique opportunity for a thorough evaluation on many dimensions, which is lacking in the literature and uniquely distinguishes our work. ## dataset to execute the described experiments we used the dataset introduced in @xcite and described in as we can observe in a cursory glance at the dataset has also revealed that a relatively significant proportion of its tweets (489 in total) contains the sentence "you've been fooled by greta thunberg". while an interesting tidbit, it is almost certainly an artefact of the data collection process and provides insight into the peculiarities of the task and the data it uses for evaluation -particularly since in an overwhelming number of cases the tweets that contain this substring are labelled as hate, individual and oppose for subtasks a, b and c, respectively. ## system description as outlined above, the primary component of our system is a large language model, namely gpt-4, which was chosen for its strong zero-shot and few-shot capability. the model was accessed via the azure openai service and was not changed and/or finetuned as part of our experimentationthe only attribute of the system that changed from one configuration to the other is the prompt that is sent to the gpt-4 api. in our experiments we utilized the 2023-07-01-preview version foot_1 and unless otherwise noted, the temperature has been set to 0 in order to make the experiments reproducible. we also utilize paralellism in order to decrease the time necessary for the whole pipeline to run. in the end, the evaluation of our models on subtask a and subtask c takes roughly 25 minutes, whereas it is possible to evaluate subtask b within 2 minutes and 30 seconds. ## results & discussion the results of our experiments can be found in model subtask a subtask b subtask c acc p r f1 rnk acc p r f1 rnk acc p r f1 rnk baseline .901 --.708 -.716 --.554 .651 --.545 -gpt-4 .935 .835 .880 .856 -.900 .545 .656 .553 -.693 .515 .513 .509 -gpt-4 few-shot (@xmath0.932 .826 .895 .855 -.927 .809 .723 .747 -.693 .502 .507 .487 -gpt-4 few-shot (@xmath1.916 .794 .886 .855 -.927 .809 .723 .747 -.702 .511 .512 .495 -gpt-4 rag (@xmath2.944 .859 .890 .874 -.887 .641 .672 .654 -.707 .517 .514 .498 -gpt-4 rag (@xmath3.941 .851 .889 .868 -.927 .781 .776 .776 2/18 .690 .668 .681 .666 -gpt-4 rag (@xmath4.942 .855 .887 .870 -.927 .733 .764 .769 -.688 .666 .678 .661 -gpt-4 rag all (@xmath5.948 .866 .899 .881 7/22 .920 .776 .762 .767 -.714 .692 .709 .692 -gpt-4 rag all (@xmath6.944 .864 .884 .874 -.920 .715 .721 .716 -.711 .687 .712 .693 12/19 gpt-4 flashrank (@xmath7.941 .853 .877 .864 -.940 .635 .617 .625 -.709 .689 .707 .693 -gpt-4 flashrank (@xmath8.941 .851 .886 .868 -.913 .733 .706 .713 -.702 .683 .703 .688 - in figure 1 we can see the confusion matrices for the best performing models (highlighted in green in ## ablation study with llama to assess to what extent would a similar approach work with a model other than gpt-4 and to provide further insight into how much of the final performance is attributable to the base model versus the other additions (e.g. rag and/or re-ranking) we conduct an ablation study in which we replace gpt-4 with llama 2 70b @xcite . we use subtask b, in which we obtained the best results with gpt-4, as the benchmark task and due to limitations of the llama's context window we further limit ourselves to @xmath10. other than that the evaluated models are identical to those described in section 4. the results can be seen in ## conclusion in this work we evaluate gpt-4 extended with retrieval augmentation and re-ranking on the task of stance, target and hate event detection. we investigate to what extent is it possible utilize gpt-4 as a black box and obtain solid performance by only changing its prompt and also compare it to the llama model. our results show that augmenting gpt-4 with retrieval mechanism can lead to it serving as a classification model with competitive performance, as evidenced by the fact that it ranked 2nd in subtask b: target detection.