NLPD ame at C limate A ctivism 2024: Mistral Sequence Classification with PEFT for Hate Speech, Targets and Stance Event Detection (2024)

Abstract: The paper presents the approach developed for the “Climate Activism Stance and Hate Event Detection” Shared Task at CASE 2024, comprising three sub-tasks. The Shared Task aimed to create a system capable of detecting hate speech, identifying the targets of hate speech, and determining the stance regarding climate change activism events in English tweets. The approach involved data cleaning and pre-processing, addressing data imbalance, and fine-tuning the “mistralai/Mistral-7B-v0.1” LLM for sequence classification using PEFT (Parameter-Efficient Fine-Tuning). The LLM was fine-tuned using two PEFT methods, namely LoRA and prompt tuning, for each sub-task, resulting in the development of six Mistral-7B fine-tuned models in total. Although both methods surpassed the baseline model scores of the task organizers, the prompt tuning method yielded the highest results. Specifically, the prompt tuning method achieved a Macro-F1 score of 0.8649, 0.6106 and 0.6930 in the test data of sub-tasks A, B and C, respectively.

## introduction climate change is an ever-growing concern that has garnered significant attention worldwide. as the severity of its impacts becomes increasingly undeniable, it has also become an issue that has sparked diverse reactions and discussions on social media platforms. within these discussions, the prevalence of hate speech, the identification of its targets, and the detection of various stances towards climate change and activist movements have become vital areas of interest. understanding the dynamics of hate speech, targets of hate speech, and different stances within climate change discourse is crucial for fostering informed discussions, addressing concerns, and promoting positive change. hate speech, defined as harmful or offensive language directed towards individuals or groups, has the potential to exacerbate division, hinder productive conversations, and impede constructive collaboration. identifying hate speech in climate change discourse provides a deeper understanding of the negative impact it can have on the overall conversation. additionally, recognizing the targets of hate speech helps shed light on the specific groups or entities facing hostility, enabling targeted interventions and support. examining the different stances towards climate change and activist movements also unveils the diversity of perspectives within these discussions. stance detection allows for the identification of supporters, skeptics, and deniers, providing a nuanced understanding of the range of viewpoints on this pressing issue. by capturing shifts in opinions, trends can be identified, informing future discussions and policy-making. natural language processing (nlp) models have proven to be valuable assets in detecting hate speech, determining its targets, and classifying stances within various domains. however, when it comes to climate change discourse, there is a need for well-annotated datasets that specifically address the unique challenges present in this field. the scarcity of such datasets poses a significant obstacle to harnessing nlp models effectively. to address this gap, thapa et al. (2024) created the climate activism stance and hate event detection shared task at case 2024 which challenged participants to develop binary and multi-class text classification systems that are able to detect hate speech, targets of hate speech as well as stance detection concerning climate change, events and movements. the shared task leveraged several aspects of the annotated english twitter dataset regarding climate discourse made by shiwakoti et al. (2024). this paper presents the system developed for this task, with the code available on the provided github link. @xcite the structure of this paper is as follows: firstly, section 2 presents a discussion of the previous related work followed by the presentation of the task and data analysis in section 3, and an overview of the developed methodology in section 4. section 5 presents the results and error analysis. finally, the paper concludes with section 6, which discusses future work, as well as the limitations during participating in the task. ## related work as social media usage continues to grow and usergenerated content becomes more prevalent, numerous studies have focused on identifying and categorizing insulting messages that target individuals or groups across different platforms. to accomplish this, researchers have utilized nlp in conjunction with machine learning. while initial studies focused solely on the english language, the need to address this issue in a multi-lingual context has emerged in recent years. many studies and shared tasks have been conducted, utilizing various terms such as abuse, aggression, cyberbullying, hate speech, and toxic or offensive language to classify these messages. semeval's 6 th shared task, offenseval: identifying and categorizing offensive language in social media, introduced the detection of offensive language on social media. the task consisted of three sub-tasks that aimed to implement binary or multi-class text classification. sub-task a sought to differentiate between offensive and non-offensive english tweets, while sub-task b aimed to identify the type of offensive tweets and whether they were targeted or not. subtask c aimed to identify the target of the offensive posts. participants were provided with a dataset containing 13,240 english tweets and a test set of 860 tweets, called the offensive language identification dataset (olid), which were annotated according to the three sub-tasks (zampieri et al., 2019). this task was extended the following year as the 12 th task of semeval 2020 named as multilingual offensive language identification in social media to encourage offensive language detection in other languages, such as arabic, danish, greek and turkish, based on the sub-tasks of the previous semeval (zampieri et al., 2020). moreover, semeval's 5 th task in 2019 addressed the issue of hate speech directed towards immigrants and women on twitter, in both english and spanish. ## results & discussion ## conclusion the climate activism stance and hate event detection shared task at case 2024 involved finetuning the llm mistral-7b with two peft methods (lora and prompt tuning) for binary and multi-class text classification. this resulted in the creation of six models that can detect hate speech, targets of hate speech, and stance regarding climate change and activist events. the approach also included adding weights to deal with class imbalance, as well as data cleaning and pre-processing. comparing the two peft methods showed that the prompt tuning method yielded the best performance by crafting the most appropriate and precise prompt for each task. both methods, particularly the prompt tuning method that was submitted, outperformed all transformer language models that were fine-tuned by the task organizers and whose scores were presented as baselines. to further improve the models' performance, future efforts should concentrate on adding more tweets in the sub-tasks, especially hate speech and targets of hate speech. although the mistral model was originally designed for text generation, it demonstrated its potential to perform sequence classification effectively as well.