HAM i S o N -baselines at C limate A ctivism 2024: A Study on the Use of External Data for Hate Speech and Stance Detection (2024)

Abstract: The CASE@EACL2024 Shared Task addresses Climate Activism online through three subtasks that focus on hate speech detection (Subtask A), hate speech target classification (Subtask B), and stance detection (Subtask C) respectively.Our contribution examines the effect of fine-tuning on external data for each of these subtasks. For the two subtasks that focus on hate speech, we augment the training data with the OLID dataset, whereas for the stance subtask we harness the SemEval-2016 Stance dataset. We fine-tune RoBERTa and DeBERTa models for each of the subtasks, with and without external training data.For the hate speech detection and stance detection subtasks, our RoBERTa models came up third and first on the leaderboard, respectively. While the use of external data was not relevant on those tasks, we found that it greatly improved the performance on the hate speech target categorization.

## introduction in recent years, the escalating global awareness of the imminent climate crisis has not only prompted an upsurge in climate activism but has also given rise to a new wave of advocacy strategies, often marked by actions not devoid of controversy. while the urgency of addressing climate change has fostered a sense of shared responsibility in society, some of the actions of climate activists have also sparked debates regarding the boundaries of acceptable dissent. when translated to the online sphere, where climate activists looking to disseminate their messages and mobilize supporters encounter both climate deniers and corporate pr, these conversations become ever more heated, often precluding sensible debate. our research aspires to contribute to a deeper understanding of the digital discourse surrounding climate activism and facilitate the creation of tools that can foster healthier online conversations while respecting the fundamental right to dissent in an age of environmental urgency. this paper delves into the intricate landscape of online climate activism, with a focus on the automated detection of hate speech in this context. specifically, our contribution looks at the effect of fine-tuning transformers on two external datasets selected for their relatedness to the tasks at hand, besides the data proposed by the task itself. for the subtasks focusing on hate speech detection and the categorization of its target, we augmented the training data with the olid @xcite dataset. in turn, for the stance detection subtask we employed the section related to climate change of the semeval-2016 stance dataset @xcite . the rest of this paper describes the data provided by the task (section 2) as well as the external data (section 3) we chose to augment it. next, we detail the system development process (section 4) and discuss the results (section 5. we finish with a brief conclusion (section 6. ## dataset and task the climaconvo dataset @xcite exposes a cross-section of the public discourse around climate change on social media. it comprises 15,309 tweets collected around a series of hashtags related to climate activism over a one-year period. the dataset contains annotations in six layers: relevance, stance, the presence of hate speech; if present, whether it is directed; when directed, the type of target; and the presence of humor. the shared task at hand, case@eacl2024 @xcite , comprises three subtasks based on two subsets of climaconvo, corresponding to 10,407 tweets. these subsets have been split in train, validation and test sets by the authors. ## external data we sourced the additional training data for our experiments from two datasets external to the task: the olid and the semeval-2016 task 6 datasets. ## methodology the present contribution has the goal of establishing state-of-the-art transformer baselines for the three subtasks, and then examine the influence of additional training data on each subtask. to this end, we developed systems based on the roberta @xcite and deberta @xcite transformers. both roberta and deberta improve upon bert (bidirectional encoder representations from transformers) @xcite by introducing different training objectives: roberta uses dynamic masking (where different tokens are masked every time the same sequence is fed to the model) and eliminates the next-sentence prediction training objective of bert. deberta adds a disentangled attention mechanism (where each word is represented using two vectors that encode its content and relative position) and enhanced masked decoding (where absolute word positions are added back). the version we use, deberta-v3 @xcite , replaces the masked language model pre-training task with replace token detection task (rtd), further improving the models capacity to capture long-range dependencies over roberta. on the other hand, it is key to note that roberta has been pre-trained on double the amount of data. common to both of these transformer architectures is the notion that they can be fine-tuned at a low computational cost while still exceeding at a number of diverse natural language understandking tasks. in the following subsections we provide technical details of how the proposed models were fine-tuned on the reference datasets: ## results and discussion results on subtasks a (hate speech detection) and c (stance detection) follow a similar pattern: our best results are achieved by the roberta models fine-tuned on subtask data only. as seen on the pattern of results on subtask b (hate speech target categorization) is different: here the impact of external data is notably positive in the results. the roberta fine-tuned on additional data is our best model on this subtask, whereas the models trained on subtask data only do not improve on the organizer's baselines. we attribute this difference to the size of the subset of tweets designated for this task. the much larger size of the chosen additional dataset (4,089 vs. 999 tweets) is more attuned to what transformer models such as roberta and deberta expect. finally, we consider that roberta models perform better than more advance deberta models on this task due to contextual knowledge being more important than the ability to capture longrange dependencies when dealing with tweet data, whose instances are short in nature. ## conclusions and future work this paper introduced carefully adjusted transformer baselines for the hate speech detection, hate speech target categorization, and stance detection in tweets subtasks proposed at case@eacl2024. based on off-the-shelf models, we have conducted a study of the effects of related external train data, with mixed results. we consider that further analysis of the results is needed before discarding the use of external data for this task. in particular, we would like to study the lexical and semantic distance between the climaconvo dataset proposed by the task and the ones chosen as additional train data, aiming to extend this analysis to other potential external datasets. this research contributes to the ongoing efforts to foster healthy online conversations surrounding climate change activism. as the field of natural language processing continues to advance, our systems serve as a foundation for future developments in hate speech and stance detection in the context of critical issues like climate change.