HAM i S o N - MTL at C limate A ctivism 2024: Detection of Hate Speech, Targets, and Stance using Multi-task Learning (2024)

Abstract: The automatic identification of hate speech constitutes an important task, playing a relevant role towards inclusivity. In these terms, the shared task on Climate Activism Stance and Hate Event Detection at CASE 2024 proposes the analysis of Twitter messages related to climate change activism for three subtasks. Subtasks A and C aim at detecting hate speech and establishing the stance of the tweet, respectively, while subtask B seeks to determine the target of the hate speech. In this paper, we describe our approach to the given subtasks. Our systems leverage transformer-based multi-task learning. Additionally, since the dataset contains a low number of tweets, we have studied the effect of adding external data to increase the learning of the model. With our approach we achieve the fourth position on subtask C on the final leaderboard, with minimal difference from the first position, showcasing the strength of multi-task learning.

## introduction the shared task on climate activism stance and hate event detection at case 2024 @xcite focuses on climate change discussions on twitter. as of late, climate change is experiencing an increase in political polarization @xcite , and these trends have revealed connections to a higher power controlling the public's discourse @xcite . this situation highlights the importance of an in-depth study of the issue and the many challenges it still poses, from the data collection to the added difficulty of multilingual approaches @xcite . this task, which studies the content of tweets in relation to hate speech and other essential characteristics, such as the target of the message, can serve to provide more insights regarding how these messages are transmitted and their common features. recent competitions have been held for the detection of hate speech or offensive language @xcite as well as the target of the message @xcite , focusing on issues such as multilingual twitter data or multimodal content. state-of-the-art results are obtained through the use of transformer-based approaches, that are capable of employing the entire context of the data. additional contextual knowledge, such as social information or newspaper articles, has also shown its effectiveness to improve a system's performance @xcite . similarly, stance detection has been a traditional research topic for shared tasks @xcite , where transformer-based approaches, along with data augmentation, tend to outperform other methods. in our approach to this task, we leverage the potential of multi-task learning (mtl) with a pretrained transformer model for the subtasks. mtl, as originally presented by @xcite , is able to extract information from one task to boost the performance of another, without the necessity of transferring the knowledge attained and the complications it poses with the differences in tasks or annotations. it also reduces the risk of overfitting @xcite due to the shared representation it generates for all the tasks. in our systems, we experiment with added datasets, to fully exploit the capabilities of mtl. we explore the effects of additional data for each of the tasks, with different levels of relatedness. to fully study that effect, we also fine-tune our systems without external data, other than the three subtasks. we aim to discover what works best in this situation, where we have three highly related subtasks, but there is a lack of data, especially for subtask b. the paper is structured as follows: in section 2 we briefly discuss the characteristics of the shared task, as well as the dataset provided. in section 3 we describe our approach by leveraging mtl and external data sources. in section 4 we include the results of our systems and discuss the approaches. finally, we highlight the conclusions in section 5. ## dataset & task the dataset for the shared task on climate activism stance and hate event detection, introduced in @xcite , contains a total of 10,407 tweets, only including the textual content. these instances were collected using hashtags linked to climate change and related activism and only selecting english tweets. finally, they were manually annotated for different tasks. we describe below each of the subtasks that are part of the shared task. the tweet distribution for each subtask is shown in ## methodology for our experiments, we use the same pre-trained transformer model throughout the different combinations for comparability purposes. the selection of the model is influenced by two main factors: generalization and robustness. models trained on domain-specific data or from select data sources, such as twitter, would not be ideal for our study, since we incorporate other corpus not twitter nor climate related. additionally, we want to ensure the selected model provides robustness in terms of textual classification tasks. these considerations justified our selection of the roberta @xcite pretrained model we used. the architecture of the mtl system corresponds with a hard parameter sharing approach: for each task we make use of one classification head and a roberta shared encoder for all of them. since the data sources are different in most cases, each input instance only corresponds with one classification task. the model uses size-proportional sampling, in regard to each of the datasets for the classification tasks, when selecting the next instance during training, with a fixed batch size of 32. as we previously introduced, we are using external data for the task. we briefly describe them below. these datasets are combined into the models displayed in regarding the preprocessing of the textual input, only the twitter data is altered. since it includes hashtags and user mentions that the transformer might not be able to represent, we need to consider a previous step for normalization. all the mentions and urls have been removed from the text. for the hashtags, we have followed a different approach by splitting the text into words using wordninja (keredson, 2019), since hashtags are usually a concatenation of words that might provide additional insight into the user's opinion. in the case of the multinli dataset, the premise and the hypothesis are combined into an input with a separator in-between the texts for the model. for our experiments, we explore the combinations of an initial set of parameters shown in since the dev labels were not available when we first trained our systems, we created our class balanced partition of 70-30 for the training and evaluation of the subtasks (except for subtask b, which had fewer instances, so we decided on 80-20). after they were made public, we also uploaded our systems using the dev partition for evaluation and the train set for training. we report all the results in the next section for a more in-depth analysis. additionally, for the best model retrained, in our first partition we use all the training data, while in the second we use the training and dev data combined. ## results & discussion the f1 results for the final configuration of the parameters uploaded for each subtask and partition is detailed in in subtask b there is a much higher difference between the results. the low amount of data, particularly compared to the other tasks the model was trained with, might have caused an imbalance when the model was learning for this task. adjusting the size of the datasets, or augmenting the data, may have a positive impact. it is also interesting to note that the best result is achieved when training with 80% of the training set and the most similar task. seemingly, adding highly related data has the best impact, securing the eighth position in the ranking. in subtask c we notice that most results are similar, although cop achieves the lowest in one run. we can observe again that additional data does not have a very high impact, but it achieves the highest result with the fourth position in the leaderboard and minimal difference to the best system. in terms of error analysis for the subtasks, we have noticed some tendencies. for subtask a, in over half of the runs, hate speech is correctly detected for a total of 98% of the class instances. meanwhile, all runs predict the wrong class for non hate speech in 10% of the instances for that class. even though non hate speech is the majority class, the system struggles to differentiate it. for subtask b we observe a similar effect, with over half of the runs being able to detect the individual and organization for over 90% of those instances. in this case, we notice the system errs while identifying the community, although that could be due to being the minority class. finally, over half the runs for subtask c tend to coincide for the support and oppose classes with 88% and 75% of accuracy respectively, although it decreases to 50% for neutral. our runs tend to predict support when the class is neutral, which could be due to noisy data or some level of ambiguity in the texts. in summary, it appears that external data has achieved the best result in subtasks b and c. even when the dataset was not as related to the subtask, it still appeared to add some additional knowledge. there is a high difference between the evaluation and the test results for subtask b, which could indicate some problems already mentioned for the data or a low sampling for the mtl models. regarding subtask a, since most of the results were very similar, the differences between the runs might be more related to randomness rather than the ineffectiveness of the additional data. ## conclusion hate speech is a growing cause of concern on social media, and it is still on the rise, spreading polarization to seemingly uncontroversial new topics, such as climate change. with our approach to this task, we propose to leverage other existing datasets through transformer-based mtl. our models present a robust approach to address data scarcity, especially for the target detection subtask, without the need to adapt annotations or merge unrelated data, while creating models with a higher capacity to generalize. our findings reveal that external data that is highly related to the task has an overall positive effect, while the lower the relatedness, the worse results we achieve. as a result from our experiments, our models have shown that the most promising performances are achieved when external data is used to improve one of the tasks. as future work, we plan on having a more balanced dataset for target identification, as well as experimenting with other pre-trained or already fine-tuned models for specific tasks that might provide additional context, such as sentiment analysis. additionally, we want to study the effect that each external dataset had on the models' predictions and their contributions to the results, which might provide insights into how to further improve this approach.