HAM i S o N -Ensemble at C limate A ctivism 2024: Ensemble of R o BERT a, Llama 2, and Multi-task for Stance Detection (2024)

Abstract: CASE @ EACL 2024 proposes a shared task on Stance and Hate Event Detection for Climate Activism discourse. For our participation in the stance detection task, we propose an ensemble of different approaches: a transformer-based model (RoBERTa), a generative Large Language Model (Llama 2), and a Multi-Task Learning model. Our main goal is twofold: to study the effect of augmenting the training data with external datasets, and to examine the contribution of several, diverse models through a voting ensemble. The results show that if we take the best configuration during training for each of the three models (RoBERTa, Llama 2 and MTL), the ensemble would have ranked first with the highest F1 on the leaderboard for the stance detection subtask.

## introduction social media is a popular tool and has adopted an essential role in this day and age. with its massive spread and usage, a global discourse arises regarding numerous topics. climate change has become a most prominent topic, as well as a very polarized one @xcite . as debates develop, the emergence of hate speech becomes a concern that must not be left unattended. although the case for freedom of speech has been voiced, it cannot be confused with a complete lack of regulations. touting that rhetoric has brought harmful effects @xcite . it delves into the paradox of tolerance, where unlimited freedom of speech can cause the corrosion of our society. rampant hate speech can create a breeding ground for intolerance and discrimination. all of these circumstances justify the necessity to study controversial public discourse, to promote online safety and inclusion. for the reasons argued above, the climate activism stance and hate event detection task at case 2024 @xcite has significant relevance. this task focuses on climate activism discourse, and it consists of three distinct subtasks: hate speech detection, target identification and stance detection. it can provide valuable knowledge on the diffusion of hate speech and the polarization of users' stances, addressing some current open challenges @xcite . as described in this paper, our proposal leverages an ensemble voting system with two different voting strategies for the stance detection subtask. ensemble voting has been used in other stance detection shared tasks @xcite , achieving the best results. these systems provide some additional advantages, such as model regularization and an increase of diversity @xcite , as they consider different approaches simultaneously. for our ensemble, we exploit three different systems: a transformer-based baseline model, a large language model and multi-task learning. beyond exploring a set of diverse systems for the proposed task, our approach has the goal of studying the effect of external data on the stance detection subtask. we aim to determine the effect that external training data has on our proposed models, and to evaluate the suitability of these external datasets towards improving a model's performance in the context of climate activism. to this end, we propose two datasets related to hate speech and stance detection that we detail below. this paper is organized as follows. in section 2 we introduce the dataset for the task. in section 3 we present the strategy for the ensemble models, as well as the additional data that were used. in section 4 we introduce our results, we discuss them in section 5, and we perform a post-competition analysis in section 6. finally, in section 7, we exhibit our conclusions and future work. ## dataset and task this shared task, climate activism stance and hate event detection, uses the climaconvo dataset in-troduced in @xcite . it consists of tweets containing hashtags from a curated list linked to climate change and climate activism, collected over a one-year period. non-english tweets were filtered out. the final dataset only reflects the textual content of the tweets and was manually annotated in six dimensions. the shared task at hand is based on a subset of climaconvo and contains 10,407 instances. below, we describe the three subtasks proposed over this dataset. ## methodology different models have been employed for the ensemble described in this paper. in this section we review the external datasets used by the models, the pre-processing step applied to all the data sources, the descriptions of each model and the characteristics of the ensemble classifier. we aim at determining whether an ensemble makes a robust model, and whether the additional context of other datasets provides an advantage to this task. ## results in total, we performed 10 experiments: 4 ensembles and 6 individual component systems. half of the runs were performed using only the case dataset and the other half using data additional to the case stance dataset. specific hyperparameters and training details are reflected in the individual papers for each system. for the case only runs, we fine-tune roberta and llama 2 models on only subtask c data. the multi-task learning (mtl) system was fined tuned on subtask a, b and c data to fully extract the knowledge from the task. regarding the runs with additional external data, the roberta systems use the climate only topic from the semeval stance dataset, while the llama 2 models make use of all the topics from that dataset. finally, the multitask learning model adds only the offensive language identification and the target tasks from the olid dataset. ## discussion as noted in figure 1 shows the confusion matrices of the best performing ensemble models, both with the majority strategy, using the case dataset and using aggregated data. a study of the errors of the different runs shows that the four sets were wrong simultaneously in 17.47% of the total number of test instances, three of them were wrong in 7.3% of the instances, two of them in 5.63% and only one of them in 8.32%. grouping by label, we observe that 11.62% of the instances labelled as support are misclassified by all models, 24.11% for those labelled as op-pose and 26.4% for neutral. grouping by ensemble strategy, we notice that for the majority one, 22.02% of instances are misclassified by the two models, while it is 24.2% for the conservative one. for the majority voting, the error for support is 11.62% of all instances labelled as support, 24.11% for oppose and 40.6% for neutral. in the case of the conservative strategy, support is 23.02%, oppose 24.11% and neutral 26.4%. based on the errors per label, it can be seen that the conservative strategy, where predictions more often skew towards neutral, afforded a worse performance. conservative ensembles were better at classifying neutral instances, but this was at the expense of the support label. since neutral instances are limited in the dataset, the use of the conservative strategy did not offer an advantage, whereas the majority more faithfully reproduced the expected distribution of labels in the dataset. however, our roberta baseline performed better than any of the ensemble strategies we submitted to the competition. in that light, we decided to conduct a post-competition examination with fewer restrictions to construct the ensemble system. ## post-competition analysis our ensemble is based on the idea of combining the diversity given by a transformer-based system (roberta), a generative model (llama 2) and a multitask learning (mtl) approach. therefore, we just selected one configuration for each of the three approaches. furthermore, we constrained ourselves to two options: whether all the systems use external datasets or none of them do. after submission, we relaxed this constraint and performed a post-competition run selecting our best roberta, llama 2 and mtl models from the training stage, regardless of whether they use external data. in this case, as shown in the difference between both ensembles is due only to the use of a different configuration of the mtl model. this shows that the diversity introduced by the best in training mtl model is valuable to the ensemble. ## conclusions and future work our contribution focused on studying the influence of external data in the context of climate activism. we have done this through three different systems, whose combination into the proposed ensemble we present in this paper. the impact of external data on this particular subtask has been limited, only being effective in the case of the mtl system, which we theorize might be due to the different classification heads for each dataset, allowing them to keep the task-specific information of each task and maintaining the encoder with the general shared knowledge. in spite of this situation, we have gained some insight regarding our ensembles. although our submitted runs do not improve the best individual result of the roberta baseline, the post competition analysis reveals that an ensemble with our best models, regardless of the training set, would have achieved the first position in the competition. as future work, a thorough study of the best combination of models, to find a higher divergence, is crucial. we have also determined that three systems might be insufficient for classification tasks with 3 classes, generating uncertainty in the test. to reduce this uncertainty, we plan on studying the effects of an ensemble with several models per approach, and of different voting strategies, such as a weighted voting schema, which could add a higher confidence level to the models and correct potential biases. a central goal of our contribution, analyzing the effect of training with external data on this dataset, remains inconclusive. the proposed additional datasets did not always improve the results. we hypothesize that an analysis of the lexical and semantic distance between task data and external data could help to determine the suitability of the chosen collections. this analysis should potentially be extended over alternative external datasets in order to make an informed choice. a similar analysis of the particular instances of climaconvo in which each of the different models of the ensemble were successful -or failed -could contribute to better determine each model's strengths and clarify an optimal ensemble strategy. as for individual models, another avenue to explore is studying the effect of other dimensions, such as pre-processing of the input data, as well as altering the threshold to assign a label to the instances. although the conservative strategy did not have a high performance, the neutral tag still proves problematic. optimizing the value for this threshold may improve the detection of this tag, thus enhancing the models. additionally, an in-depth study of the effect of external data, and how each model performs for those tasks, would be necessary to determine why it is not as effective in the case of roberta and llama-2 and how we can improve it.