M ason P erplexity at C limate A ctivism 2024: Integrating Advanced Ensemble Techniques and Data Augmentation for Climate Activism Stance and Hate Event Identification (2024)

Abstract: The task of identifying public opinions on social media, particularly regarding climate activism and the detection of hate events, has emerged as a critical area of research in our rapidly changing world. With a growing number of people voicing either to support or oppose to climate-related issues - understanding these diverse viewpoints has become increasingly vital. Our team, MasonPerplexity, participates in a significant research initiative focused on this subject. We extensively test various models and methods, discovering that our most effective results are achieved through ensemble modeling, enhanced by data augmentation techniques like back-translation. In the specific components of this research task, our team achieved notable positions, ranking 5th, 1st, and 6th in the respective sub-tasks, thereby illustrating the effectiveness of our approach in this important field of study.

## introduction in the ever-evolving landscape of climate change activism, encouraging meaningful conversations and comprehending how things change throughout events depends critically on the ability to recognize hate speech and the understanding of attitude during these events. this paper presents our effort in the shared task on hate speech and stance detection during climate activism @xcite , where our goal is to develop effective models for hate speech detection, target identification, and stance detection. this task consists of three subtasks that work together to support an integrated approach to event identification. the goal of the first subtask is to identify whether the given text contains hate speech or not. the second subtask focuses on identifying if people, groups, or communities are targets of hate speech. lastly, stance detection provides insight into the dynamics of climate activism protests * denotes equal contribution by assessing the support, opposition, or neutrality indicated within texts. our paper serves as a comprehensive system description, outlining the approaches and models used to address these subtasks within the framework of activist events related to climate change. we present our ensemble method for identifying hate speech, which combines robust models like xlm-roberta-large @xcite , bertweet-large (ushio and camacho-collados, 2021a), and fbert @xcite . notably, for target detection, the best-performing model is bertweet-large (ushio and camacho-collados, 2021b) while bertweet-base (nguyen et al., 2020) excels in stance detection. we also discuss our fine-tuning strategies and dataset augmentation techniques, demonstrating our commitment to refining model performance. our approach's effectiveness is demonstrated by our remarkable f1 scores of 0.8885, 0.7858, and 0.7373. furthermore, our team named masonperplexity has secured 5 th , 1 st , and 6 th ranks in the respective subtasks, underscoring the competence of our models in comparison to peers. through this paper, we aim to contribute to the advancement of hate speech and stance detection in the context of climate activism, fostering a safer and more informed space for dialogue and understanding during crucial events. we employ ensemble methods to better classify the texts -our approach increases the accuracy metrics for the first sub-task where we encounter a comparatively larger amount of data. we also use data augmentation methods, which further improve our results. ## related works the paper @xcite explores the rising concern of hate speech on the internet and its potential impact. it emphasizes machine learning and deep learning models in automatically identifying hate speech. in @xcite , english tweets are subjected to supervised classification using n-gram features and a linear svm classifier. even at 78% accuracy, it is still difficult to discern offensive language from hate speech. by combining recurrent neural networks and user features, @xcite outperforms current systems and achieves a remarkable f-score of 0.9320 on a twitter dataset. additionally, warner and hirschberg @xcite define hate speech and despite limitations in capturing larger language patterns, svm classification can detect anti-semitic speech with 94% accuracy. the literature on target detection in hate speech unveils valuable insights through various studies in the field. @xcite focuses on dutch facebook comments, exploring hateful metaphors to enhance hate speech type and target detection. the study incorporates manual metaphor annotations as features for svm and bert models, observing improvements in f1 scores. conversely, @xcite proposes a hierarchical annotation scheme for offensive language in english tweets, creating the olid dataset. the study employs svm, cnn, and bilstm models, achieving notable results and providing a valuable resource for offensive language research. stance detection, a crucial aspect of nlp, involves determining a person's position towards a concept. @xcite outlines the significance and challenges in this domain, emphasizing its relation to sentiment analysis, emotion detection, and other tasks. it highlights the evolution facilitated by shared tasks, varied approaches, including traditional svms and newer lstm models, and the necessity of annotated datasets. additionally, @xcite introduces a multitasking approach, enhancing performance on multiple datasets, and showcasing the potential of incorporating auxiliary tasks. furthermore, @xcite contributes a valuable stance-annotated turkish twitter dataset, showcasing the diversity of research efforts in stance detection. ## datasets from the tables for subtask 1, subtask 2, and subtask 3, it is evident that the dataset @xcite is imbalanced across different labels. ## experiments in subtask a, we initially employ gpt3.5 (openai, 2023) zero shot and few shot prompting with test f1 score 0.66 and 0.73. the prompt provided to gpt3.5 is available in figure 1 . role: you are a helpful ai assistant. you are given the task @xmath0. definition: <subtask_definition>. you will be given a text to label @xmath1. task: generate the label for this text in the following format: <@xmath2. thanks. then we use bertweet large (ushio and camacho-collados, 2021a) @xcite , xlm-r @xcite , hate-bert @xcite and fbert @xcite . following this, we adopt a weighted ensemble approach (ensemble 1) for the best three models (bertweet, xlm-r, fbert). similarly, we perform another weighted ensemble approach (ensemble 2) with the same models only replacing bertweet with hate-bert, as these two models show the same f1 score on test data with the same setting. however, the former ensemble strategy yields the highest f1 score for this task. to address class imbalance in subtask a, we implement back translation by converting the training data of those specific labels that have a smaller ratio with respect to the whole training set through various languages, including xosha to twi to english, lao to pashto to yoruba to english, yoruba to somali to kinyarwanda to english, and zulu to oromo to shona to tsonga to english. this approach significantly contributed to improving the overall f1 score of ensemble 1 from 0.85 to 0.88 and ensemble 2 from 0.86 to 0.89. we follow the approach of back translation of @xcite . for this, we select languages that demonstrate limited or no cultural overlap with the original language featured in the dataset. xosha, twi, lao, pashto, yoruba, somali, kinyarwanda, zulu, oromo, shona, and tsonga are languages that are very diverse culturally and geographically. this diversity underscores the significance of con-sidering a wide range of cultural and geographical influences when working with these languages. by intentionally selecting these languages without cultural overlap, we introduce a purposeful aspect of diversity, mitigating potential biases, and enhancing the dataset with a broader spectrum of linguistic expressions. moreover, the ensemble method with majority voting is also proven helpful in this type of case where a single model may not label the data correctly due to class imbalance @xcite . for instance, when two out of three models predict a sentence as a hate event, the sentence is subsequently labeled as a hate event through the application of majority voting. in subtask b, we utilize bertweet-large (ushio and camacho-collados, 2021a), bert base @xcite , and xlm-r @xcite . additionally, like subtask 1, we implement back translation using the same language sequences mentioned earlier to address class imbalance. notably, bertweet large (ushio and camacho-collados, 2021a) demonstrates the highest f1-score among these models. we also use gpt3.5 zero shot and few shot prompting with 0.63 and 0.64 test f1 scores. bertweet-large (ushio and camacho-collados, 2021a), bert base @xcite , and bertweet base (nguyen et al., 2020) models are applied in subtask c for stance detection. among these models, the bertweet base achieves the highest f1 score. f1 score for gpt3.5 zero shots and few shot prompting are 0.63 and 0.67. hyperparameters of all the models used excluding gpt3.5 in the experiments are available in figure 4 . ## results the results in tables 5 , 6 , and 7 provide a comprehensive evaluation of various nlp models across the three subtasks of the shared task. in subtask a, our ensemble approach (ensemble 2 with hate-bert, xlm-r and fbert models) secures the fifth rank. for subtask b, bertweet large secures the top position (rank 1), while in subtask c, we achieve the sixth rank utilizing bert-base. ## error analysis upon evaluating our models' performance across the three subtasks, we identify several key sources of errors that contributed to limiting our scores. in subtask a on hate speech detection, our ensemble model struggles with longer text segments that express hate in subtle or nuanced ways. the models are not always able to pick up on the underlying mocking or criticism woven into complex rhetorical devices. additionally, sarcasm and irony continue to pose challenges, as models interpret literally what is meant to convey the opposite meaning. for subtask b on target identification, errors frequently occur in distinguishing between organizations and communities as categories. our models have difficulty consistently applying the definitions and criteria that delineate these two groups as targets of hate speech. there are also inconsistencies in labeling individual people who are associated with or represent a broader community. regarding subtask c on stance detection, our models struggle to some extent with longer text segments, having more trouble identifying stances from among nuanced discussions. shorter, more direct statements of opposition or support were simpler for the models to categorize accurately. to visualize label-wise models' performance we can see the figures 2 , 3 , and 4 of confusion matrices for all the subtasks. figure 3: confusion matrix for target detection 135 figure 4: confusion matrix for stance detection ## conclusion in conclusion, the masonperplexity team has made significant strides in the domain of detecting climate activism stances and hate events on social media. through a comprehensive evaluation of various models, our research underscores the efficacy of ensemble modeling coupled with data augmentation techniques like back-translation. our achievements in the shared task, marked by rankings of 5 th , 1 st , and 6 th in the respective subtasks, reflect the potential of our methodologies in addressing the complexities of sentiment analysis in the context of climate activism. there are several avenues for future research. firstly, addressing the challenge of label imbalance in our dataset could enhance the accuracy and reliability of our models. exploring advanced techniques in data sampling or synthetic data generation may provide viable solutions. secondly, the refinement of label quality through more rigorous annotation processes or leveraging semi-supervised learning techniques could further improve model performance. finally, the integration of large language model (llm) fine-tuning presents a promising direction. fine-tuning pre-trained models specifically for the nuances of climate activism discourse and hate speech detection could yield more nuanced and contextually aware results. additionally, expanding our research to include multilingual datasets would enhance the applicability and relevance of our findings in a global context, fostering a more comprehensive understanding of public sentiment on climate issues worldwide.