Z - AGI Labs at C limate A ctivism 2024: Stance and Hate Event Detection on Social Media (2024)

Abstract: In the digital realm, rich data serves as a crucial source of insights into the complexities of social, political, and economic landscapes. Addressing the growing need for high-quality information on events and the imperative to combat hate speech, this research led to the establishment of the Shared Task on Climate Activism Stance and Hate Event Detection at CASE 2024. Focused on climate activists contending with hate speech on social media, our study contributes to hate speech identification from tweets. Analyzing three sub-tasks - Hate Speech Detection (Sub-task A), Targets of Hate Speech Identification (Sub-task B), and Stance Detection (Sub-task C) - Team Z-AGI Labs evaluated various models, including LSTM, Xgboost, and LGBM based on Tf-Idf. Results unveiled intriguing variations, with Catboost excelling in Subtask-B (F1: 0.5604) and Subtask-C (F1: 0.7081), while LGBM emerged as the top-performing model for Subtask-A (F1: 0.8684). This research provides valuable insights into the suitability of classical machine learning models for climate hate speech and stance detection, aiding informed model selection for robust mechanisms.

## introduction in the ever-evolving landscape of our digital era, an expansive tapestry of data unfolds, revealing profound insights into the intricate dynamics of social, political, and economic systems. the narratives of citizen responses to covid policies @xcite @xcite and the unfolding russia-ukraine conflict stand out as crucial chapters @xcite , vividly demonstrating the indispensable role of event-centric data in unraveling the multifaceted tapestry of real-world scenarios. these narratives underscore the pressing need for sophisticated tools capable of discerning and addressing hate speech, ultimately leading to the inception of the shared task on climate activism stance and hate event detection at case 2024 @xcite . within the realms of social media platforms, where climate activists converge to share insights, mobilize support, and voice concerns, instances of hate speech can emerge, casting a shadow over the collaborative spirit of the movement. sub-task a of our shared task @xcite , hate speech detection, emerges from the very fabric of these real-world scenarios, challenging participants to meticulously scrutinize textual content for the presence of hate speech. navigating the landscape of hate speech requires a profound understanding of its targets. real-world examples abound, illustrating instances where individual activists, environmental organizations, and entire communities face the brunt of hateful rhetoric. sub-task b, targets of hate speech identification, mirrors these authentic situations by urging participants to categorize hate speech targets into "individuals," "organisations," or "communities." in witnessing the unfolding narratives of climate activism, the importance of understanding stance dynamics becomes evident. real-world scenarios often involve a spectrum of sentiments -from unwavering support to vehement opposition or maintaining a neutral stance. sub-task c, stance detection, captures the essence of these dynamic narratives, prompting participants to decipher the sentiments expressed in textual content. by doing so, participants contribute to a deeper understanding of how the collective sentiment shapes the discourse surrounding climate change events. the shared task thus emerges not as a detached academic exercise @xcite but as a direct response to the challenges faced in the trenches of climate activism. through real-world instances and tangible connections, participants are invited to be catalysts for positive change, developing tools that align with the authentic dynamics of the digital discourse in climate change activism. in this endeavor, the shared task serves as a bridge between the virtual and the real, fostering a more resilient and empathetic space for those advocating for a sustainable and equitable future. in this paper, we describe our approach to tackle the challenges. from here, the report continues in the following manner: in section 2, we give an overview of the dataset for each subtask and describe the challenge at hand. in section 3, we present our approach in detail, covering the intricacies of our experimental set-up, cross-validation strategy, models used, and intuition behind them. in section 4, we brief the results from the experiments section. then, we conclude in section 5 with the final takeaways, our standings, and the scope of future work. ## dataset description this section provides an overview of the dataset designed to facilitate the exploration and evaluation of these key aspects. ## results all the subtasks were evaluated using f1 score, precision, recall, and accuracy. it is evident from the results matrix 1 that the lstm based model poses a strong competition in performance for all the subtasks nearing the best score for all the sub- in subtask-a, the lgbm model on top of tf-idf performed the best for us with a f1-score of 0.8684 while models like naive bayes, logistic regression, random forest and catboost on top of tf-idf were not too far away. in subtask-b, the catboost model on top of tf-idf performed the best with a score of 0.5604 while models like naive bayes, logistic regression and random forest were close with scores of 0.5482, 0.5577 and 0.5496 respectively. in subtask-c, the catboost model on top of tf-idf performed the best with a score of 0.7081, while models like logistic regression and xgboost on top of tf-idf score 0.7075 and 0.6994 respectively and came very close. we also performed fine-tuning using transformers but the outcomes were inadequate, so we decided to use simpler models in order to achieve better performance. we were eventually able to surpass the baseline f1 scores for subtask-a: 0.708(bert(kenton and toutanova, 2019)), subtask-b: 0.554(bert), and subtask-c: 0.5495(climate-bert @xcite ) that were provided by the organizer. note that, all the scores mentioned are the performance on the hidden test set and directly taken from the system-run report provided on the competition website after finalized leaderboard 2, 3. ## conclusion in summary, our research contributes crucial insights into hate speech and stance detection within climate activism. employing classical machine learning models, such as lstm, xgboost, lgbm, and catboost, revealed nuanced variations in performance. notably, catboost emerged as a strong performer, showcasing f1 scores of 0.5604 and 0.7081 for subtask-b and subtask-c. lgbm excelled in subtask-a with an impressive f1 score of 0.8684. this study guides model selection for robust hate speech detection. as we conclude, our findings serve as a valuable resource for advancing tools aligned with the authentic dynamics of digital discourse in climate change activism.