Overview of the Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion (2023)

Abstract: Hope serves as a powerful driving force that encourages individuals to persevere in the face of the unpredictable nature of human existence. It instills motivation within us to remain steadfast in our pursuit of important goals, regardless of the uncertainties that lie ahead. In today’s digital age, platforms such as Facebook, Twitter, Instagram, and YouTube have emerged as prominent social media outlets where people freely express their views and opinions. These platforms have also become crucial for marginalized individuals seeking online assistance and support[1][2][3]. The outbreak of the pandemic has exacerbated people’s fears around the world, as they grapple with the possibility of losing loved ones and the lack of access to essential services such as schools, hospitals, and mental health facilities.

## introduction hope serves as a powerful driving force that encourages individuals to persevere in the face of the unpredictable nature of human existence. it instills motivation within us to remain steadfast in our pursuit of important goals, regardless of the uncertainties that lie ahead @xcite . in today's digital age, platforms such as facebook, twitter, instagram, and youtube have emerged as prominent social media outlets where people freely express their views and opinions. these platforms have also become crucial for marginalized individuals seeking online assistance and support @xcite @xcite 1 https://codalab.lisn.upsaclay.fr/competitions/11076 @xcite . the outbreak of the pandemic has exacerbated people's fears around the world, as they grapple with the possibility of losing loved ones and the lack of access to essential services such as schools, hospitals, and mental health facilities. as a result, people have turned to online forums as a means to fulfill their informational, emotional, and social needs. through social networking sites, individuals can connect with others, experience a sense of social inclusion, and cultivate a feeling of belonging by actively participating in online communities @xcite . the presence of these factors has a profound impact on both physical and psychological well-being, as well as mental health @xcite . by leveraging the power of hope and utilizing online platforms, individuals are able to find solace, support, and resources during challenging times @xcite . these digital spaces offer an avenue for people to seek guidance, share their experiences, and foster connections with others who may be going through similar struggles. through virtual networks, individuals can combat feelings of isolation, gain access to valuable information, and receive emotional support, all of which contribute to their overall well-being and mental resilience @xcite . it is important to acknowledge the pivotal role that hopes and online platforms play in providing a lifeline to individuals in need. as we navigate through unpredictable circumstances, these digital resources serve as beacons of light, reminding us that we are not alone in our struggles and that there is a collective effort to support one another @xcite . through the convergence of hope, technology, and human connection, we can weather the storm and find strength in our shared experiences, ultimately emerging stronger and more resilient as a global community. ## related work despite the optimistic nature of hope's speech, it has garnered relatively little attention within the research community. this lack of research interest could be attributed to the absence of available labeled datasets. in recent years, there has been a notable increase in attention towards this issue, thanks in part to the efforts of the organizers of the lt-edi-eacl2021 workshop who shared a labeled dataset. several frameworks were submitted during this workshop to address the challenge of detecting hope speech @xcite . while numerous research endeavors have focused on filtering out hateful and offensive comments from social media posts, the identification of hopeful comments has received comparatively less consideration (chakravarthi, 2020). counter-narratives, denoting informed textual responses, have surfaced as a notable strategy, drawing recent attention from experimenters @xcite . this approach to counter-narratives aims to balance the preservation of freedom of speech while preventing excessive content blocking. @xcite took the initiative to construct and release a counterspeech dataset using youtube comments. however, the central concept of directly intervening with textual responses can inadvertently escalate hostility. although it is beneficial for content creators to comprehend why their comments or posts were removed or blocked, and subsequently adjust their discourse and attitudes favorably, this intervention can sometimes exacerbate tensions. this has directed our research focus toward the exploration of positive content, such as messages of hope, and promoting such constructive activities. ## task description the hope speech in our context refers to comments or posts on youtube that provide support, comfort, recommendations, motivation, and understanding. while a comment or post in the dataset could consist of multiple sentences, the average sentence length across the corpus is one. annotations in the dataset are done at the level of individual comments or posts. participants were provided with datasets via the codalab website foot_0 for the specific languages bulgarian, english, hindi, and spanish for development, training, and testing purposes. ## dataset the shared task's dataset comprises comments in four distinct languages: english, spanish, bulgarian, and hindi, totaling 27,545, 5,859, 3,203, and 2,159 comments, respectively. these comments are sourced from social media platforms like youtube and twitter. for the english language subset, we utilized the hopeedi dataset introduced @xcite . this dataset focuses on socially significant subjects, including equality, diversity, and inclusion, addressing topics like lgb-tiq issues, covid-19, women in stem, dravidian languages, black lives matter, and more. the inter-annotator agreement was assessed using krippendorf's alpha. these dataset details are shown in ## methodology in this shared task, there are eight teams actively participated and implemented their models. they evaluated their model's performance on our hope speech detection shared task: hate-alert: @xcite the participants employed two types of transformer-based models, namely mbert and xlmr-base, in their study. in the first run, they conducted fine-tuning on the mbert model. for the second and third runs, they took the cls embeddings from both the mbert and xlmr models, subjected them to two dense layers, and ultimately utilized a classification head. notably, the utilization of cutting-edge transformerbased models for the classification task is a noteworthy aspect of their approach. the results have demonstrated the superiority of these transformerbased models over earlier deep-learning models like lsrm and cnn-gru. mucs: @xcite the mucs team's journey began with enhancing models for a pivotal project. armed with determination, they fused bert embeddings with syllable tf-idf, promising innovative insights. venturing further, they intertwined bert embeddings with tf-idf and resampling. across three runs, they refined their approach, each iteration marking a step towards precision. this dedication culminated in a triumphant unveiling, a testament to ingenuity, poised to reshape their field. team-tamil: @xcite in our pursuit of enhancing model performance, we embarked on a journey fueled by innovation. with the creative fusion of mpnet embeddings, we engineered a powerful and distinct form of representation. as the threads of our endeavor wove together, an h2o model emerged, infused with the essence of our collective efforts. this union of cutting-edge techniques resulted in a transformative leap forward, poised to unravel new insights and pave the way for future advancements. iic team: @xcite the method employed in this task involved utilizing bidirectional lstm (long short-term memory) and bilstm with embeddings. additionally, xlm-roberta models were employed for each language. since the datasets used in this task exhibited significant class imbalances, various techniques were ap-plied to address this issue. to balance the datasets, a combination of undersampling and other augmentation methods was implemented. ml ai iiitranchi: @xcite this team implemented text classification, the bag-ofwords (bow) model is used, and there are multiple steps in the procedure. the labeled dataset was first prepared by being divided into a training set and a testing set. the text data should be next preprocessed by reducing noise, standardizing the text format, and deleting stopwords. then, they used a vectorization method, such as countvectorizer, to construct the bow representation. this method turns each document into a numerical vector based on word frequencies. to train the classifier, divide the bow representation into features and labels. then select a classification algorithm-random forest and adaboost, for example-and train the model using the training data. lastly, make predictions on the testing set to assess the model. tercet: @xcite the method that they have employed for this task is support vector machines (svm). given a higher precision, f1 score, and weighted averages as compared to models such as random forest, logistic regression, and naïve bayes models, svm was a good fit for classifying the given test datasets. the process starts with preprocessing of the text data such as removing punctuation, emoticons, and stop words. to convert the text data into a form that is usable by the model, they used a tf-idf vectorizer algorithm and utilized the data in the svm model. ranganayaki: @xcite the data set is preprocessed to translate emojis, convert text to lowercase, username removal, and extra space removal. the vocabulary of english and hindi data is formed to correct spelling mistakes in training and testing data using levenshtein distance. preprocessed data is converted into fasttext embedding of dimension 100x100. the dataset is oversampled using adasyn oversampling to handle class imbalance. the data is then fed into a capsule network, to form the model, which is used to make predictions. vtu bgm: (sanjana m. kavatagi and biradar, 2023) employing layer differential tuning, the team harnessed the ulmfit model for advanced feature generation. ulmfit was ingeniously designed to overcome limited labeled data challenges for specific tasks, unfolding through pretraining and fine-tuning stages. the initial pretraining phase involved training a language model on an extensive, unlabeled text corpus like wikipedia, grasping universal language patterns and semantics. the proposed method embraced layered fine-tuning of ulmfit, capitalizing on its core principle: first, pre-trained on general language data, then finetuned on task-specific datasets. this approach facilitated adaptation to task intricacies, leading to enhanced performance across a spectrum of nlp endeavors. ## result the submissions received for the classification of english, bulgarian, hindi, and spanish datasets were 6, 5, 5, and 3, respectively. among these, the team "hate-alert" secured the top rank for both bulgarian and hindi languages, achieving a macro average f1 score of 0.75 and 0.68, respectively. they employed transformer-based models, specifically mbert and xlmr-base, demonstrating the effectiveness of these models. their approach involved fine-tuning mbert, utilizing cls embeddings from both mbert and xlmr and employing dense layers along with a classification head. this innovative use of cutting-edge transformerbased models showcased their superiority over earlier deep-learning models like lsrm and cnn-gru, as depicted in the team "mucs" also secured the first rank and in the spanish languages with the macro f1 score of 0.61, which is shown in for the english language classification, two teams, namely "tercet-english" and "ml ai iiitranchi," both achieved the top rank with a macro f1 score of 0.50. "tercet-english" adopted support vector machines (svm) as their method of choice, benefitting from higher precision, f1 scores, and weighted averages compared to other models such as random forest, logistic regression, and naive bayes. their preprocessing involved text data cleaning and conversion using a tf-idf vectorizer algorithm, subsequently utilized in the svm model. on ## conclusion this paper provides an overview of the hope speech detection shared task conducted during lt-edi-ranlp 2023, with a specific focus on four languages: bulgarian, english, hindi, and spanish. the task attracted participation from eight teams, each submitting predictions for evaluation. the resulting rank list, featuring macro f1 scores as outlined in the result section, is presented. in essence, this paper succinctly captures the essence of the lt-edi-ranlp 2023 hope speech detection shared task. it emphasizes the diverse range of strategies adopted by participating teams and underscores the prominence of machine learning and transformerbased methods, which have contributed to notable enhancements in performance.