Overview of the Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion (2022)

Abstract: Hope Speech detection is the task of classifying a sentence as hope speech or non-hope speech given a corpus of sentences. Hope speech is any message or content that is positive, encouraging, reassuring, inclusive and supportive that inspires and engenders optimism in the minds of people. In contrast to identifying and censoring negative speech patterns, hope speech detection is focussed on recognising and promoting positive speech patterns online. In this paper, we report an overview of the findings and results from the shared task on hope speech detection for Tamil, Malayalam, Kannada, English and Spanish languages conducted in the second workshop on Language Technology for Equality, Diversity and Inclusion (LT-EDI-2022) organised as a part of ACL 2022. The participants were provided with annotated training & development datasets and unlabelled test datasets in all the five languages. The goal of the shared task is to classify the given sentences into one of the two hope speech classes. The performances of the systems submitted by the participants were evaluated in terms of micro-F1 score and weighted-F1 score. The datasets for this challenge are openly available

## introduction social media platforms such as facebook, twitter, instagram and youtube have attracted millions of people to share content and express their opinions. these platforms also serve as a medium for marginalised people who want to receive online help and support from others @xcite @xcite . with the pandemic outbreak, the population from several parts of the world is affected by the fear of losing their loved ones and the loss of access to basic services such as schools, hospitals and mental health care centres @xcite . as a result, people turn to online forums to meet their informational, emotional, and social needs @xcite . online social networking sites provide a platform for people to network, feel socially included, and gain a sense of belonging as part of a community. people's physical and psychological well-being, as well as mental health, are greatly influenced by these factors @xcite @xcite . although social media platforms have these positive aspects, social media content also has a large amount of spiteful or negative posts due to the lack of any mediating authority @xcite @xcite @xcite . in order to tackle this problem, social media posts are analysed to identify and control the spread of negative content using methods such as hate speech detection @xcite , offensive language identification @xcite , homophobia/transpohibia detection @xcite and abusive language detection @xcite . technologies focused on curbing hate speech and offensive language have their own drawbacks, such as training data bias @xcite , and controlling user expression by imposing barriers on modes of speech, thus affecting the principles of equality, diversity and inclusion. therefore, we turn our attention towards spreading positivity rather than curbing individual expression to address negative comments. to this end, last year, we organised the first shared task on hope speech detection for equality, diversity and inclusion in eacl 2021 for english and two under-resourced languages tamil and malayalam @xcite . the english dataset contained monolingual youtube comments, while those of tamil and malayalam contained code-mixed comments. continuing our efforts in this direction, this year, we have organised the second shared task on hope speech detection by extending the dataset with two additional languages, kannada and spanish. it has been launched at the second workshop on language technology for equality, diversity and inclusion (lt-edi-2022), held as a part of acl 2022. in the context of this shared task, hope speech refers to any social media comment that is positive, encouraging, reassuring, inclusive or supportive that inspires and engenders optimism in people's minds. hope speech detection refers to the task of classifying a given comment into one of the following classes hope_speech or non_hope_speech. the participants of the shared task were provided with development, training and testing datasets in all the five languages. the comments in tamil, kannada and malayalam datasets were code-mixed @xcite . this is because the dataset consists of youtube comments and it is very common for speakers of these languages to use code-mixed language in online interactions. we conducted the shared task as a post/comment-level classification task. in this paper, we present the overview of the dataset, the results of the competing systems, and the findings of this shared task. the codalab competition website foot_1 will remain open to allow researchers to access the datasets and build upon this work. ## task description the goal of the proposed shared task is to classify a given social media comment as hope speech or non-hope speech. the participants were provided with training, development, and test datasets in five languages (english, tamil, malayalam, kannada, and spanish). the annotations of the datasets were made at the comment/post level. a comment/post may contain more than one sentence, but the average sentence length of the corpus is one. the participants could choose to take part in classifying one or more languages. leader-board results were published for each language. some sample sentences from the datasets and their annotations are provided below. the comments have also been translated into standard english for the benefit of the reader. for english, tamil and malayalam languages we used the hopeedi dataset from @xcite . the data was collected on a wide range of socially relevant topics such as equality, diversity and inclusion, including lgbtiq issues, covid-19, women in stem, dravidian languages, black lives matter, etc. the inter-annotator agreement was verified using krippendorf's alpha. the kannada hope speech dataset contains 6,176 posts collected from youtube video comments on various topics, such as social oppression, marginalisation and mental health, indo-china border issues, or the banning of mobile apps in india. the details of dataset construction, corpus statistics, interannotator agreement and code-mixing issues are presented in detail in @xcite . the spanish hope speech dataset consists of lgtbi-related tweets that were collected using the twitter api (june 27, 2021 to july26, 2021). as seed for the search a lexicon of lgbitq-related terms, such as #orgullolgtbi or #lgtb was used. a tweet is marked as hs (hope speech) if the text: i) explicitly supports the social integration of minorities; ii) is a positive inspiration for the lgtbi community; iii) explicitly encourages lgtbi people who might find themselves in a situation; or iv) unconditionally promotes tolerance. on the contrary, a tweet is marked as nhs (non hope speech) if the text: i) expresses negative sentiment towards the lgtbi community; ii) explicitly seeks violence; or iii) uses gender-based insults. ## systems we begin this section by presenting a brief summary of the baselines established for this shared task based on the submissions received last year. we then briefly describe each of the proposals submitted this year. readers are encouraged to consult the participants' individual papers for a more detailed understanding. ## results and discussion the total of submissions received for the classification of english, tamil, malayalam, kannada and spanish datasets were 13,7,9,7 and 6 respectively. three teams submitted their results for all the languages, while the other participants made their submissions for a subset of the languages. two teams obtained first rank in english with a macro average of 0.550. one of them (m d gowda et al., 2022) used a resampling technique to deal with imbalanced data and 1d cnn-lstm architecture to address the classification problem. the other team used random forest classifier to classify the comments. transformer-based pretrained models were used in five studies out of which one of them used multilingual indicbert model for classifying english, tamil, malayalam and kannada languages. this model achieved first and second ranks on kannada and malayalam languages respectively. among other submissions, the popular choice was an ensemble of various machine learning classifiers such as logistic regression, multinomial naive bayes, random forest, support vector machines. however, we observed that the performances of the ml classifiers used for this shared task were slightly lower than the baseline performances of ml models used last year. lstm, bil-this paper presents the description of the second shared task on hope speech detection for equality, diversity and inclusion organized at the second workshop on language technology for equality, diversity and inclusion (lt-edi-2022), held as a part of acl 2022. in the 2021 edition this shared task was organized for english and two under-resourced languages, tamil and malayalam, and for this edition, two new languages, kannada and spanish, have been incorporated. in total, 126 participants signed up for the for the shared task and finally 13,7,9,6, and 7 teams submitted their results for english, kannada, malayalam, spanish and tamil, respectively. we hope that this shared task makes a lasting contribution to the nlp field.