H ope EDI : A Multilingual Hope Speech Detection Dataset for Equality, Diversity, and Inclusion (2020)

Abstract: Over the past few years, systems have been developed to control online content and eliminate abusive, offensive or hate speech content. However, people in power sometimes misuse this form of censorship to obstruct the democratic right of freedom of speech. Therefore, it is imperative that research should take a positive reinforcement approach towards online content that is encouraging, positive and supportive contents. Until now, most studies have focused on solving this problem of negativity in the English language, though the problem is much more than just harmful content. Furthermore, it is multilingual as well. Thus, we have constructed a Hope Speech dataset for Equality, Diversity and Inclusion (HopeEDI) containing user-generated comments from the social media platform YouTube with 28,451, 20,198 and 10,705 comments in English, Tamil and Malayalam, respectively, manually labelled as containing hope speech or not. To our knowledge, this is the first research of its kind to annotate hope speech for equality, diversity and inclusion in a multilingual setting. We determined that the inter-annotator agreement of our dataset using Krippendorff’s alpha. Further, we created several baselines to benchmark the resulting dataset and the results have been expressed using precision, recall and F1-score. The dataset is publicly available for the research community. We hope that this resource will spur further research on encouraging inclusive and responsive speech that reinforces positiveness.

## introduction with the expansion of the internet, there has been substantial growth all over the world in the number of marginalised people looking for support online @xcite @xcite . recently, due to the lockdowns enforced as a consequence of the covid-19 pandemic, people have started to look at online forums as an emotional outlet when they go through a tough time. the importance of the online life of the marginalised population, such as women in the fields of science, technology, engineering, and management (stem), people who belong to the lesbian, gay, bisexual, transgender, intersex and queer/questioning (lgbtiq) community, racial minorities or people with disabilities have been studied, and it has been proven that the online life of vulnerable individuals produces a significant impact on their self-definition @xcite @xcite . furthermore, according to @xcite , @xcite and @xcite , the social networking activities of a vulnerable individual play an essential role in shaping the personality of the individual and how they look at society. comments/posts on online social media have been analysed to find and stop the spread of negativity using methods such as hate speech detection @xcite , offensive language identification @xcite and abusive language detection @xcite . according to @xcite , technologies developed for the detection of abusive language do not consider the potential biases of the dataset that they are trained on. the systematic bias in the datasets causes abusive language detection to be biased and may discriminate against one group over another. this will have a negative impact on minorities. we should turn our work towards spreading positivity instead of curbing an individual's freedom of speech by removing negative comments. therefore, we turn our research focus towards hope speech. hope is commonly associated with the promise, potential, support, reassurance, suggestions or inspiration provided to participants by their peers during periods of illness, stress, loneliness and depression @xcite . psychologists, sociologists and social workers in the association of hope have concluded that hope can also be a useful tool for saving people from suicide or harming themselves @xcite . the hope speech delivered by gay rights activist harvey milk on the steps of the san francisco city hall during a mass rally to celebrate california gay freedom day on 25 june 1978 foot_1 inspired millions to demand rights for equality, diversity and inclusion @xcite . however, to the best of our knowledge, no prior work has explored hope speech for women in stem, lgbtiq individuals, racial minorities or people with disabilities in general. moreover, although people of various linguistic backgrounds are exposed to online social media language, english is still at the centre of ongoing trends in language technology research. recently, some research studies have been conducted on high resourced languages, such as arabic, german, hindi and italian. however, such studies usually use monolingual corpora and do not examine code-switched textual data. code-switching is a phenomenon where the individual switches between two or more languages in a single utterance @xcite . we introduce a dataset for hope speech identification not only in english but also in under-resourced code-switched tamil (iso 639-3: tam) and malayalam (iso 639-3: mal) languages @xcite @xcite . the key contributions of this paper can be summarised as follows: ## related works when it comes to crawling social media data, there are many works on youtube mining @xcite , mainly focused on exploiting user comments. @xcite did an opinion mining and trend analysis on youtube comments. the researchers made an analysis of the sentiments to identify their trends, seasonality, and forecasts, and it was found that user sentiments are well correlated with the influence of real-world events. severyn et al. (2014) did a systematic study on opinion mining targeting youtube comments. the authors developed a comment corpus containing 35k manually labelled data for modelling the opinion polarity of the comments based on tree kernel models. chakravarthi et al. (2020a) and chakravarthi et al. (2020b) collected comments from youtube and created a manually annotated corpus for the sentiment analysis of under-resourced tamil and malayalam languages. methods to mitigate gender bias in natural language processing (nlp) have been extensively studied for the english language @xcite . some studies have investigated gender bias beyond the english language using machine translation to french @xcite and other languages @xcite . @xcite studied the gender and dialect bias in automatically generated captions from youtube. technologies for abusive language @xcite , hate speech @xcite and offensive language detection (nogueira dos @xcite @xcite are being developed and applied without considering the potential biases @xcite @xcite . however, current gender debiasing methods in nlp are not sufficient to debias other issues related to edi in end-to-end systems of many language technology applications, which causes unrest and escalates the issues with edi, as well as leading to more inequality on digital platforms @xcite . counter-narratives (i.e. informed textual responses) is another strategy, which has received the attention of researchers recently @xcite . a counter-narrative approach was proposed to weigh the right to freedom of speech and avoid over-blocking. @xcite created and released a dataset for counterspeech using comments from youtube. however, the core idea to directly intervene with textual responses escalates hostility even though it is advantageous to the writer to understand why their comment/post has been deleted or blocked and then favourably change the discourse and attitudes of their comments. so we turn our research to finding positive information such as hope and encouraging such activities. recently, a work by @xcite and @xcite analysed how to use hope speech from a social media text to diffuse tension between two nuclear power nations (india and pakistan) and support minority rohingyas refugees. however, the author's definition of hope is just defined to diffuse tensions and violence. it does not take other perspectives of hope and edi. the authors did not give more information such as the inter-annotator agreement, diversity in annotators and the details of the dataset. the dataset is not publicly available for research. it was created in english, hindi and other languages related known to the rohingyas. our work differs from the previous works in that we define hope speech for edi, and we introduce a dataset for english, tamil and malayalam on edi of it. to the best of our knowledge, this is the first work to create a dataset for edi in tamil and malayalam, which are under-resourced languages. ## hope speech hope is considered significant for the well-being, recuperation and restoration of human life by health professionals. hope can be defined as an optimistic state of mind that depends on a desire for positive results regarding the occasions and conditions of one's life or the world at large, and it is also presentand future-oriented @xcite . hope can also come from inspirational talk about how people face difficult situations and survive them. hope speech engenders optimism and resilience that positively influences many aspects of life, including work @xcite , college @xcite and other aspects that make us vulnerable @xcite . we define hope speech for our problem as "youtube comments/posts that offer support, reassurance, suggestions, inspiration and insight". hope speech reflects the belief that one can discover pathways to their desired objectives and become roused to utilise those pathways. our work aims to change the prevalent way of thinking by moving away from a preoccupation with discrimination, loneliness or the worst things in life to building the confidence, support and good qualities based on comments by individuals. thus, we have provided instructions to annotators that if a comment/post meets the following conditions, then it should be annotated as hope speech. non-hope speech includes comments that do not bring positivity, such as the following: non-hope speech is different from hate speech. some examples are shown below. hate speech or offensive language detection dataset is not available for code-mixed tamil and codemixed malayalam @xcite , and it does not take into account lgbtiq, women in stem and other minorities. thus, we cannot use existing hate speech or offensive language detection datasets to detect hope or non-hope for edi of minorities. ## dataset construction we focused on collecting data from the social media comments on youtube foot_2 , which is the most widely used platform in the world to express an opinion about a particular video. we avoided taking comments from personal coming out stories of lgbtiq people as it had references to personal details, we manually removed the videos for personal coming out stories. for english, we collected data on recent topics of edi, including women in stem, lgbtiq issues, covid-19, black lives matters, united kingdom (uk) versus china, united states of america (usa) versus china and australia versus china from youtube video comments. the data was collected from videos of people from english-speaking countries, such as australia, canada, the republic of ireland, united kingdom, the united states of america and new zealand. for tamil and malayalam, we collected data from india on the recent topics regarding lgbtiq issues, covid-19, women in stem, the indo-china war and dravidian affairs. india is a multilingual and a multi-racial country. linguistically, india can be divided into three major language families, namely dravidian, indo-aryan and tibeto-burman languages @xcite @xcite . the recent dispute on the indo-china border has triggered racism on the internet towards people with mongoloid features even though they are indians from the north-eastern states. similarly, the national education policy, which advocates for the introduction of sanskrit or hindi has escalated issues regarding the linguistic autonomy of dravidian languages in the state of tamil nadu. we used the youtube comment scraper foot_3 to collect comments. we collected data on the above topics from november 2019 to june 2020 . we believe that our dataset will diffuse hostility and inspire hope. our dataset is produced as a multilingual resource to allow cross-lingual studies and approaches. in particular, it contains hope speech in english, tamil and malayalam. ## benchmark experiments we reported our dataset using a wide range of standard classifiers on the unbalanced settings of the dataset. the experiment was applied on the token frequency-inverse document frequency (tf-idf) of tokens. we used sklearn foot_10 library to create baseline classifiers. for the multinomial naive bayes, we set @xmath0.7. we used a grid search for the k-nearest neighbors (knn), support vector machine (svm), decision tree and logistic regression. more details about the parameters of the classifier will be published in the code. our models were trained on the training dataset; the development set was used to fine-tune the model, and it was evaluated by predicting the labels for the held-out test set, as shown in ## conclusion as online content increases massively, it is necessary to encourage positivity such as in the form of hope speech in online forums to induce compassion and acceptable social behaviour. in this paper, we presented the largest manually annotated dataset of hope speech detection in english, @xcite @xcite @xcite @xcite we believe that this dataset will facilitate future research on encouraging positivity. we aim to promote research in hope speech and to encourage positive content in online social media for equality, diversity and inclusion. in the future, we plan to extend the study by introducing a larger dataset with further fine-grained classification and content analysis.