Findings of the Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion (2021)

Abstract: Hope is considered significant for the well-being, recuperation and restoration of human life by health professionals. Hope speech reflects the belief that one can discover pathways to their desired objectives and become roused to utilise those pathways. To encourage research in natural language processing towards positive reinforcement approach, we created a hope speech detection dataset. This paper reports on the shared task of hope speech detection for Tamil, English, and Malayalam languages. The shared task was conducted as a part of the EACL 2021 workshop on Language Technology for Equality, Diversity, and Inclusion (LT-EDI-2021). We summarize here the datasets for this challenge which are openly available at https://competitions.codalab.org/competitions/27653 , and present an overview of the methods and the results of the competing systems. To the best of our knowledge, this is the first shared task to conduct hope speech detection.

## introduction in the recent years, there has been an exponential rise in the number of studies focusing on the detection and management of hate speech and offensive language in social media @xcite @xcite . however, this has led to controlling user expression instead of improving user experience. @xcite @xcite . this has also resulted in putting barriers on modes of expression of different groups of people resulting in the violation of the principles of equality, diversity, and inclusion (edi). for instance, some nlp systems have classified the comments made in african american english as offensive language without accommodating the linguistic features peculiar to the dialect. hence there is a need for a shift in the approaches taken to handle hate speech without compromising on the principles of edi. we propose to shift the prevailing research direction in natural language processing from controlling negativity (curbing hate speech etc.) to encouraging positivity (promoting hope speech). hope speech is any expression that is positive, encouraging, supportive, and / or inspires promise of the future @xcite . for the shared task organised in this connection, participants were provided with development, training and test dataset in english as well as in two under-resourced languages -tamil and malayalam. tamil (iso 639-3: tam) belongs to the dravidian language family and is widely spoken in the southern state of tamil nadu in india, sri lanka, malaysia and singapore @xcite @xcite @xcite . malayalam (iso 639-3: mal) also belongs to the dravidian language family and is spoken in the indian state of kerala and the union territories of @xcite . both tamil and malayalam have their own scripts which are alphasyllabaries like other indic scripts i.e. they are partially alphabetic and partially syllabic. the tamil language was written using tamili, vattezhuthu, chola, pallava and chola-pallava scripts at different points in history. the modern tamil script descended from the chola-pallava script that was conceived around the 6th century ce @xcite . malayalam was first written with vattezhuthu script that evolved from tamili script around 4-5th century ce @xcite . modern malayalam is written using the vattezhuttu alphabets extended with symbols from the grantha script to accomodate non-native sanskrit sounds @xcite . although they have their own scripts, the social media comments in these languages are often written in latin script as it is easy to input. our dataset for the hope speech for edi shared task was created from user-generated content in tamil, malayalam, and english (chakravarthi and muralidaran, 2021). the user-generated comments in our dataset for tamil and malayalam were codemixed @xcite . we proposed a comment/post level classification task. the goal for the participants was to classify a given youtube comment into either 'hope speech', 'non hope speech' or 'not tamil / not malayalam / not english'. our codalab website will remain open to allow researchers to access the data and build upon this work. ## task description we define the hope speech for our problem as "youtube comments/posts that offer support, reassurance, suggestions, inspiration and insight". a comment/post within the corpus may contain more than one sentence, but the average sentence length of the corpus is one. the annotations in the corpus are made at a comment/post level. the participants were provided with development, training and test dataset in english, tamil, and malayalam. a few examples from the dataset are given below along with their translations and hope speech class annotations. thanks doctor. not malayalam ## systems we first describe the baseline systems from the hopeedi paper. next, we briefly describe the general approach taken by each participating team. we encourage the readers to refer to the individual papers of each team for more details. ## results and discussion overall, we received a total of 31,31 and 30 submissions for english, malayalam and tamil tasks. it is interesting to note that the top performing teams in all the three languages predominantly used xlm-roberta to complete the shared task. one of the top ranking teams for english used context-aware string embeddings for word representations and recurrent neural networks and pooled document embeddings for text representation. among the other submissions, although bi-lstm was popular, there were other machine learning and deep learning models that were used. however, they did not achieve good results compared to the roberta based models. the top scores were 0.61, 0.85 and 0.93 for tamil, malayalam and english respectively. the range of scores was between 0.37 to 0.61, 0.49 to 0.85 and 0.61 to 0.93 for tamil, malayalam and english datasets respectively. it can be seen that the f1 scores of all the submissions on the tamil dataset were considerably lower than those of malayalam and english. it is not suprising that the english scores were better because many approaches used variations of pretrained transformer based models trained on english data. due to codemixing at various levels the scores are naturally lower for malayalam and tamil datasets. among these two, the systems submitted performed badly on tamil data. the identification of the exact reasons for the bad performance in tamil requires further research. however, one possible explanation for this could be that the distribution of 'hope speech', 'non hope speech' and 'not-tamil' classes is starkly different from those of english and malayalam. in the english dataset there were just 27 out of 28,451 comments which were labelled as not-english by the annotators. in the remaining two classes, the number of non-hope speech comments were significantly higher than hope speech comments. similarly in malayalam there were only 888 out of 10,705 comments that were not malayalam and overwhelmingly higher proportion of non-hope speech in the dataset. however in tamil dataset, there were 2,483 comments out of 20,198 comments that were not tamil. moreover the proportion of non-hope speech comments in tamil (9,816 comments) is close enough to the number of hope speech labels (7,899 comments). this made the classifications in english and malayalam a binary prediction task rather than a three class classification and naturally the former systems performed better than the latter. the participants rarely faced challenges during the training, testing and submission phase. however, there were minor inconsistencies in the labelling of classes that was duly rectified on codalab immediately. the intial round of peer review offered significant suggestions to the participants in terms of rewriting paper in a more presentable way. these suggestions of the reviewers were duly incorporated before the final submissions. ## conclusion in this paper, we have presented the first shared task results on hope speech detection for equality, diversity, and inclusion. we received a wide range of entries that satisfied the goals of the shared task. we hope that hopeedi shared task makes a lasting contribution to the nlp field.