CFILT IIT B ombay@ LT - EDI - EACL 2021: Hope Speech Detection for Equality, Diversity, and Inclusion using Multilingual Representation from T ransformers (2021)

Abstract: With the internet becoming part and parcel of our lives, engagement in social media has increased a lot. Identifying and eliminating offensive content from social media has become of utmost priority to prevent any kind of violence. However, detecting encouraging, supportive and positive content is equally important to prevent misuse of censorship targeted to attack freedom of speech. This paper presents our system for the shared task Hope Speech Detection for Equality, Diversity, and Inclusion at LT-EDI, EACL 2021. The data for this shared task is provided in English, Tamil, and Malayalam which was collected from YouTube comments. It is a multiclass classification problem where each data instance is categorized into one of the three classes: ‘Hope speech’, ‘Not hope speech’, and ‘Not in intended language’. We propose a system that employs multilingual transformer models to obtain the representation of text and classifies it into one of the three classes. We explored the use of multilingual models trained specifically for Indian languages along with generic multilingual models. Our system was ranked 2nd for English, 2nd for Malayalam, and 7th for the Tamil language in the final leader board published by organizers and obtained a weighted F1-score of 0.92, 0.84, 0.55 respectively on the hidden test dataset used for the competition. We have made our system publicly available at GitHub.

## introduction the prominence of web-based media is expanding quickly because it is being used to create and share content, even by those who are ignorant of online media. several web platforms allow users to add textual feedback on non-textual content, such as images, photos, animations, etc. with millions of videos posted by its users and billions of comments on all these videos, youtube is undoubtedly the most famous of them. online social media comments/posts have been examined to identify and avoid the propagation of negativity using strategies such as detecting abusive language detection @xcite and hate speech @xcite . there is a lot of work that is being done to remove the negativity from the web but hope speech detection focuses to spread positivity by detecting content that is encouraging, positive, and supportive. when it comes to hope speech detection there has not been much work done but recently the nlp community has started showing interest in this area. in a work by @xcite , they have analyzed youtube comments and performed the task of hope speech detection to identify hostilitydiffusing content. here the authors have not taken other aspects like equality, diversity, and inclusion into account. chakravarthi et al. (2020) did work in indian languages where they manually annotated the youtube comments for tamil and malayalam languages for performing sentiment analysis. in a similar work, chakravarthi (2020) released the dataset consisting of youtube comments with hope and not-hope speech annotation. hate speech is a well-researched area related to hope speech. according to the survey done by @xcite , automatic hate detection was needed due to a large number of people using the net and the massive scale with which the web is growing. @xcite used deep neural networks for the task and they were able to outperform the best performing method by up to 5 percentage points in macro-average f1-score. multilingual bert @xcite is a variant of bert @xcite that has been heavily used by the nlp community. @xcite in their work showed that multilingual rep-resentation by multilingual bert handles cross linguality without being explicitly trained for it. it also handles transfer across scripts and to codeswitching fairly well. @xcite proposed another variant of the bert model called xlm-roberta by pre-training multilingual models at scale. there have been various attempts to tackle problems related to indian languages by training transformer models specifically for indian languages. indic bert @xcite and muril ( https://tfhub.dev/google/ muril/1 ) are two such transformer-based language models. we plan to tackle the hope speech discovery for the english, tamil, and malayalam language by obtaining representation from multilingual transformer models. tamil and malayalam are dravidian languages locally spoken in the states of tamil nadu and kerala respectively on indian territory. for a country like india, where people speak many languages, code-mixing is fairly common @xcite @xcite . the dataset for this task is code-mixed such as tag, inter-sentential, and intra-sentential (chakravarthi, 2020). the shared task was launched as a part of the first workshop on language technology for equality, diversity, inclusion (lt-edi-eacl-2021) (chakravarthi and muralidaran, 2021). it was conducted for english, tamil, and malayalam language categories. the task was to classify a given piece of text scraped from youtube comments into one of the three possible categories. these three possible categories or labels were: the rest of the paper is paper is organized as follows. in section 2, we provide the dataset details and statistics. section 3 consists of our system description and architecture details. in section 4, we describe our experimental setups and report our results. we conclude this paper in section 5 and briefly discuss our future plans for tackling this problem. ## dataset the organizers of lt-edi 2021 have provided the dataset for training, validation, and testing of the systems for the shared tasks. the dataset consists of pairs of text and their corresponding label. training and validation sets were provided with labels for developing the systems for the given shared task and a test set was supplied without ground truth labels for a fair evaluation and publishing final results and team ranking among participants.however, the labels for test data were also made available once competition concluded and final results were declared. however, the labels for test data were also made available once the competition concluded and the final results were declared. the dataset was unevenly distributed across three possible classes for all three languages. ## system description the core deep learning model of our system consists of bert-based transformer model in a multilingual setting. we did not perform any kind of pre-processing of the text data to avoid computational overhead at run time and to evaluate efficacy of multilingual transformers on raw text. we obtained a pooled 768-dimensional vector representation for the entire raw text of each instance. this vector is then fed to a softmax layer which gives the probability distribution of the sentence being from given three possible classes. this setup of trained end-to-end with cross-entropy loss function and f1-score as an evaluation metric. ## experiments and results we have experimented with various multilingual transformer-based models that were trained on multiple languages together. multilingual models pretrained specifically for indian languages were also employed for the given shared tasks. ## conclusion and future work in this paper, we have described our system submitted for the shared task in hope speech detection and reported its performance. we have extensively experimented with the possibility of employing multilingual models for given tasks. we report the performance of four different multilingual models which include transformer models specifically trained for indian languages. our experiments showed that transformer models pre-trained on a smaller set of languages have the potential to perform at par or better than models trained on hundreds of languages. in the future, we plan to introduce some linguistic-based features and combined them with multilingual transformer representation to improve the overall effectiveness of the system. we also plan to systematically study the effect of abolishing the 'not in intended language' class from the dataset as it opens up the opportunity to have a single system or deep learning model for all three