Hopeful Men@ LT - EDI - EACL 2021: Hope Speech Detection Using Indic Transliteration and Transformers (2021)

Abstract: This paper aims to describe the approach we used to detect hope speech in the HopeEDI dataset. We experimented with two approaches. In the first approach, we used contextual embeddings to train classifiers using logistic regression, random forest, SVM, and LSTM based models. The second approach involved using a majority voting ensemble of 11 models which were obtained by fine-tuning pre-trained transformer models (BERT, ALBERT, RoBERTa, IndicBERT) after adding an output layer. We found that the second approach was superior for English, Tamil and Malayalam. Our solution got a weighted F1 score of 0.93, 0.75 and 0.49 for English, Malayalam and Tamil respectively. Our solution ranked 1st in English, 8th in Malayalam and 11th in Tamil.

## introduction the spread of hate speech on social media is a problem that still exists today. while there have been attempts made at hate speech detection @xcite to stop the spread of negativity, this form of censorship can also be misused to obstruct rights and freedom of speech. furthermore, hate speech tends to spread faster than non-hate speech @xcite .while there has been a growing amount of marginalized people looking for support online @xcite , there has been a substantial amount of hate towards them too @xcite . therefore, detecting and promoting content that reduces hostility and increases hope is important. hope speech detection can be seen as a rare positive mining task because hope speech constitutes a low percentage of overall content @xcite . there has been *these authors contributed equally to this work. work done on hope speech or help speech detection before that has used logistic regression and active learning techniques @xcite . in our paper, we will be doing the hope speech detection task on the hopeedi dataset @xcite ) which consists of user comments from youtube in english, tamil and malayalam. in this paper, we will first look at the task definition, followed by the methodology used. we will then look at the experiments and results followed by conclusion and future work. ## task definition the given problem is a comment level classification task for the identification of "hope speech" within youtube comments, wherein they are to be classified as "hope speech", "not hope speech" and "not in intended language". the data provided in the task was annotated at a per-comment basis wherein a comment could be composed of more than one sentence. ## methodology this section talks about the methodology that we have used to solve the task. as shown in figure 1 , the pipeline involves preprocessing, language detection, transliteration (for indian languages), and hope speech detection. these steps are described in this section. ## experiments initially, the entire database is preprocessed to remove extra tab spaces, punctuations, emojis, mentions and links. in the case of malayalam and tamil, we also transliterate the entire database. then we distributed our experimentation procedure into two different approaches. in the first approach, we finetune our pre-trained masked language models using the train and validation splits for the purpose of making them more suitable to the subsequent classification task. thereafter, contextual embeddings for each sentence in the dataset are produced by calculating the average of the second to last hidden layer for every single token in the sentence. we then trained logistic regression, random forest, svm and rnn-based classifier models using these embeddings. in the second approach, all the sentences are encoded into tokens using the respective tokenizers and then we add a linear layer on top of the pre-trained model layers after dropout. all the layers of the devised model are then trained such that the error is back propagated through the entire architecture and the pre-trained weights of the model are modified to reflect the new database. for both these approaches, we then calculated predictions for the test split and reported performance metrics. for english, we try out three different pre-trained models: "roberta-base", "bertbase-uncased", and "albert-base-v2" for both the approaches. for tamil and malayalam however, only the indicbert model is applicable for either approach. ## conclusion and future work in this paper, we presented our approach for hope speech detection in english, tamil, and malayalam on the hopeedi dataset. we used two approaches. the first approach involved using contextual embeddings to train various classifiers. the second approach involved using a majority voting ensemble of 11 models which were obtained by fine-tuning pre-trained transformer models. the second approach using the roberta-base model was the best performing model for english, giving a weighted f1 score of 0.93. the second approach using indicbert model gave the best performance for tamil and malayalam, giving a weighted f1 score of 0.75 for malayalam and 0.49 for tamil. in the future, we plan to fine-tune transformers pretrained on code mixed data. data augmentation methods like synonym replacement and random insertion could be used to fine-tune the model on more data.