GCDH @ LT - EDI - EACL 2021: XLM - R o BERT a for Hope Speech Detection in E nglish, M alayalam, and T amil (2021)

Abstract: This paper describes approaches to identify Hope Speech in short, informal texts in English, Malayalam and Tamil using different machine learning techniques. We demonstrate that even very simple baseline algorithms perform reasonably well on this task if provided with enough training data. However, our best performing algorithm is a cross-lingual transfer learning approach in which we fine-tune XLM-RoBERTa.

## introduction in recent years, the spread of negative comments and hatred through social media has created a focus on the detection of misinformation and hate speech in the nlp community. on the other hand, hope speech detection is a relatively new task. it can be employed to identify the positive aspects of large collection of social media posts, e.g. detecting pro-peace voices during politically heated situations. these detection efforts can potentially counter-act the perceived majority of hatred, and be a means to prevent harsh measures such as the disabling of internet access @xcite . the ability to identify and therefore foster these aspects of online communication is a step towards a more positive representation of the internet. the hypothesis is: if hate-speech can incite violence, hope speech can ease tensions. for the first time to our knowledge, a shared task has been announced to identify hope speech in a multi-lingual dataset of youtube comments. (chakravarthi and muralidaran, 2021) ## data the dataset consists of three different subsets. it contains 10,705 comments for malayalam, of which 8,564 are assigned for training, 1,070 for validation, and 1,071 for testing. for tamil, the organizers provided 20,198 comments, of which 16,160 were designated for training, 2,018 for validation, and a test set of 2,020 comments. the english data consists of 28,451 comments in total, 22,762 for training, 2,843 for development, and 2,846 for testing. the organizers provided annotations for the classes hope speech, non hope speech, and not-"target language", where "target language" is either malayalam, tamil, or english. figure 1 illustrates how the different classes are represented in the data sets. notably, we can observe that the tamil data is more balanced, whereas the malayalam and english data sets show an over-representation of non hope speech data. furthermore, only 22 comments in the english training set are labelled "not-english", but 12 % of the comments in the tamil training set are "not-tamil", and 8 % of the malayalam comments are "not-malayalam". annotations for the hope speech class include utterances that convey a generally bright prospect to the future, are supportive, insightful, and promote values such as inclusiveness and equality, among others. some of these concepts for the english training set are illustrated in figure 2 . for a full description of the annotation process, see @xcite . ## baseline using scikit-learn @xcite , we implemented two baseline algorithms for the task of hope speech detection, a naive bayes algorithm, and a support vector machine. from the short texts, we first extracted a number of features including unigrams, bigrams, length and number of stopwords. additionally, we assigned emotion scores for all three languages based on the nrc emotion lexicons @xcite . however, we found that solely relying on the textual data using a simple vectorizer based on tf-idf yielded better results. as a first baseline, we implemented a complement naive bayes model @xcite . we iteratively found the best smoothing parameters α = 1.3 for malayalam, α = 10.5 for tamil and α = 1.4 for english. for the second baseline, we used an svm with a degree of three and linear kernel. we found the best regularization parameters @xmath0.3 for malayalam, @xmath1.1 for tamil and @xmath2.4 for english. ## transformer model our main model is based on xlm-roberta @xcite , a pre-trained neural language model that uses the transformer architecture (more specifically, we use the xlm-roberta-base version from the huggingface transformers library @xcite ). it is particularly useful for this task, because it is already pre-trained on 100 different languages, including malayalam, tamil and english. we preprocess the provided datasets by tokenizing the comments using the pre-trained senten-cepiece @xcite tokenizer that belongs to xlm-roberta. all comments are padded or truncated to a length of 128 tokens. xlm-roberta is able to implicitly infer the language used in a text based on the tokens that it contains, which enables it to deal with code-switching. this also means that we can use batches that contain multiple languages at the same time during training. ## conclusion in this paper we presented several machine learning models trained for hope speech detection in three different languages (malayalam, tamil and english). as baseline models we used svm and complement naive bayes classifiers for each language. the best results were achieved by a crosslingual transfer learning approach in which we finetuned xlm-roberta. in the competition, this model achieved the first rank for malayalam and english and the fourth rank for tamil. in conclusion, we provide a cross-lingual model that is both effective and relatively fast to train.