NLP - CUET @ LT - EDI - EACL 2021: Multilingual Code-Mixed Hope Speech Detection using Cross-lingual Representation Learner (2021)

Abstract: In recent years, several systems have been developed to regulate the spread of negativity and eliminate aggressive, offensive or abusive contents from the online platforms. Nevertheless, a limited number of researches carried out to identify positive, encouraging and supportive contents. In this work, our goal is to identify whether a social media post/comment contains hope speech or not. We propose three distinct models to identify hope speech in English, Tamil and Malayalam language to serve this purpose. To attain this goal, we employed various machine learning (SVM, LR, ensemble), deep learning (CNN+BiLSTM) and transformer (m-BERT, Indic-BERT, XLNet, XLM-R) based methods. Results indicate that XLM-R outdoes all other techniques by gaining a weighted f_1-score of 0.93, 0.60 and 0.85 respectively for English, Tamil and Malayalam language. Our team has achieved 1st, 2nd and 1st rank in these three tasks respectively.

## introduction nowadays, online and social media platforms have enormous influence and impact on people's societal life. when people undergo a challenging or unfavourable time, they start to find emotional support from their friends, relatives or even virtual platforms to overwhelm this situation. due to covid-19 pandemic, various online forums have become a popular medium of seeking help, suggestion or support. thus, researchers are trying to develop a computational model that can find positive and supportive social media information. in general, the hope speech contains words of inspiration, promise and suggestions. chakravarthi (2020) considered those words as hope speech that offer suggestions, reassurance, support, insight and inspiration. hope speech can be beneficial to save individuals who wish to harm themselves or even attempt to suicide. such speech inspires people during the period of depression, loneliness and stress with the words of promise, suggestions and support @xcite . the major concern of this research is to originate a computational model on top of this dataset to identify hope speech from the social media posts/comments. lack of resources on hope speech research, scarcity of training corpora and multilingual code-mixing are the key concerns to develop such models. machine learning (ml), and deep learning (dl) based techniques can be utilized to address the problem of hope speech detection. in recent years, transformers have gained immense popularity due to its ability to handle the dependencies between input and output with both attention and recurrence. consequently, many nlp tasks have accomplished using the transformer-based model to obtain the state-of-the-art performance @xcite . the principal contributions in this research as listed below: the rest of the paper organized as follows: works related to hope speech detection discussed in section 2. task and dataset are described in detail in section 3. section 4 explains the various techniques used to develop the model for performing the assigned task. experimental findings and error analysis of the models are introduced in section 5. ## related work with the substantial growth of the internet and online contents, several methods have been developed to identify, classify and stop the expansion of negativity such as hate speech detection @xcite , hostility detection @xcite , aggressive language identification @xcite , and flagging abusive contents @xcite . however, very few researchers have put their focus on the other side that is hope speech detection. a little work has conducted till to date in this research avenue of nlp. @xcite analyzed how hope speech can be utilized to mitigate tension between two rival (pakistan and india) countries. supporting texts regarding rohingya community culled from social media in hindi and english languages @xcite . however, details of the dataset such as inter-annotator agreement, diversity of annotators were not clearly described. chakravarthi (2020) developed a multilingual code-mixed hope speech dataset for equality, diversity and inclusion (hopeedi) in english, tamil and malayalam language. data collected from social media like facebook, youtube in trending topics, i.e. covid-19, lgbtiq issues, and india-china war. their models achieved the highest weighted f 1 score of 0.90, 0.56, 0.70 with a decision tree, naive bayes, logistic regression techniques for english, tamil and malayalam languages. ## task and dataset descriptions in this shared task, we have to perform multi-class classification where we aim to identify whether a given comment contains hope speech or not. our system goal is to classify a post/comment into one of the three predefined classes: hope speech (hs), not hope speech (nhs) and not intended language (nil). the shared task organizers (chakravarthi and muralidaran, 2021) developed a hope speech corpus in multilingual code-mixed setup. a total of 28541 (for english), 20198 (for tamil) and 10705 (for malayalam) texts are available in the corpus. this corpus partitioned into three independent sets: train, validation and test. initially, the model is developed on top of the train set, and model hyperparameters are tuned based on the validation set's performance. finally, the model evaluated on the unseen instances of the test set. further investigation on the training set revealed that the training set is highly imbalanced where several documents in 'not intended language' class are much lower than 'hope speech' and 'not hope speech' classes. the average number of words in 'not intended language' is approximately four words for tamil and malayalam languages. the model generalization capability on unseen data might degrade due to the lower number of examples on these classes. detail analysis of the training set presented in ## methodology this section provides a brief discussion of the schemes and techniques employed to address the task (section 3). initially, different feature extraction techniques are exploited with machine learning, and deep learning approaches for a baseline evaluation. moreover, we also applied transformers to obtain a better outcome. figure 1 shows the abstract process of hope speech detection. architectures and parameters of different approaches are described in the following subsections. 4.1 feature extraction techniques ml and dl based techniques are incapable of processing strings or plain text from the raw forms. thus, extracting of appropriate or relevant features is a prerequisite to train ml and dl based systems. we utilize tf-idf @xcite , and fasttext @xcite techniques for extracting features from the texts. tf-idf: tf-idf is a measure that calculates the relevancy of a word to a document in a collection of documents. we calculate the tf-idf value of unigram features for all the languages. during the calculation minimum and maximum document frequency value set at 1. word embedding: embedding features can capture the semantic meaning of a word. to get embeddings features for all the languages, we used keras embedding layer with embedding size 100. during the training phase, keras tries to find the optimal values of the embedding layer's weight matrix by doing simple matrix multiplication and thus create a mapping of each unique words into a vector of real numbers. we utilize the full vocabulary of the corpus and choose maximum input text length 100, 50, and 80 respectively for english, tamil, and malayalam data. fasttext: to alleviate the problem of out of vocabulary words in keras embeddings, we use fasttext embedding. instead of learning vectors directly for words, fasttext represents each word as n-gram of characters. therefore even if a word was not encountered during training, it could be split into n-gram to get its embedding. pre-trained @xcite embedding vectors are used to accomplish the tasks of each language. we retain the default embedding dimension 300 for fasttext embedding. ## results and analysis this section presents a comprehensive performance analysis of various machine learning, deep learning and transformer models for three languages (english, tamil and malayalam). weighted f 1 score uses to determine the excellence of the models. in some cases, other evaluation metrics like precision and recall also considered. in deep learning, the combination of cnn and bilstm experiments with two different embedding features (i.e. ## conclusion this paper describes and analyses the several ml, dl, and transformer-based methods that we have adopted to participate in the hope speech detection shared task at eacl 2021. employing tf-idf, embedding features initially, we performed experiments with ml (lr, svm, ensemble) and dl (cnn+bilstm) approaches. the outcome shows that the ensemble technique achieved higher performance compared to other ml/dl models. further, transformer-based techniques are employed to improve the overall performance. the xml-r model outperformed all the models' performance by achieving the highest weighted f 1 -score of 0.931, 0.854, and 0.602 respectively for english, tamil malayalam language. in the future, contextualized embeddings (such as elmo, flair) and transformers ensemble might explore to investigate the system's performance.