KU _ NLP @ LT - EDI - EACL 2021: A Multilingual Hope Speech Detection for Equality, Diversity, and Inclusion using Context Aware Embeddings (2021)

Abstract: Hope speech detection is a new task for finding and highlighting positive comments or supporting content from user-generated social media comments. For this task, we have used a Shared Task multilingual dataset on Hope Speech Detection for Equality, Diversity, and Inclusion (HopeEDI) for three languages English, code-switched Tamil and Malayalam. In this paper, we present deep learning techniques using context-aware string embeddings for word representations and Recurrent Neural Network (RNN) and pooled document embeddings for text representation. We have evaluated and compared the three models for each language with different approaches. Our proposed methodology works fine and achieved higher performance than baselines. The highest weighted average F-scores of 0.93, 0.58, and 0.84 are obtained on the task organisers’ final evaluation test set. The proposed models are outperforming the baselines by 3%, 2% and 11% in absolute terms for English, Tamil and Malayalam respectively.

## introduction in recent years, social media became an integral part of human life and people started spending more time on these platforms. but people are mindful of social media behaviour and putting less personal information in the public domain @xcite . now social media behaviors changed quite dramatically and we are living not just in a pandemic, but also in an "infodemic", where fake news is becoming more common @xcite . conversations on the internet are often a reflection of the conversations that one makes offline. several ai techniques are adopted to analyse the online comments in social media, which are intensified on the detection of negative com-ments such as hate speech detection, offensive language identification and abusive language detection (chakravarthi, 2020a). hate speech is widely used in media, internet and public discourse which seen or read in the media expressing disapproval, hatred, or aggression towards minorities, could lead to violence and form negative impact on minorities. hope speech detection is a new task related to hate speech detection for finding and highlighting positive comments or supporting content, rather than just filtering hostile content @xcite . for this task, youtube comments/posts that offer support, reassurance, suggestions, inspiration and insight are recognized as hope speech. in bilingual and multilingual communities linguistic code-switching occurs in social groups @xcite @xcite . it is incredibly important in many social groups, when an individual uses a group's dialect or accent, the audience is more receptive to the content @xcite . recently, many researchers are focused on high resource languages using monolingual corpora but less attention is given to code-switching languages especially under resourced languages like indian languages @xcite . in our work, we used hope speech dataset for equality, diversity and inclusion (hopeedi) not only in english but also code-switched tamil and malayalam (chakravarthi, 2020a). ## related work the authors @xcite introduced a novel task for detecting hostility diffusing content from comments in social media, dubbed hope-speech detection. the authors analysed and studied the importance of automatic identification of user-generated hope speech web content that diffuse tension and violence among people in an international crisis. finally, the obtained results are very promising and automatic recognition of hope speech may also find applications in many other contexts. but they restricted the definition of hope into diffuse tension and violence not considering the other perspectives of hope. chakravarthi (2020a) constructed a multilingual hope speech dataset for equality, diversity and inclusion (hopeedi) containing user-generated comments from youtube for english and two lowresource languages, malayalam and tamil. the authors considered much more perspectives support, reassurance, suggestions, inspiration and insight of the hope and edi. to facilitate future research on encouraging positivity, the authors make this dataset publicly available and created several baselines to benchmark the proposed dataset. ## materials and methods this section describes the dataset used for our experiments and technical description of the proposed methodology. ## results and analysis this section presents the results and analysis of our experiments, which we have explained in the previous sections. as well as, we present the baseline results obtained by the authors (chakravarthi, 2020a), to compare the performance with our proposed models. the final evaluation results obtained for participants' submissions by task organisers are also presented in overview paper @xcite . the best-performing classifiers on hopeedi datasets are considered as baseline models for this task (see as shown, all our proposed models for hope edi detection task in three languages work well and show significant improvement than the bestperformed baseline models. the results indicate that a hope edi detection classifier with good precision and recall can be constructed using deep learning approaches. it may be due to the representational power of pre-trained word embeddings or language models to capture semantic and lexical structure. it has virtually replaced the feature engineering part of supervised machine learning classifiers. for english and malayalam, our model with stacked word embeddings and pooled document embedding achieved the highest performance on the hopeedi dataset with a weighted average f-score of 0.93 and 0.84 respectively. but in tamil, stacked pwe with rnn document embedding performed better than other models with weighted average f-score of 0.58. furthermore, it can be seen that all malayalam language models (0.79,0.83,0.84) achieved greater improvement than baseline models(0.73). other two languages exhibit slight improvement in proposed models than baseline models. in most cases, it can be observed from the also, we can observe that the "non-hopespeech" class for english shows similar performance in both models and "hope-speech" class hold a remarkable improvement over baseline models. in tamil and malayalam the majority class "non-hope-speech" possesses approximately similar f-scores for baseline and proposed the best model. another notable and interesting observation from figure 2 and 3 is that the two minority classes in malayalam dataset, "hope-speech" and "not-malayalam" shows comparable performance approximately 30% above f-score than the baseline model. also, "not-tamil" class shows approximately 25% above f-score than the baseline model. these higher scores indicate that deep learning techniques can classify the correct class regardless of label distribution better than traditional machine learning technique. this may because our deep learning techniques rely on pre-trained embeddings and language model, which is a contextualized word representation allowing a word to be associated with multiple word vectors, whereas the classical techniques rely on merely manually selected features. ## conclusion we have presented a deep learning technique with contextual aware embeddings for hope speech de-tection task in three languages: english, malayalam and tamil (code-switched). we have used contextual string embedding (flair) and pre-trained word embeddings (pwe) for word representations and recurrent neural network (rnn) and pooled document embedding with max pool operation for text representations. all the three models were evaluated for each language and compared the baseline models using the dataset given by hope speech detection for equality, diversity, and inclusion (hopeedi) shared task organizers. all model performances are measured using weighted average f-score due to the imbalanced class distribution in the dataset. we have obtained the highest f-scores 0.93, 0.58 and 0.84 for three languages english, tamil and malayalam respectively, which significantly improved performance over baselines (0.90, 0.56, and 0.73). for the minority class 'not-language', the proposed best model improved 35% and 25% performance than baselines for malayalam and tamil, respectively. based on these observations, we conclude that the deep learning models with contextual string embeddings are well suited for hopeedi detection task with an imbalanced dataset. we could also achieve good performance results with moder-ate resources (one gpu and a small corpus), even without optimizing hyperparameters. the performance can be improved further by fine-tuning hyperparameters and pre-trained contextual embeddings, incorporating different attention mechanisms and increasing the size of the training dataset.