Autobots@ LT - EDI - EACL 2021: One World, One Family: Hope Speech Detection with BERT Transformer Model (2021)

Abstract: The rapid rise of online social networks like YouTube, Facebook, Twitter allows people to express their views more widely online. However, at the same time, it can lead to an increase in conflict and hatred among consumers in the form of freedom of speech. Therefore, it is essential to take a positive strengthening method to research on encouraging, positive, helping, and supportive social media content. In this paper, we describe a Transformer-based BERT model for Hope speech detection for equality, diversity, and inclusion, submitted for LT-EDI-2021 Task 2. Our model achieves a weighted averaged f1-score of 0.93 on the test set.

## introduction with the proliferation of the internet, the number of marginalized people looking for help and online support has grown significantly worldwide 1 . recently, social media networks (smn), online blogs, and online support groups (osg) have emerged as popular online support sources. these online support networks play an essential role in marginalized people, such as individuals belonging to lgbtqia+ (lesbian, gay, bisexual, transgender, intersexqueer/questioning, and asexual people) community, people with similar health problems or disabilities, and individuals who belong to racial and ethnic minorities. and some of the research studies @xcite have shown that smns and osgs significantly influence people's self-identification and self-understanding. so it's necessary to detect the positive content from online sources. in this paper, we discussed the identification of hope speech for equality, diversity, and inclusion (edi) from the youtube comments. hope can be defined as a state of mind that brings fortitude, support, and reassurance to life. hope can also come 1 https://www.statista.com/topics/2409/digital-health/ from a motivational discussion about how people deal with difficult situations and cope with them. hope speech has a positive impact on various aspects that harm people's lives. we describe the hope speech for our task as "social media comments that give inspiration, suggestions, insight, and support". we have done various experiments on the provided hope speech dataset for equality, diversity, and inclusion problem using different state-of-the-art machine learning and deep learning models. ## shared task description we are attempting this shared task for english language only. the lt-edi-2021 task 2 (chakravarthi and muralidaran, 2021) is as follows: given a youtube comment or post, the system should identify its class. we have three classes 'hope speech', 'not hope speech' and 'other language.' we classify the comment into the 'hope speech' category when it promotes hope, optimism, faith, support, reassurance, suggestions, or it offers edi values. if a comment doesn't have attributes mentioned in the hope speech class, classify it into the 'not hope speech' class. we categorize the comment into the 'other language' class when the comment not in the english language. recently, to support the rohingyas refugees, @xcite analyze how the hope speech from the social media comments can be used to reduce tensions between india and pakistan during the puluma attack. in this work, the authors intend to find the hope speech in smns can reduce the strain and violence between two nations. the authors created a corpus of 921,235 english youtube comments posted by 392,460 users to accomplish this work. one of the notable research work for hope speech detection is hopeedi (chakravarthi, 2020) corpus creation. in this work, authors created a corpus from user-generated youtube comments for english, tamil, and malayalam. then they developed various machine learning models for benchmark results on the dataset. ## dataset description this paper used the corpus (hopeedi) provided by the "hope speech detection" organizers to train and tune the models. the hopeedi for english dataset containing 28,451 samples, in that 2484 samples belongs to the 'hope speech' class, 25,940 samples belong to the 'not hope speech' class, and the remaining 27 samples belong to the 'other language' class. ## experiments and results to validate our approach, we conducted 3 experiments: initially, we check the baseline model's (svm, mlp, lr, xgboost) results with n-gram level term frequency -inverse document frequency (tf-idf) vectors. secondly, we trained the lstm, rnn, and cnn models with pretrained glove @xcite and fasttext (bojanowski et al., 2016) word embeddings. these models gave the better results than the baseline models but not at all contributing to the 'other language' class whose training samples very less. thirdly, we used the transformer based bert model with a rule based language identification model. this approach performed very well and gave the 0.93 wighted f1-score on test set. the rule based language identification model helped to identify the 'other language' class samples. the as shown, transformer based bert model performed better than other ml and dl learning models for our hope speech detection problem. the supervised machine learning algorithms performed unsatisfactorily due to data imbalance problem. we tried to handle this class imbalance problem using synthetic minority oversampling technique (smote) @xcite , but it did not work well. latterly, we used a rule-based language identification model to remove the non-intended language comments from our dataset. while observing the 'other language' class comments, we find that some of the comments are wrongly tagged. the comment's actual content is in the english language, but the annotators tagged the comment as 'other language'. and if we observe the word cloud of the 'other language' class in figure 4 , we can only see english words only. we note that this is also a problem for not predicting 'other language' comments. to find the right set of hyper-parameters, we used the validation dataset. we implemented the ml algorithms using scikit-learn foot_3 and dl using keras foot_4 . to implement bert model used the hug-gingface foot_5 , it's a pytorch based transformer library. and all details of bert model hyper-parameters are summarized in ## conclusion this paper described a transformer-based pretrained bert model with a rule-based language identification system to detect hope speech in the youtube comments. the bert model helped for the better contextual representation of words in the comment, and the language identification modelassisted in detecting 'other language' comments. in future work, we will handle the class imbalance problem efficiently by improving the dataset. and we will also explore with other transformer models like roberta, xlnet, albert, flair, elmo, etc., for a superior hope speech detection.