IIITS urat@ LT - EDI - ACL 2022: Hope Speech Detection using Machine Learning (2022)

Abstract: This paper addresses the issue of Hope Speech detection using machine learning techniques. Designing a robust model that helps in predicting the target class with higher accuracy is a challenging task in machine learning, especially when the distribution of the class labels is highly imbalanced. This study uses and compares the experimental outcomes of the different oversampling techniques. Many models are implemented to classify the comments into Hope and Non-Hope speech, and it found that machine learning algorithms perform better than deep learning models. The English language dataset used in this research was developed by collecting YouTube comments and is part of the task “ACL-2022:Hope Speech Detection for Equality, Diversity, and Inclusion”. The proposed model achieved a weighted F1-score of 0.55 on the test dataset and secured the first rank among the participated teams.

## introduction social networking platforms such as instagram, facebook, linkedin, and youtube have become the default place for worldwide users to spend time @xcite @xcite . these social platforms are not only used to share success but also used to ask for help during emergency @xcite . as per the report foot_0 , on average, six hours in a week, every indian uses the social networking platform. among them, teenagers and some professionals are more active to share their life events. people have two images: one for the real world where they live and another for the virtual world, like the images on social platforms where people are connected to their close friends and communicate with strangers in the virtual environment @xcite . language is a primary requirement for communication. languages like hindi, english, japanese, gujarati, marathi, tamil, and others are used to express success, life events like job promotion, being selected as the best team member, etc @xcite @xcite @xcite . tamil is one of the world's longest-surviving classical languages. tamil is a member of the southern branch of the dravidian languages, a group of about 26 languages indigenous to the indian subcontinent. it is also classed as a member of the tamil language family, which contains the languages of around 35 ethno-linguistic groups, including the irula and yerukula languages @xcite @xcite @xcite . the earliest old tamil documents are small inscriptions in adichanallur dating from 905 bc to 696 bc. tamil has the oldest ancient non-sanskritic indian literature of any indian language @xcite @xcite @xcite . everyone needs feelings like happiness, sadness, anger, and the motivation for failure in their hard time @xcite . among all, the comments having the context of "well-being" are termed as "hope speech". more specifically, hope speech reflects the belief that one can discover and become motivated to use pathways to achieve one's desired goals @xcite @xcite . the other category of comments can be abuse, demotivate, neutral, race, or sexually oriented and similar ones which are termed as "non-hope speech". such comments do not live long in the physical world where people speak something today that might not be remembered after a few days or months, even the reachable to the limited region. however, if the same is communicated via a social platform, it will re-main active and affect the victim for a long-time @xcite . the social platform is polluted with hateful content @xcite and is a challenging task to filter. moreover, finding the hopeful message becomes another challenging task because of their low appearance. people who are in trouble are expecting a solution for their issues. for example, if a person becomes a victim of cybercrime like borrowing money from a bank account. then they will reach out to the concerned authority hoping that their money will be rolled back into the account. if people face issues with the company rules and regulations, they ask for opinions via social posts hoping that someone will suggest the right solution to get rid of it. these social platforms receive huge content from worldwide users from different genres like entertainment, promotion, publicity, achievement, political news, etc. every genre has both positive and negative comments. all of the mentioned scenarios are common in human life, where directly or indirectly, people always expect some positive news with hope (chakravarthi, 2020). finding hope speech content from social platforms manually is challenging and not a feasible option. hence there is a need of automated tools which can be helpful for hope-oriented comment detection @xcite . to address the said problem, this research uses both traditional machine learning (ml) models and deep learning (dl) based models to find the bestsuited technique to detect such hope speech. the dataset used in this research was taken from lt-edi-acl2022 workshop. the major contributions are as follows: the rest of the paper is organized as follows: section 2 discusses the relevant research works. section 3 describes the overview of the task in detail. section 4 explains the data preparation for the experiment followed by experimental setup in section 5. section 6 discusses the experimental outcomes of different models. finally, the work is concluded in section 7 with limitation and future scope. ## related works even though the hope speech is termed as positive vibes, very less attention is received from the research community to address it. the reason behind less research in the domain may include the unavailability of the labeled dataset. in the last few years, this problem has received some fruitful attention while the organizer of the lt-edi-eacl2021 shared a labeled dataset. some of the submitted frameworks in the lt-edi-eacl2021 workshop is to address this hope speech detection issue. many research works have reported to filter the hateful, and offensive comments from the social post in recent years @xcite . however, identifying the hopeful comments received less attention (chakravarthi, 2020; @xcite . @xcite used transformer-based models like bert, albert, distilbert, and similar ones to classify the comments into three categories: hope, non-hope, and other categories. dataset of three languages were used in their research, english, malayalam, and tamil. for the english language, the ulmfit model achieved the best weighted f1-score value of 0.9356. (upadhyay et al., 2021) also used the transformer-based model to classify the comments into hope, nonhope, and other categories. deep learning models -convolutional neural network (cnn), long short term memory (lsmt), and bidirectional lstm approaches were used by (saumya and mishra, 2021) on all three datasets. their best-performing cnn-lstm model achieved an f1-score of 0.91 on english. ## task and dataset overview in lt-edi-acl2022, task 1 was hope speech detection for equality, diversity, and inclusion, where the event organizer provided an annotated dataset for three languages tamil, malayalam, and english. the dataset was labeled into two categories: 'hope speech and non-hope speech'. the shared task's objective was to build an automated model that predicts the comments are hope speech or non-hope speech. initially, the training dataset was released. later, the validation and test dataset was released by the organizer. this research uses only english comments for the experiment. the training dataset had a total of 20778 numbers of non-hope speech sample whereas in hope speech 1962 sample. 2569 non-hope speech and 272 hope speech samples were present in the validation dataset. finally, the test dataset was released without any label on which the final rank of the participated teams was decided (chakravarthi and muralidaran, 2021; chakravarthi, 2020; hande et al., 2021). ## data preprocessing as the dataset was compiled with comments collected from youtube, it consisted of many irregularities like the use of emoticons/emojis, short text, customized fonts, and tagged users. all these need to be cleaned for the data to be passed onto the model for training. during the preprocessing of the data, the emojis were replaced with their mapped meaning by using demoji library 2 . tagged users and punctuation were removed and also removed all custom fonts and numerals, single-character words, and multiple spaces that were introduced by the previous steps. ## experimental setup this section discusses a detailed experimental procedure used for the model development. the traditional ml techniques, namely, logistic regression (lr), random forest (rf), naive bayes (nb), and extreme gradient boosting (xgb), are selected for the experiment. the performance of these models is evaluated with precision, recall, and f1 score @xcite . firstly, a total of 5000 features were extracted from the processed data using tf-idf vectorization with 1-5 n-grams, which was further scaled using the min-max scalar. the oversampling techniques mentioned above were used to balance the dataset before passing it to the model. before oversampling, the total train data size was 22,740. after oversampling, the total number of samples increased to 41,556, with both the classes divided equally. the balanced dataset was then passed to the ml classifiers with the help of 10-fold cross-validation over the training dataset. we implemented all the combinations of the selected classifiers and oversampling techniques. the average accuracy obtained using a 10-fold cross-validated approach is shown in further, deep learning techniques like dnn, dnn with embeddings (dnn+emb), cnn, lstm, and bilstm are implemented to address this issue. the dnn model is comprised of a simple four-layer neural network with 256, 128, and 64 neurons at the hidden layer with a single output neuron. in dnn + emb, we have implemented an additional embedding layer of 120 dimensions. a single convolution layer is used in cnn, followed by a maxpooling layer and hidden layers of 128 and 64 neurons. similarly, the lstm and bilstm networks are implemented with 256 memory units with the same amounts of hidden layers. the output layer consisted of a single neuron with sigmoid activation for each model. after further hyperparameter tuning, we concluded by using the adam optimizer with a learning rate of 0.0001 and binary cross-entropy as the optimization function. the model was trained with the smote oversampled train data and was validated with the provided validation data set, the results of which are provided in ## results in this section, the experimental outcomes of the different models will be discussed. we are comparing the performances of the ml models based on the 10-fold cross-validated outcomes reported in ## conclusion social platforms have become a medium to share opinions, achievements, successes, and failures.