MUCIC @ LT - EDI - ACL 2022: Hope Speech Detection using Data Re-Sampling and 1 D Conv- LSTM (2022)

Abstract: Spreading positive vibes or hope content on social media may help many people to get motivated in their life. To address Hope Speech detection in YouTube comments, this paper presents the description of the models submitted by our team - MUCIC, to the Hope Speech Detection for Equality, Diversity, and Inclusion (HopeEDI) shared task at Association for Computational Linguistics (ACL) 2022. This shared task consists of texts in five languages, namely: English, Spanish (in Latin scripts), and Tamil, Malayalam, and Kannada (in code-mixed native and Roman scripts) with the aim of classifying the YouTube comment into “Hope”, “Not-Hope” or “Not-Intended” categories. The proposed methodology uses the re-sampling technique to deal with imbalanced data in the corpus and obtained 1st rank for English language with a macro-averaged F1-score of 0.550 and weighted-averaged F1-score of 0.860. The code to reproduce this work is available in GitHub.

## introduction hope is vital for human health, recovery, and restoration, according to health professionals. one of the goals of hope speech is to express the belief that someone can get motivated to move on in life to achieve the desired goals @xcite . positive vibes and hope content push human beings to take steps to create a better tomorrow through sustaining optimism and resilience during hardships. the advent of social media has enabled people from all over the world to connect with each other and to express their feelings or opinions in a positive, negative, or neutral manner @xcite @xcite @xcite . in social media, hope resides in positive and motivating content which helps to maintain healthy social media ecosystems. the recent advancements in social media have changed the lifestyle of many people and their daily life is extended with the virtual territory of the internet and social networks. social media platforms are influencing users' daily lives in a very large way. users may share positive vibes and hope or motivating content with the intention of positive suggestions for peace or to overcome situations like covid-19, war, election and etc., @xcite . several internet forums have also become popular for giving aid, advice, or support. further, when users are going through a difficult or unfavorable moment, in addition to seeking emotional support from family, friends and relatives, they may also knock the virtual platforms to get through the situation @xcite . the freedom and anonymity in social media have provided users an opportunity to share their opinions and comments without revealing their identity @xcite . this allows the users to share any kind of content including negative content such as abusive or hate speech and fake news. the majority of the social media analysis tasks deal with identifying the negative content such as hate speech, abusive language, fake news, etc. (chakravarthi, 2020b), with the aim of avoiding such content and having healthy social media. however, very few works have focused on social media analysis for positive vibes such as supportive, motivative, and hope content. in general, hope speech includes words of encouragement, motivation, promise, and advice @xcite . identifying such a content and promoting them in social media can be an alternative solution for having healthy and promising social media. in this direction, hopeedi 2 chakravarthi et al. (2022a) shared task calls the researchers to address the challenges of hope speech detection in youtube comments. the objective of the shared task is to classify the youtube comments in five languages, namely: english, spanish in latin scripts and tamil, kannada and malayalam in code-mixed native and roman scripts, into "hope", "not-hope" or "not-intended" categories. to tackle the challenges of hope speech detection in english and code-mixed kannada and malayalam texts, we -team mucic, present a methodology based on re-sampling the minority class ("hope" class) and using 1d convolutional neural network with long short-term memory (1d conv-lstm) for classification. the proposed methodology obtained first rank in the shared task for english texts with a weighted-averaged f1score of 0.860, while the same methodology for code-mixed malayalam and kannada texts did not perform well to our expectations. the rest of paper is organized as follows: section 2 gives a brief description of the best performing teams in (lt-edi-2021) 3 chakravarthi and muralidaran (2021) and section 3 presents the proposed methodology followed by the results in section 4. the paper concludes with the future work in section 5. ## related work researchers are attempting to create computational models to identify positive and supportive text on social media. despite the various machine learning (ml) and deep learning (dl) approaches for text classification (tc), transformers also have grown in prominence in recent years due to their ability to handle dependencies between input and output with both attention and recurrence. as a result, several natural language processing (nlp) tasks such as sentiment analysis, hope and hate speech detection etc., modeled as tc are using the transformer based models to achieve cutting-edge performance. this section presents a summary of the models submitted to hopeedi foot_3 shared task chakravarthi (2020b). a multilingual dataset of youtube comments in english, and code-mixed tamil and malayalam languages was released for public access. the dataset containing @xcite in english, tamil and malayalam languages respectively are distributed into two main categories, namely: "hope" and "not-hope" (and an extra category for texts in not-intended language). term frequency-inverse document frequency (tf-idf) features were used to train k-nearest neighbors (knn), support vector machines (svm), decision trees (dt), and logistic regression (lr) classifiers. among all the classifiers, dt classifier obtained a weighted-averaged f1-scores of 0.46, 0.51, and 0.56 for english, tamil, and malayalam texts respectively. balouchzahi et al. (2021) proposed a method that utilizes a combination of tf-idf vectors of words, char sequences, and syntactic n-grams to train: (i) a voting classifier of three estimators, namely: lr, extreme gradient boosting (xgb), and multi-layer perceptron (mlp) and (ii) keras neural network-based model. they also trained a bidirectional encoder representations from transformers (bert) language model from scratch using the given dataset and then used it for hope speech detection. for the voting classifier, the authors obtained 1 st , 2 nd , and 3 rd ranks with weightedaveraged f1-scores of 0.85, 0.92, and 0.59 for malayalam, english, and tamil texts respectively. @xcite preprocessed texts by removing punctuation symbols, emotions, and hashtags and then transliterated tamil and malayalam texts back to their native scripts. using multilingual bert (mbert) embedding as weights for convolutional neural network (cnn) classifier, they secured 1 st , 3 rd , and 4 th ranks for english, malayalam, and tamil texts. similarly, finetuning mbert for malayalam and tamil and using bert for english, @xcite obtained weighted-averaged f1-scores of 0.46, 0.81, 0.92 for tamil, malayalam, and english texts respectively. @xcite tried two different approaches to detect hope speech in the hopeedi dataset. in the first method they used contextual embeddings to train lr, random forest (rf), svm, and lstm classifiers. using a majority voting ensemble of bert, roberta, albert and lstm models in their second model, they obtained weighted-averaged f1-scores of 0.93, 0.75, and 0.49 for english, malayalam, and tamil texts respectively. in another transformer-based method m k and a p (2021), the authors used bidirectional long short-term memory (bilstm), universal language model fine-tuning (ulmfit), bert, al-bert, distilbert, roberta, and charbert for english and multilingual language versions of the mentioned transformers for tamil and malayalam. mbert for malayalam and multilingual distilbert for tamil obtained weighted-averaged f1-scores of 0.85 and 0.59 respectively. ulmfit model for english texts secured the 2 nd rank with a weightedaveraged f1-score of 0.92. emojis, punctuation marks, mentions, hashtags, etc., are removed from the dataset in the study conducted by @xcite . after cleaning the dataset, word2vec and fasttext were used to build the feature vectors which were fed to bilstm using an attention-based technique. this approach obtained an weighted-averaged f1-score of 0.73 and 9 th rank for malayalam dataset. ## methodology the proposed methodology consists of preprocessing and model construction steps to classify the given text into "hope", "not hope" or "not-intended" categories and the framework of the proposed methodology is shown in figure 1 . description of pre-processing and model construction steps are given below: ## experiments and results several experiments were conducted to classify a youtube comment into "hope", "not-hope" or "not-intended" categories for each language and the models that performed well on the development sets were applied to the test sets for the evaluation. ## conclusion this paper describes the models submitted to hopeedi shared task at acl 2022 to classify the youtube comments in english and code-mixed kannada and malayalam texts into "hope", "not-hope" or "not-intended" categories. the proposed methodology addresses the hope speech detection by using smote technique to resolve the data imbalance problem and 1d conv-lstm model for classification. for english texts, the proposed methodology performed the best and achieved 1 st rank with a m_f1-score of 0.550 but did not perform well for kannada and malayalam texts. as future work, we would like to extend our experiments on various feature types such as stylistic and psychological and explore different word embeddings along with language models.