EDIO ne@ LT - EDI - EACL 2021: Pre-trained Transformers with Convolutional Neural Networks for Hope Speech Detection. (2021)

Abstract: Hope is an essential aspect of mental health stability and recovery in every individual in this fast-changing world. Any tools and methods developed for detection, analysis, and generation of hope speech will be beneficial. In this paper, we propose a model on hope-speech detection to automatically detect web content that may play a positive role in diffusing hostility on social media. We perform the experiments by taking advantage of pre-processing and transfer-learning models. We observed that the pre-trained multilingual-BERT model with convolution neural networks gave the best results. Our model ranked first, third, and fourth ranks on English, Malayalam-English, and Tamil-English code-mixed datasets.

## introduction nowadays, people often use social media websites to share their views and thoughts. the thoughts might be positive or negative. much work has been done towards identifying the negative thoughts, i.e., hate speech and offensive content identification on social media @xcite . research is now shifting to the field of analyzing positivity via hope speech detection on social media. hope is a positive state of mind, an expectation of positive outcomes concerning events and circumstances in one's life @xcite . hope drives an individual to move forward. hope can be a useful tool for each individual to maintain a stable and optimistic attitude towards life. in a multilingual society, people usually express their thoughts by mixing two or more languages in a single utterance. this form of language contact is known as code-mixing @xcite . code-mixed data is real-world unprocessed data that has non-standard variations of spelling and does not follow a grammatical structure @xcite . any automated hope-speech detection tool will face challenges in this aspect. so the analysis of code-mixed hope speech detection is necessary to handle the real-time social media data. the european association of computational linguistics 2021 has organized a language technology workshop for equality, diversity, and inclusion (chakravarthi and muralidaran, 2021) with a shared task to cultivate positivity and promote research on code-mixed hope speech data. the goal of this task is to identify whether a given comment contains hope-speech or not. in this work, we address the issue of hope-speech detection on code-mixed dravidian youtube comments. this paper presents a pre-trained multilingual bert encoder with cnn as a classifier for the hope speech classification data. the paper is organized as follows. section 2 provides related work on hope speech detection. section 3 provides information on the task and datasets section 4 describes the proposed work. section 5 presents the experimental setup and the performance of the model. section 6 concludes our work. ## related work hope speech detection is a novel topic with a significantly limited amount of research done in this field. @xcite propose a novel task to automatically detect hope speech on web content that may play a positive role in diffusing hostility on social media triggered by heightened political tensions during a conflict between two nuclear power nations. ## task description we need to identify the hope in the code-mixed english, tamil-english, and malayalam-english youtube comments for hope speech detection. for the english language, data was collected related to the following issues, including women in stem, lgbtiq issues, covid-19 pandemic, racism, and black lives matters, united kingdom (uk) versus china, united states of america (usa) versus china, and australia versus china from youtube video comments. for tamil and malayalam, we collected data from india on the recent topics regarding lgbtiq issues, covid-19, women in stem, the indo-china border dispute. each comment or post is annotated with hopespeech or non-hope-speech and not-tamil/not-malayalam labels. the dataset is divided into train, development, and test sets for the given hope speech task. the details of the dataset are given in the ## implementation we give the input text to the pre-trained multilingual bert model, and the pre-trained bert model gives encoded information as output. now we need to learn a classification model for the given encoded information. we pass the encoded information as embeddings to the above cnn classifier. the classifier applies convolutions, max-pooling, and finally, a feed-forward network for classification with dropout. ## experiments the section presents the baselines, hyper-parameter settings, and analysis of observed results. the baselines used for the proposed work is, 1. svm with tf-idf and sub-word level tokenization: this baseline uses term frequency and inverse document frequency-based vectorization @xcite for feature representation and the support vector machine @xcite to classify the data. the code-mixed data contains transliterated text with a non-standard representation of words. we used sub-word-level tokenization to extract the tokens to capture a better sentence representation. note: we consider two formats, processed and original data for comparing the baselines, where "processed" refers to applying the pre-processing method on the original data. ## conclusion in this paper, we addressed hope speech identification in english and dravidian code-mixed languages. we used pre-trained multilingual bidirectional encoder representations to obtain the word embeddings, and we used convolutional neural networks for classification. we compared the method with other baselines. the results showed that using back-transliteration helped the model capture the pre-trained word embeddings based on the native script. cnn helped the model extract feature representations better than feed-forward networks, which increased the model's performance. in the future, we will work on cross-lingual meta word embeddings to handle the multilingual scenario in the code-mixed datasets.