ZYJ @ LT - EDI - EACL 2021: XLM - R o BERT a-Based Model with Attention for Hope Speech Detection (2021)

Abstract: Due to the development of modern computer technology and the increase in the number of online media users, we can see all kinds of posts and comments everywhere on the internet. Hope speech can not only inspire the creators but also make other viewers pleasant. It is necessary to effectively and automatically detect hope speech. This paper describes the approach of our team in the task of hope speech detection. We use the attention mechanism to adjust the weight of all the output layers of XLM-RoBERTa to make full use of the information extracted from each layer, and use the weighted sum of all the output layers to complete the classification task. And we use the Stratified-K-Fold method to enhance the training data set. We achieve a weighted average F1-score of 0.59, 0.84, and 0.92 for Tamil, Malayalam, and English language, ranked 3rd, 2nd, and 2nd.

## introduction with the development of network media, people can express their opinions on the internet at any time, among which there will be some hope speech. hope speech will encourage people to stand firm in their beliefs and follow the path of their goals. at the same time, comments or posts containing hope speech can often contain factors such as equality, diversity, and inclusion. they can also provide support, reassurance, suggestions, inspiration, and insight, etc., which can change people's emotions from negative to positive. the proportion of comments and posts on media platforms that contain hopeful statements can reflect the current emotional tendencies of online communities. as the proportion of words containing hope increases, the corresponding posts conveying negative emotions such as pessimism and despair will decrease, thus promoting social harmony. because the net-work media platform needs to support multiple languages and faces users with different backgrounds, the multi-lingual, multi-modal, and non-standard writing style of online media posts makes it a very challenging task to effectively detect whether the comment or post belongs to the hope speech. our team participated the shared task on hope speech detection for equality, diversity, and inclusion at @xcite @xcite . the goal of this task is to identify whether a given comment contains hope speech or not. this is a comment/post level classification task. the task is divided into three sub-tasks based on three languages, namely english, tamil, and malayalam. given a youtube comment, the systems submitted by the participants should classify it into 'hope speech', 'not hope speech' and 'not in intended language'. in this task, we use xlm-roberta as the base model to extract features and combine 12 layers of output with an embedding layer to do the classification task. in order to train the weight of each layer, we add an attention mechanism for the model. during the training, we process the training data through the stratified-k-fold method, each data set is trained to get the different weights, and finally, the best-predicted value is obtained through a voting mechanism. the rest of our paper is structured as follows. section 2 describes the related work. methods are described in section 3. experiments and results are described in section 4. the conclusion is drawn in section 5. ## related work while hope speech detection is a relatively new task, it can be categorized as a problem of sentiment analysis @xcite ) and text classification. a great deal of work has been done by many researchers and practitioners from in-dustry and academia. @xcite proposed a simple unsupervised learning @xcite algorithm for classifying reviews as recommended or not recommended. basiri et al. ( 2020 )proposed an attention-based bidirectional cnn-rnn deep model (abcdm) for polarity classification of long comments and short tweets, two independent bidirectional lstm and gru layers are used to extract past and future context by considering temporal information flow in both directions. chen and ke proposed a convolutional neural network-regional long short-term memory (cnn-rlstm) that combines cnn and regional lstm. at the sentence level @xcite , the emotional information of the whole sentence is retained through cnn network, and through regional lstm effectively distinguish between different target emotional polarity. chen et al. proposed a multi-head attentionmemory network (mnhma) based on hierarchy for aspect-based sentiment analysis, which makes full use of the long-term semantic relationship of a given aspect term @xcite in a sentence to reduce the loss of aspect information. ## 2e-5