TEAM HUB @ LT - EDI - EACL 2021: Hope Speech Detection Based On Pre-trained Language Model (2021)

Abstract: This article introduces the system description of TEAM_HUB team participating in LT-EDI 2021: Hope Speech Detection. This shared task is the first task related to the desired voice detection. The data set in the shared task consists of three different languages (English, Tamil, and Malayalam). The task type is text classification. Based on the analysis and understanding of the task description and data set, we designed a system based on a pre-trained language model to complete this shared task. In this system, we use methods and models that combine the XLM-RoBERTa pre-trained language model and the Tf-Idf algorithm. In the final result ranking announced by the task organizer, our system obtained F1 scores of 0.93, 0.84, 0.59 on the English dataset, Malayalam dataset, and Tamil dataset. Our submission results are ranked 1, 2, and 3 respectively.

## introduction and background according to relevant statistics, as of 2020, the number of people active in social media worldwide has reached 3.6 billion. based on the analysis of the current growth rate, in the next 5 years, the total number of social media users will be 1.3 times the current number @xcite . the covid-19 virus began to spread globally in 2019, and so far, more than 200 countries and regions have been affected by the epidemic to varying degrees. the number of fake news and negative comments circulating on online social media is 9 times the usual @xcite . in the past few years, eliminating abusive and hate speech in online social media comments has been widely used on various social media platforms. most of these elimination systems are developed for languages with richer corpus resources, such as chinese, spanish, english, etc. @xcite @xcite . there is not much work on hope speech detection for example, chakravarthi et al. developed a data set of hope speech to complete the recognition of hope speech in online social media @xcite . the task goal proposed by the task organizer team of "shared task on hope speech detection for equality, diversity and inclusion" is to identify the content of posts related to hope speech, not hope speech, and not in intended language from the dataset of hope speech obtained on youtube @xcite . so this sharing task is a comment/post level classification task. also, this is the first task of identifying hopeful speeches for equality, diversity, and inclusiveness in a multilingual environment. the annotation data set we get contains three different languages (english, malayalam, and tamil). in recent studies in the field of natural language processing(nlp), the pre-trained language model can achieve state-of-the-art results in many nlp tasks @xcite . also, combined with the task description and the multilingual characteristics of the data set, in terms of model and method selection, we chose to use fine-tuned xlm-roberta @xcite to complete our task. we have not done much work on data preprocessing. our work focuses mainly on how to reduce the impact of multi-language and mixed code on the results. this is also a difficult problem that needs to be solved in this shared task. therefore, in the method, we consider combining the semantic information of term frequency-inverse document frequency (tf-idf) and xlm-roberta. then use inception blocks @xcite as the shared layer to learn the output information of tf-idf and xlm-roberta. then use this combined model to train on the training set of this task to get the trained weights. finally, the trained weights are used to predict the result of the test set. however, combined with our previous analysis, it is not difficult to know that just identifying negative content in social media is not complete. because not only do we need to avoid negative information in our lives, we also need more positive and hopeful information @xcite . some recent studies and reports have also revealed to us the importance of hope and the impact of hopelessness @xcite @xcite . there is not much research and work on hope speech detection, but the recent research and results of hope speech detection by palakodety et al. have revealed to us that hope speech detection has begun to attract the attention of the academic community @xcite . chakravarthi et al. used support vector machine (svm), naive bayesian (nb), k-nearest neighbors (knn), decision tree (dt), logistic regression (lr), and other methods on the hope speech data set to get good results (chakravarthi, 2020). combine their work on the hope speech data set and the code-mixing in the data set in this task. we chose a pre-trained language model(xlm-roberta) that is currently widely used in the field of natural language processing and fine-tuned it to complete the task. ## conclusion in this task, our team used the xlm-roberta model with cross-language capabilities as the basis and combined it with the tf-idf algorithm to complete this task. our experimental results confirm the effectiveness of our method, and at the same time find the problem of our method and parameter settings on the tamil language task data set. our work in the data preprocessing part is not detailed enough. we ignore the influence of stop words, emoticons, and other special symbols on the results. this is a flaw in our work. in future work, we will make up for the shortcomings of our work in this task and improve our methods. besides, we will continue to pay attention to the progress of related work on low-resource language communities, and the progress of hope speeches in the field of natural language processing.