T eam UNCC @ LT - EDI - EACL 2021: Hope Speech Detection using Transfer Learning with Transformers (2021)

Abstract: In this paper, we describe our approach towards utilizing pre-trained models for the task of hope speech detection. We participated in Task 2: Hope Speech Detection for Equality, Diversity and Inclusion at LT-EDI-2021 @ EACL2021. The goal of this task is to predict the presence of hope speech, along with the presence of samples that do not belong to the same language in the dataset. We describe our approach to fine-tuning RoBERTa for Hope Speech detection in English and our approach to fine-tuning XLM-RoBERTa for Hope Speech detection in Tamil and Malayalam, two low resource Indic languages. We demonstrate the performance of our approach on classifying text into hope-speech, non-hope and not-language. Our approach ranked 1st in English (F1 = 0.93), 1st in Tamil (F1 = 0.61) and 3rd in Malayalam (F1 = 0.83).

## introduction hate speech and the need for its moderation on social media platforms has recently become the focus of research @xcite . the need for detection of hate speech is clear, and worthy of the efforts. however, detecting and removing hate speech, while providing important benefits to conversations online, should be supplemented by other efforts to improve human connection and communication. there is a clear need to find common themes in community and build bridges to reduce the recent wave of polarization which has gripped many social media platforms @xcite . there has been extensive work including proposed tasks and datasets on hate speech detection online. there have been multiple semeval foot_1 tasks to detect hate speech, including binary (2 classes, @xcite ), and multi-label (>2 classes with overlaps @xcite ) annotations. hate speech detection has also further been expanded to cover explainability for the approaches used by @xcite , who propose ha-texplain in which the span of text constituting the hate speech must also be detected. however, hate speech is just one facet of human behavior, especially on social media @xcite @xcite . there are equally interesting studies on prosocial behaviors online including solidarity @xcite ) and altruism @xcite , as well as hope speech. the hope speech detection for equality, diversity and inclusion task at lt-edi-2021 @ eacl2021 (chakravarthi and muralidaran, 2021) marks a step towards helping push for more positive, uplifting speech on social media. exacerbated by the pandemic, and the subsequent need of communication to move to online communities, research in the detection of "hope speech", which promotes positive, uplifting discussion to build support, is an important step @xcite . findings from this effort could have an overall positive effect in the real world. the task mainly focuses on multilingual classification for identifying speech associated with promise, potential, support, reassurance, suggestions or inspiration provided to participants by their peers during periods of illness, stress, loneliness and depression @xcite . in addition to being a multi-class problem, each language (english, tamil and malayalam) also had differing amounts of class imbalance. we describe our approach in detail in section 2 and present our results in section 3. ## method roberta is an improved bert @xcite model by facebook ai which achieves stateof-the-art results on several natural language understanding (nlu) tasks including glue @xcite and squad @xcite . roberta is improved through training bert for a longer duration on longer sequences, increasing the data quantity, removing the sentence prediction objective during pre-training, and changing the masking pattern applied during pre-training. with the aim of improving cross-lingual language understanding (xlu), xlm-roberta was developed using a transformer-based masked language model (mlm). xlm-roberta was pre-trained using 2 terabytes of commoncrawl data @xcite containing one hundred languages. xlm-roberta outperforms its multilingual mlms mbert @xcite and xlm (lample and @xcite . we thus do not propose a novel system, and instead rely on fine-tuning these two large transformers for the task of hope speech detection. we use the roberta transformer @xcite for english. we select xlm-roberta @xcite for tamil and malayalam tasks due to its robustness and pre-training on low-resources languages including tamil and malayalam. for our implementation, we use the simple transformers 3 library which is built upon the transformers library by huggingface @xcite . we use adam @xcite as our optimizer. our hyperparameters are presented in we present detailed results for each language below. we also compare with baselines provided in the original dataset paper (chakravarthi, 2020), and 3 https://simpletransformers.ai/ include them in the results in tables 2 , 3 , and 4. we report results to 2 significant digits, since the baseline provided by the task is also limited to 2 significant digits. we also provide the dataset distribution in these tables, in the "support" column. we discuss our findings further in section 4. evaluation for english. ## qualitative evaluation in this section, we analytically go through sample records in the english test set to better understand the dataset, and to evaluate the strengths and weaknesses of our approach. analytical evaluation. we use sentencetransformers @xcite to further evaluate our approach. sentence transformers were created by utilizing word-level representation models such as bert and roberta for better downstream computational performance on sentence-level tasks, since utilizing wordlevel representations for tasks such as determining the similarity of 2 sentences takes much longer than computing sentence-level representations. we use the paraphrase-xlm-rmultilingual-v1 pretrained model to get our sentence embeddings, and then use k-means clustering using scikit-learn @xcite to cluster the test set into 3 clusters, the same number of classes as the classification task. we provide clustering results, showing cluster assignments of data samples and whether our classifiers labelled them correctly or not in figure 1 . for english (figure 1a ), there are fewer misclassified samples, but most of them lie in cluster 1, shown on the bottom right of the umap scatter plot 1. this cluster, upon further observation, mostly consists of youtube comments related to the black lives matter (blm) movement. we conclude that the our approach specifically struggles with correctly classifying such utterances, and elaborate with illustrative examples in the next section. for tamil (figure 1b ), we observe closely clustered data points, a property that is also reflected in our classification results as seen by the misclassified points (shown by square markers in the scatter plot). we see a similar trend as english in malayalam (figure 1c ), however an observational analysis is not possible for malayalam and tamil since we do not speak the language, and wish not to rely on automatic translation systems for informal, code-mixed text (chakravarthi, 2020). observations. in ## conclusion we found that fine-tuning roberta (english) and xlm-roberta (tamil and malayalam) for classification performed well. providing class weights helped handle the data imbalance, leading to a balanced performance over the baselines. we submitted the results of our approach using the roberta and xlm-roberta models without class weights as our final submission for the hope speech detection task. admittedly, our approach does not break new ground in terms of novelty of models or architecture; we instead rely upon fine-tuning pre-trained models. however, we present it as a first step in this direction, and aim to build on our success in future iterations. the hope speech detection task was an important and necessary step towards promoting positive speech online. our goal is to keep participating in the effort to detect and promote more positive speech online. additionally, we hope to extend this task to covid19 data, and work towards understanding community support during the pandemic.