IDIAP _ TIET @ LT - EDI - ACL 2022 : Hope Speech Detection in Social Media using Contextualized BERT with Attention Mechanism (2022)

Abstract: With the increase of users on social media platforms, manipulating or provoking masses of people has become a piece of cake. This spread of hatred among people, which has become a loophole for freedom of speech, must be minimized. Hence, it is essential to have a system that automatically classifies the hatred content, especially on social media, to take it down. This paper presents a simple modular pipeline classifier with BERT embeddings and attention mechanism to classify hope speech content in the Hope Speech Detection shared task for Equality, Diversity, and Inclusion-ACL 2022. Our system submission ranks fourth with an F1-score of 0.84. We release our code-base here https://github.com/Deepanshu-beep/hope-speech-attention .

## introduction and related work social media today plays a vital role in spreading hatred and provoking people, which gives rise to hate-related crimes @xcite . various hate-related terror attacks usually have a history of hate-related content in their social media accounts. thus, large organizations such as facebook, youtube, twitter are working tirelessly to detect and bring down such hateful content from their platforms. since hate content must not be confused with freedom of speech and expression, thus it becomes quite challenging to reduce the number of false positives. previously, multiple experiments have been performed for hope speech detection, and there are various datasets available as well for it. the top-performing model presented in the hope speech detection shared task at the lt-edi-2021 workshop used the xlm-roberta language model @xcite , a combination of the xlm and roberta language model. further, they extracted the weighted output of the final layer of the xlm-roberta model using tf-idf to filter out the error that might cause due to the mixing of various languages supported by the model. @xcite presented a transformer-based ensembled architecture consisting of a bert pre-trained model and a language identification model. the language identification model was used to detect if the input isn't in english. on the other hand, the bert language model was just responsible for the binary classification of hope speech. @xcite presented a simple classification model which initially created the static bert @xcite embeddings matrix of the data to extract the contextual information of the data and then experimented with various deep neural networks (dnn) to train a binary classifier. seeing the dominance of transformers to solve multiple complex applications in natural language processing (nlp) became the motivation for our model. hence, we encode the data using the contextualized bert embeddings and train an attention network. though it is a simple architecture with relatively few parameters, it performs efficiently and can be verified through the results. ## experimental setup in this section, we give a detailed explanation of the experimental setup for the proposed model and depict the summarized view of it in figure 1 . ## experiment results and analysis we test our model for the english dataset of the hopeedi database. the classification report for our proposed and the top-performing model over the test set can be seen in ## conclusion in this paper, we described the shared task submission on hope speech detection for english dataset. we propose a pipeline classifier architecture that uses an attention mechanism over the contextualized bert embeddings. for future work, we intend to work upon other languages of the hopeedi database and experiment with different neural network architectures combined with contextual embeddings.