Self-Contextualized Attention for Abusive Language Identification (2021)

Abstract: The use of attention mechanisms in deep learning approaches has become popular in natural language processing due to its outstanding performance. The use of these mechanisms allows one managing the importance of the elements of a sequence in accordance to their context, however, this importance has been observed independently between the pairs of elements of a sequence (self-attention) and between the application domain of a sequence (contextual attention), leading to the loss of relevant information and limiting the representation of the sequences. To tackle these particular issues we propose the self-contextualized attention mechanism, which trades off the previous limitations, by considering the internal and contextual relationships between the elements of a sequence. The proposed mechanism was evaluated in four standard collections for the abusive language identification task achieving encouraging results. It outperformed the current attention mechanisms and showed a competitive performance with respect to state-of-the-art approaches.

## introduction the integration of social media platforms into the everyday lives of billions of users has increased the number of online social interactions, promoting the exchange of different opinions and points of view that would otherwise be ignored by traditional media. the use of these social media platforms has revolutionized the way people communicate and share information. unfortunately, not all of these interactions are constructive, as the presence of abusive language (al) has spread to these media. al is characterized by the presence of insults, teasing, criticism and intimidation @xcite . mainly, it includes epithets directed at an individual's characteristic, which are personally offensive, degrading and insulting. because of its negative social impact @xcite , the automatic identification of al has stimulated the interest of social media companies and governments @xcite . derived from this, multiple efforts have been made to combat the proliferation of al, starting from the codes of conduct, norms and regulations in the content publication on social media foot_0 , to the use of natural language processing (nlp) for the computational analysis of language @xcite . concerning the several efforts and approximations made by the nlp community, one of the most relevant issues in the al identification task is to distinguish between the use of profane words and vulgarities in offensive and non-offensive texts. this indicates that the importance and interpretation of each word is highly context dependent, and, accordingly, this particular issue evidences one of the reasons why traditional bag-of-words methods tend to generate many false positives in their predictions. few works related to this task have explored the importance of words according to their context; particularly, the use of deep learning (dl) approaches with the addition of the attention mechanism (am) has been explored as an alternative to solve this issue @xcite @xcite . the idea behind the use of the am is to provide the classification model with the ability to focus on a subset of inputs (or features), handling in this way the importance of words in accordance to their context. due to their outstanding performance in many nlp tasks, several am have been proposed in recent years @xcite , which can be divided into two main approaches: self-attention (sa) @xcite and contextual attention (ca) @xcite mechanisms. specifically, sa takes the relationships among words within the same sentence, whereas, ca selectively focuses on words with respect to some external query vector, which adjusts according to the training task. the more important the word is in determining the answer to that query, the more focus it is given. despite their outstanding performance, both approaches have their own limitations. on one hand, ca ignores the internal relationships between the words of a sequence, correspondingly, sa does not consider the global relationships within the words of different sequences, which causes the loss of relevant information in the application domain (training task). clearly, the limitations of these am are complimentary and a hybrid am could overcome the individual issues. in this work we extend the use of the am by proposing the self-contextualized attention (sca) mechanism, an am that trades off the previous limitations, by taking advantage of both sa and ca mechanisms. the proposed sca mechanism is designed to be applied to any sequence of word encoding features, nevertheless, due to the high context-dependency of words that this specific task has, in this work we exclusively focus on the al identification task. the main contributions in this paper are: after identifying a deep neural network (dnn) architecture that is rather stable and well-performing, we propose and integrate the sca mechanism into the dl architecture, subsequently we conduct a quantitative and qualitative study of the effectiveness of our proposed am against the use of sa, ca and some other novel approaches to the al identification task. to the best of our knowledge this is the first effort in combining both am variants. this paper is organized as follows: in section 2, we present some previous works related to the al identification task, along with other hybrid am approaches. in section 3, we describe our proposed sca mechanism, as well as the employed classification framework; in section 4, we present the datasets used to evaluate our sca mechanism, their implementation details, as well as the external resources fed to the classification framework. section 5 reports and discusses our quantitative and qualitative results. finally, section 6 summarizes our findings and discusses future work. ## related work considering the well-acknowledged increase of al on social media platforms, several datasets @xcite @xcite and evaluation campaigns @xcite @xcite , have been proposed in order to mitigate the impact of such a kind of messages. the detection of al has been mainly addressed from a supervised perspective, considering a great variety of features. initial works used a combination of hand-crafted features such as bag-of-words representations, considering word and character ngrams @xcite , as well as, syntactical and linguistical features @xcite . aiming to improve the generalization of the classifiers, some other works have explored the use of dl by taking word or character sequences from texts to learn abusive patterns without the need for explicit feature engineering; the use of word embeddings as features predominates in these works @xcite @xcite . more recently, there has been a trend within the nlp community regarding the use of transformers for the improvement of text representations. in particular, for the identification of al, transfer learning has been applied considering different pre-trained models, such as elmo, gpt-2 and bert @xcite . regarding the classification stage, a vast range of approaches and techniques have also been proposed. these approaches could be divided into two main categories; the first category relies on traditional classification algorithms such as naive bayes, support vector machines (svm), logistic regression and random forest @xcite @xcite . on the other hand, the second category includes dl approaches, which rely on the use of convolutional neural networks (cnn) and recurrent neural networks (rnn), to accomplish the tasks of feature extraction @xcite and dependency learning @xcite . in addition to this, the combination of both types of neural networks have been used for the development of powerful structures that capture order information between the extracted features @xcite . finally, most recent works in abusive al identification have considered dl architectures with the addition of an am. one of the first works introducing attention into the task used the sa mechanism to detect abuse in portal news and wikipedia @xcite . subsequently, @xcite showed that the use of ca introduced by @xcite improved the results of sa in this task. later in @xcite the use of the ca is extended at a word n-grams level, showing the advantages in the usage of word sequences when identifying al. regarding other tasks outside the al identification, some hybrid ams have been proposed for the combination and representation of different instances and modalities @xcite , unlike these hybrid approaches, the proposed sca mechanism combines the features of the sa and ca mechanisms at an instance level. motivated by these previous works and with the goal of creating an am that handles both, the internal and external relationships between words, in this paper we propose the sca mechanism. ## self-contextualized attention this section is divided into two subsections. first we introduce our proposed sca mechanism, which is designed to be applied to any sequence of encoding features. subsequently, we present the dnn architecture used as our classification framework. for more details related to the ams, we refer the reader to the following work: @xcite . ## experimental settings this section presents the experimental settings. first, we introduce the four evaluation datasets, which correspond to twitter collections. then, with the purpose of facilitating the replicability of our results, we present our method's implementation details, starting from the text preprocessing phase, up to the configuration of the classification framework. ## experimental results this section is organized in three subsections. sections 5.1 and 5.2 present the quantitative results of the experimentation, corresponding to the comparison of our proposed sca mechanism against the baselines and state-of-the-art results. finally, section 5.3 presents some qualitative results of the sca mechanism, through the analysis and visualization of the attention values. ## conclusions and future work one of the main problems in the use of current ams is the loss of contextual or internal information between the elements of a sequence. to tackle this issue we proposed the sca mechanism, which integrates the sa and ca mechanisms for the construction of a representation that considers both, the internal and contextual relationships between the elements of a sequence. due to the highly context-dependent interpretation of words in the al identification, in this work we explore the use of the proposed sca mechanism in the al identification. the results obtained in four collections, considering different kinds of al, were encouraging; they improved state-of-the-art approaches in 2 out of 4 datasets. in addition to this, the sa and ca mechanisms were evaluated against the sca mechanism, the results show a quantitative and qualitative improvement in the use of the sca mechanism, which allowed concluding that the use of the sca mechanism is useful for discriminating between offensive and non-offensive contexts. since the most recent approaches are based on transformers, as future work we plan to explore the use of our proposed sca mechanism in the design of a multi-head sca architecture. additionally, we consider exploring new ways of combining the sa and ca mechanisms, as well as some novel approaches in the building of the sca mechanism without the need of computing the sa and ca mechanisms individually. finally, we consider the application of the proposed sca mechanism in other related tasks where the interpretation of words is highly context dependent such as the detection of deception or the detection of depressed social media users.