H urt BERT : Incorporating Lexical Features with BERT for the Detection of Abusive Language (2020)

Abstract: The detection of abusive or offensive remarks in social texts has received significant attention in research. In several related shared tasks, BERT has been shown to be the state-of-the-art. In this paper, we propose to utilize lexical features derived from a hate lexicon towards improving the performance of BERT in such tasks. We explore different ways to utilize the lexical features in the form of lexicon-based encodings at the sentence level or embeddings at the word level. We provide an extensive dataset evaluation that addresses in-domain as well as cross-domain detection of abusive content to render a complete picture. Our results indicate that our proposed models combining BERT with lexical features help improve over a baseline BERT model in many of our in-domain and cross-domain experiments.

## introduction the automatic classification of abusive and offensive language is a complex problem, that has raised a growing interest in the natural language processing community in the last decade or so @xcite @xcite . several benchmarks have been introduced to measure the performance of mostly supervised machine learning systems tackling such problems as text classification tasks @xcite . the evaluation of abusive and offensive language, however, is not straightforward. among the issues, it has been observed how the topics discussed in the messages composing the benchmark datasets introduce biases, interfering with the modeling of the pure pragmatic phenomena by the supervised models trained on the respective training sets @xcite . among the recent neural architectures, bert (bidirectional encoder representations from trans-formers @xcite ), is considered the state of the art in several nlp tasks, including abusive and offensive language detection. for example, in the semeval 2019 task 6 @xcite , seven out of the top-ten teams used bert, including the top team. the knowledge encoded in such model, based on transformer neural networks, is induced by a pre-training performed on a large quantity of text, then fine-tuned to a specific dataset in order to learn complex correlations between the natural language and the labels. one disadvantage to models such as bert is that no additional external knowledge is taken into consideration, such as linguistic information from a lexicon. in this paper, we propose a hybrid methodology to infuse external knowledge into a supervised model for abusive language detection. we propose to add extra lexical features with bert at sentenceor term-level, with the goal of improving the quality of its prediction of abusive language. in particular, we investigate the inclusion of features from a categorized lexicon in the domain of offensive and abusive language, with the aim of supporting transfer knowledge in that domain across datasets. we perform extensive, in-domain and crossdomain experimentation, to evaluate the performance of models which are trained on one dataset and tested on other datasets. cross-domain classification of abusive content has been proposed to address the diverse topical focuses and targets as exhibited in different datasets developed from the research community in the last years @xcite @xcite . for example, some datasets proposed for hate speech detection focus on racism or sexism @xcite , in line with the target-oriented nature of hate speech, while others on offensive or abusive language without tar-geting a specific vulnerable group @xcite . this makes it difficult to know if a model that performs well on one dataset will generalize well for other datasets. however, several actors -including institutions, ngo operators and ict companies to comply to governments' demands for counteracting online abuse 1have an increasing need for automatic support to moderation @xcite or for monitoring and mapping the dynamics and the diffusion of hate speech dynamics over a territory @xcite considering different targets and vulnerable categories. in this scenario, there is a considerable urgency to investigate computational approaches for abusive language detection supporting the development of robust models, which can be used to detect abusive contents with different scope or topical focuses. when addressing this challenge, the motivation for our proposal is the hypothesis that the addition of lexical knowledge from an abusive lexicon will soften the topic bias issue @xcite , making the model more stable against cross-domain evaluation. our extensive experimentation with many different datasets shows that our proposed methods improve over the bert baseline in the majority of the in-domain and cross-domain experiments. ## related work the last ten years have seen a rapidly increasing amount of research work on the automatic detection of abusive and offensive language, as highlighted by the success of international evaluation campaigns such as hateval @xcite on gender-or ethnic-based hate speech, offenseval @xcite on offensive language, or ami @xcite , automatic misogyny identification) on misogyny. several annotated corpora have also been established as benchmarks besides the data produced for the aforementioned shared tasks for several languages, for instance, @xcite for racism and sexism in english, @xcite for hate speech italian and @xcite for abusive language in arabic. we refer to @xcite for a systematic and updated review of resources and benchmark corpora for hate speech 1 see for instance the code of conduct on countering illegal hate speech online issued by eu commission @xcite . the vast majority of approaches proposed in the literature are based on supervised learning, with statistical models learning the features of the target language and their relationship with the abusive phenomena from an annotated corpus. most works propose variations on neural architectures such as recurrent neural networks (especially long shortterm memory networks), or convolutional neural networks @xcite ). an investigation on what type of attention mechanism (contextual vs. self-attention) is better for abusive language detection using deep learning architectures is proposed in @xcite . character-based models have also been proposed for this task @xcite . more recently, models based on the transformer neural network architecture have gained prominence, thanks to their ability of learning accurate language models from very large corpora in an unsupervised fashion, and then being fine-tuned to specific classification tasks, such as abusive language detection, with relatively little amount of annotated data. several ideas have been proposed in the literature to improve the performance of bert for abusive language detection. for example, finetuning large pre-trained language models in @xcite . a complementary approach to supervised learning towards the detection of abusive and offensive language is the use of language resources such as lexicons and dictionaries. @xcite proposed a method to induce a list of english words to capture abusive language. @xcite introduced an english lexicon covering hate speech, racism, sexism, and homophobia. other languages have relatively less resources with respect to english, apart perhaps from arabic, for which two lexical resources are available, by @xcite , with focus on obscenity, and by @xcite . a notable exception is hurtlex @xcite , a multilingual lexicon of offensive words, created by semi-automatically translating a handcrafted resource in italian by linguist tullio de mauro (called parole per ferire, "words to hurt" @xcite ) into 53 languages. lemmas in hurtlex are associated to 17 non-mutually exclusive categories, plus a binary macro-category indicating whether the lemma reflects a stereotype. the number of lemmas in any language of hurtlex is in the order of thousands, depending on the lan-guage, and they are divided into the four principal parts of speech: noun, adjective, verb, and adverb. in our earlier research, we used a technique called retrofitting to enhance word embeddings using hurtlex, for detecting aggression in english, hindi, and bengali @xcite . in this work, we propose to infuse the lexical knowledge from hurtlex into a bert model with the goal to improve the efficacy of abusive and offensive language prediction models. specifically, we utilize different representations of the hurtlex categories as they are found in the data, utilize them with a bert model, and explore how they affect the detection accuracy. to the best of our knowledge, the utilization of a hate lexicon, especially one that is based on this kind of structure with multiple categories, with a bert model has not been explored before. we fully describe our methods in the following section. ## methodology in this paper, we explore two models based on how they utilize the lexical features extracted from the hate speech lexicon, hurtlex @xcite . both of our proposed models utilize two inputs: (a) the sentence tokens (bert's usual input), and (b) a vector we create based on the categories in hurtlex as they are found in our data. all the data we explore in this work are in english, so we used only the english version of hurtlex and leave the multilingual aspect for future work. specifically, we use the english section of hurtlex version 1.2 2 . it contains 6,072 entries, of which 2,268 are in the conservative subset (these are terms with higher confidence). our models both start using the bert layer, which takes three inputs consisting of id, mask and segment -see figures 1 and 2 . the output of this bert layer connects to a dense layer. please note that specific details and parameters for the bert baseline as well as any layers in our models are presented in section 4. regarding hurtlex features, we have two ways of extracting features: encodings and embeddings. in the first architecture (see figure 1 ), based on the words in the train set, we find their categories in hurtlex and then derive a vector of hurtlex cate-2 https://github.com/valeriobasile/ hurtlex figure 2: hurtbert-emb, our model using hurtlex embeddings our second model explores using hurtlex embeddings with an lstm, as shown in figure 2 . the hurtlex embedding is a 17-dimension one-hot encoding of the word presence in each of the lexicon categories. this model is named hurtbert-emb. one of the main differences between the embedding and the encoding is that the embedding is word-level, while the encoding is commentlevel. therefore, for the case of hurtlex encodings, one record (e.g. one tweet) generates one 17-dimensional vector (which we call hurtlex encoding). while, for the case of the hurtlex embeddings, every word in the comment has one 17dimensional vector representation (which is the hurtlex embedding). the hurtlex embedding also passes through an embedding layer, which goes into an lstm and a dense layer, as shown in figure 2 . in the end, the encoding is a simple representation that reflects if a category from the lexicon is found in the words of comment (or more accurately, how many times this category is found). while the embedding-based model also represents non-linear interactions between the features, that is, linguistically, the role of the hurtlex words in the sentence. finally, for both models, we concatenate the dense layer from the bert output and the dense layer from the hurtlex output, before passing into a dense layer with sigmoid activation as the predictor layer (see the bottom part of both figures 1 and 2 ). ## experiments and results overall, we follow the experimental setup of @xcite . we exploit the bert pre-trained models available on tensorflow-hub foot_0 , which facilitate us to integrate bert on top of keras architecture foot_1 . specifically, we use the bert-uncased model, with 12 transformer blocks, 12 self-attention heads, and hidden layer dimension 768. based on performance in early experiments, our models use learning rate of e -5, batch size 32, and maximum sequence length of 50. we implement early stopping and model checkpoint based on the development set evaluation to avoid overfitting during the training process. for the lstm in figure 2 , we use 32 nodes, and the dense layers in figures 1 2 are 256 and 16 nodes respectively (all dense layers except last have relu activation). ## conclusions and future work in this work, we explore how to combine a bert model with features extracted from a hate speech lexicon. the lexical features are extracted based on multiple categories in the lexicon and according to how these categories are found in the data. the lexical features can be represented at the comment level as simple numerical encodings or at the word level as embeddings that aim to learn the relationships of the lexical features in the context of the data. we conduct extensive experimentation, with in-domain as well as cross-domain training. we observe that our methods improve on the bert baseline in the large majority of the cases, with high gains in some cases. it proves our hypothesis that the additional features from lexical knowledge can improve the bert performance, providing a domain-agnostic feature in a cross-domain setting. for our future work, we will explore different languages to take advantage of the multilingual aspect of our lexicon. we also plan to delve deeper into the study of the relationships between our models and the linguistic aspects and phenomena in the various abusive and offensive datasets.