f BERT : A Neural Transformer for Identifying Offensive Content (2021)

Abstract: Transformer-based models such as BERT, XLNET, and XLM-R have achieved state-of-the-art performance across various NLP tasks including the identification of offensive language and hate speech, an important problem in social media. In this paper, we present fBERT, a BERT model retrained on SOLID, the largest English offensive language identification corpus available with over 1.4 million offensive instances. We evaluate fBERT’s performance on identifying offensive content on multiple English datasets and we test several thresholds for selecting instances from SOLID. The fBERT model will be made freely available to the community.

## introduction to cope with the spread of offensive content and hate speech online, researchers have worked to develop automatic methods to detect such posts automatically. early efforts included methods that used various linguistic features in tandem with linear classifiers @xcite while, more recently, deep neural networks (dnns) @xcite , transfer learning @xcite , and pre-trained language models @xcite @xcite have led to even further advances. as evidenced in recent competitions, the performance of these models varies with the sub-task that they are designed to address as well as the datasets used to train them. for example, classical statistical learning models such as the support vector machine (svm) have outperformed neural transformers in hate speech detection at hat-eval 2019 @xcite and in aggression detection at trac 2018 @xcite . however, for both of these tasks in offenseval 2019 and 2020 @xcite , which focused on the identification of more general offensive language identification, pre-trained transformer-based models such as bert @xcite outperformed other neural architectures and statistical learning methods. the introduction of representations learned through the bidirectional encoding inherent to neural transformers (bert) @xcite has been driven much progress in areas in nlp such as language understanding, named entity recognition, and text classification. the base model is pretrained on a large english corpus, e.g., wikipedia, bookcorpus @xcite , using unsupervised masked language modeling and next sentence prediction objectives to adjust the model weights. various other transformer-based models have also been introduced including roberta @xcite , xlnet @xcite , xlm-r @xcite . all of these models, however, are trained on general-purpose corpora for better language understanding, generally lacking domainspecific knowledge. to cope with this limitation, more recently, domain-specific models have been trained and/or fine tuned to different domains such as finance (finbert) @xcite , law (legal-bert) @xcite , scientific texts (scibert) @xcite , and microblogging (bertweet) @xcite . @xcite recently released hate-bert, a bert transformer model for abusive language detection trained on the reddit abusive language english dataset (ral-e). hatebert achieves competitive performance on a few benchmark datasets but relies heavily on manually annotated labels. moreover, hatebert was trained on a task-specific dataset (aggression) instead of a more general dataset that encompasses multiple types of offensive language (e.g. hate speech, cyberbullying, profanity) like the popular olid @xcite used in offfenseval 2019 at semeval. to address this gap, in this study, we present fbert, a pre-trained bert model trained on solid @xcite , a recently released large dataset crated using olid's general annotation model but using semi-supervised learning instead of manually annotated labels. solid contains over 1.4 million english tweets with offensive scores greater than 0.5. we show that the proposed fbert outperforms both a plain bert implementation and hatebert on various offensive and hate speech detection tasks. the contributions of this paper are as follows: ## related work the use of large pre-trained transformer models has become widespread in nlp. this includes several recently developed offensive language identification systems based on transformer architectures such as bert @xcite . these systems have achieved top performance in popular competitions such as hasoc 2019 @xcite @xcite , offenseval 2019 and 2020 @xcite , and trac 2020 @xcite . the great performance obtained by these systems provides further evidence that pre-trained transformer models are a good fit for the kind of semantic understanding required when identifying offensive content online. most of the top systems submitted to the aforementioned competitions @xcite @xcite , however, use models pre-trained on standard contemporary texts. user generated content and offensive language online, however, contain its own set of distinctive features that models trained on standard texts may fail to represent. therefore, fine-tuning pre-trained models to this challenging domain is a promising but under explored research direction. to the best of our knowledge, a recent first attempt to fine-tune a bert model to deal with offensive language online, hatebert @xcite , shows promising results for english on multiple datasets. in this paper, we address some of the limitations of hatebert, discussed in the introduction of this paper, and present fbert, a new bert-based offensive language model made freely available to the research community. ## data the limited size of existing datasets has been a bottleneck for offensive language identification. olid @xcite , the dataset used in offen-seval 2019 and arguably the most popular dataset for this task, contains only 14, 100 tweets. olid is annotated using a hierarchical annotation taxonomy and, as a result, only a sub-set of the corpus is annotated in the lower levels of the taxonomy, i.e., only a few hundred instances. more recently, following the olid taxonomy, @xcite released a large-scale offensive language identification dataset (solid) with over 9 million english tweets. the data is collected using the twitter streaming api. the annotations include labels learned using semi-supervised methods. one important difference between solid and olid is that solid is collected using random seeds, which has been shown to decrease topic bias compared to the target keywords used in olid. all the usernames and urls are replaced with placeholders and tweets with less than two words or 18 characters were discarded. for retraining fbert, we have selected over 1.4 million offensive instances from solid. we considered multiple offensive score thresholds from the solid dataset including all instances with scores between 0.5 and 1.0 arranged in five bins with 0.1 increments. the number of instances in each range is available in ## experiments to determine the effectiveness and portability of the trained fbert, we conducted a series of experiments using benchmark datasets and compared our model with a general-purpose bert model. we used the same set of configurations for all the datasets evaluated in order to ensure consistency between all the experiments. this also provides a good starting configuration for researchers who intend to use fbert on a new dataset. we used a batch-size of eight, adam optimiser with learning rate 1e-4, and a linear learning rate warm-up over 10% of the training data. during the training process, the parameters of the transformer model, as well as the parameters of the subsequent layers, were updated. the models were trained using only training data. furthermore, they were evaluated while training using an evaluation set that had one fifth of the rows in training data. we performed early stopping if the evaluation loss did not improve over ten evaluation steps. all the models were trained for three epochs. the rest of the parameters are shown in ## results we first present the results for the solid data selection thresholds in we then compare the performance of fbert with bert and hatebert. in the hateval sub-task a, we see that fbert has outperformed bert by increasing the test macro f 1 score by over 23%. this empirically demonstrates the advantage and generalization power of our domain-specific retrained bert model. the best model @xcite in this task used an svm model with a radial basis kernel, exploiting sentence embeddings from google's universal sentence encoder as features. the results are shown in finally, as observed in the experimental results presented above, we observe that fbert has outperformed the abusive language hatebert model in all of the experiments. the proposed fbert has also performed efficiently in all the aggression detection tasks. this validates the effectiveness of the proposed domain-specific transformer model for offensive and hateful language classification tasks. the proposed fbert model is effective across different datasets and objectives, providing a powerful model to be used for hateful/offensive content identification. ## conclusion over the years, neural transformer models have outperformed previous state-of-the-art deep learning models across various nlp tasks including offensive and hate speech detection tasks. nevertheless, these transformers are usually trained on general corpora which lack tweet and offensive languagespecific cues. previous studies have shown that domain-specific fine-tuning or retraining of models before attempting to solve downstream tasks can lead to excellent results in multiple domains. as discussed in this paper, fine-tuning/retraining a complex models to identify offensive language has not been substantially explored before and we address this gap by proposing fbert, a bert-baseuncased model that has been learned using over 1.4 million offensive instances from the solid dataset. the shifted fbert model better incorporates domain-specific offensive language and social media features. the fbert model achieves better results in both offenseval and hateval tasks and in the hs & o dataset over bert and hatebert. in future work, we would like to investigate the performance of fbert both at the post-and tokenlevel identification stages. furthermore, we will expand fbert to multiple languages. since our approach is based on a semi-supervised dataset, it is easily expandable to other languages as well. we plan to extend this process to other transformer models such as xlnet @xcite , roberta @xcite and @xcite . finally, fbert is publicly available on hugging face model hub @xcite ). 4