Exploiting Auxiliary Data for Offensive Language Detection with Bidirectional Transformers (2021)

Abstract: Offensive language detection (OLD) has received increasing attention due to its societal impact. Recent work shows that bidirectional transformer based methods obtain impressive performance on OLD. However, such methods usually rely on large-scale well-labeled OLD datasets for model training. To address the issue of data/label scarcity in OLD, in this paper, we propose a simple yet effective domain adaptation approach to train bidirectional transformers. Our approach introduces domain adaptation (DA) training procedures to ALBERT, such that it can effectively exploit auxiliary data from source domains to improve the OLD performance in a target domain. Experimental results on benchmark datasets show that our approach, ALBERT (DA), obtains the state-of-the-art performance in most cases. Particularly, our approach significantly benefits underrepresented and under-performing classes, with a significant improvement over ALBERT.

## introduction in today's digital age, the amount of offensive and abusive content found online has reached unprecedented levels. offensive content online has several detrimental effects on its victims, e.g., victims of cyberbullying are more likely to have lower self-esteem and suicidal thoughts @xcite . to reduce the impact of offensive online contents, the first step is to detect them in an accurate and timely fashion. next, it is imperative to identify the type and target of offensive contents. segregating by type is important, because some types of offensive content are more serious and harmful than other types, e.g., hate speech is illegal in many countries and can attract large fines and even prison sentences, while profanity is not that serious. to this end, offensive language detection (old) has been extensively studied in recent years, which is an active topic in natural language understanding. existing methods on old, such as @xcite , mainly focus on detecting whether the content is offensive or not, but they can not identify the specific type and target of such content. @xcite analyze a corpus of around 16k tweets for hate speech detection, make use of meta features (such as gender and location of the user), and employ a simple n-gram based model. @xcite evaluate the performance of some deep learning models, including bert @xcite , and achieve the state of the art results on a newly collected old dataset, olid @xcite . although promising progress on old has been observed in recent years, existing methods, especially the deep learning based ones, often rely on large-scale well-labeled data for model training. in practice, labeling offensive language data requires tremendous efforts, due to linguistic variety and human bias. in this paper, we propose to tackle the challenging issue of data/label scarcity in offensive language detection, by designing a simple yet effective domain adaptation approach based on bidirectional transformers. domain adaptation aims to enhance the model capacity for a target domain by exploiting auxiliary information from external data sources (i.e., source domains), especially when the data and labels in the target domain are insufficient @xcite @xcite @xcite . in particular, we aim to identify not only if the content is offensive, but also the corresponding type and target. in our work, the offensive language identification dataset (olid) @xcite is considered as target domain, which contains a hierarchical multi-level structure of offensive contents. an external large-scale dataset on toxic comment (toxcom) classification is used as source domain. albert @xcite is used in our approach owing to its impressive performance on old. a set of training procedures are designed to achieve domain adaptation for the old task. in particular, as the external dataset is not labelled in the same format as the olid dataset, we design a separate predictive layer that helps align two domains. extensive empirical evaluations of our approach and baselines are conducted. the main contributions of our work are summarized as follows: ## related work in this section, we briefly review related work on offensive language detection and transformers. offensive language detection. offensive language detection (old) has become an active research topic in recent years @xcite . @xcite experimented with a variety of models and observe promising results with bert and svc based models. @xcite employed a gru based rnn with 100 dimensional glove word embeddings @xcite . additionally, they develop a modified sentence offensiveness calculation (msoc) model which makes use of a dictionary of offensive words. @xcite evaluated three models on the olid dataset, including logistic regression, lstm and bert, and results show that bert achieves the best performance. the concept of transfer learning mentioned in @xcite is closely related to our work, since the bert model is also pretrained on external text corpus. however, different from @xcite , our approach exploits external data that are closely related to the old task, and we propose a new training strategy for domain adaptation. transformers. transformers @xcite are developed to solve the issue of lack of parallelization faced by rnns. in particular, transthe score between two words signifies how related they are. due to the parallelization, transformers train rapidly on modern day gpus. some representative transformer-based architectures for language modeling include bert @xcite , xl-net @xcite and albert @xcite . bert employs the deep bidirectional transformers architecture for model pretraining and language understanding @xcite . however, bert usually ignores the dependency between the masked positions and thus there might be discrepancy between model pretraining and finetuning. xl-net is proposed to address this issue, which is a generalized autoregressive pretraining method @xcite . another issue of bert is the intensive memory consumption during model training. recently, some improved techniques such as albert @xcite are proposed to reduce the memory requirement of bert and therefore increases the training speed. in this paper, we leverage the recent advances on transformers and design a domain adaptation approach for the task of offensive language detection. ## conclusion in this paper, we propose a simple yet effective domain adaptation approach to train bidirectional transformers for offensive language detection. our approach effectively exploits external datasets that are relevant to offensive content classification to enhance the detection performance on a target dataset. experimental results show that our approach, albert (da) obtains the state-of-theart performance in most tasks, and it significantly benefits underrepresented and under-performing classes.