Unsupervised Domain Adaptation in Cross-corpora Abusive Language Detection (2021)

Abstract: The state-of-the-art abusive language detection models report great in-corpus performance, but underperform when evaluated on abusive comments that differ from the training scenario. As human annotation involves substantial time and effort, models that can adapt to newly collected comments can prove to be useful. In this paper, we investigate the effectiveness of several Unsupervised Domain Adaptation (UDA) approaches for the task of cross-corpora abusive language detection. In comparison, we adapt a variant of the BERT model, trained on large-scale abusive comments, using Masked Language Model (MLM) fine-tuning. Our evaluation shows that the UDA approaches result in sub-optimal performance, while the MLM fine-tuning does better in the cross-corpora setting. Detailed analysis reveals the limitations of the UDA approaches and emphasizes the need to build efficient adaptation methods for this task.

## introduction social networking platforms have been used as a medium for expressing opinions, ideas, and feelings. this has resulted in a serious concern of abusive language, which is commonly described as hurtful, obscene, or toxic towards an individual or a group sharing common societal characteristics such as race, religion, gender, etc. the huge amount of comments generated every day on these platforms make it increasingly infeasible for manual moderators to review every comment for its abusive content. as such, automated abuse detection mechanisms are employed to assist moderators. we consider the variations of online abuse, toxicity, hate speech, and offensive language as abusive language and this work addresses the detection of abusive versus non-abusive comments. supervised classification approaches for abuse detection require a large amount of expensive annotated data @xcite . moreover, models already trained on the available annotated corpus report degraded performance on new content @xcite @xcite . this is due to phenomena like change of topics discussed in social media, and differences across corpora, such as varying sampling strategies, targets of abuse, abusive language forms, etc. these call for approaches that can adapt to newly seen content out of the original training corpus. annotating such content is non-trivial and may require substantial time and effort @xcite . thus, unsupervised domain adaptation (uda) methods that can adapt without the target domain labels @xcite , turn out to be attractive in this task. given an automatic text classification or tagging task, such as abusive language detection, a corpus with coherence can be considered a domain @xcite . under this condition, domain adaptation approaches can be applied in cross-corpora evaluation setups. this motivates us to explore uda for cross-corpora abusive language detection. a task related to abuse detection is sentiment classification @xcite , and it involves an extensive body of work on domain adaptation. in this work, we analyze if the problem of cross-corpora abusive language detection can be addressed by the existing advancements in domain adaptation. alongside different uda approaches, we also evaluate the effectiveness of recently proposed hate-bert model @xcite that has finetuned bert @xcite on a large corpus of abusive language from reddit using the masked language model (mlm) objective. furthermore, we perform the mlm fine-tuning of hatebert on target corpus, which can be considered a form of unsupervised adaptation. our contribution is summarised below: the remaining of this paper is structured as follows: section 2 discusses the shifts across different abusive corpora. section 3 surveys some recently proposed uda models for sentiment classification and discusses the main differences in the approaches. section 4 presents the experimental settings used in our evaluation. the results of our evaluation and a discussion on performances of different approaches are present in section 5. finally, section 6 concludes the paper and highlights some future work. moreover, sampling strategies across datasets also introduce bias in the data @xcite , and could be a cause for differences across datasets. for instance, @xcite sample tweets containing keywords from a hate speech lexicon, which has resulted in the corpus having a major proportion (83%) of abusive content. as mentioned by @xcite , tweets in @xcite originate from the united states, whereas @xcite sample them without such a demographic constraint. apart from sampling differences, the targets and types of abuse may vary across datasets. for instance, even though women are targeted both in @xcite and @xcite , the former involves more subtle and implicit forms of abuse, while the the latter involves explicit abuse involving profane words. besides, religious minorities are the other targeted groups in @xcite , while african americans are targeted in @xcite . owing to these differences across corpora, abusive language detection in a cross-corpora setting remains a challenge. this has been empirically validated by @xcite al. (2019); arango et al. (2019); swamy et al. (2019) ; @xcite with performance degradation across the cross-corpora evaluation settings. thus, it can be concluded that the different collection time frames, sampling strategies, and targets of abuse would induce a shift in the data. ## unsupervised domain adaptation as discussed by @xcite ; @xcite , a coherent type of corpus can typically be considered a domain for tasks such as automatic text classification. we, therefore, decide to apply domain adaptation methods for our task of crosscorpora abuse detection. besides, uda methods aim to adapt a classifier learned on the source domain d s to the target domain d t , where only the unlabeled target domain samples x t and the labeled source domain samples x s are assumed to be available. we denote the source labels by y s . in this work, we use the unlabeled samples x t for adaptation and evaluate the performance over the remaining unseen target samples from d t . ## conclusion and future work this work analyzed the efficacy of some successful unsupervised domain adaptation approaches of cross-domain sentiment classification in crosscorpora abuse detection. our experiments highlighted some of the problems with these approaches that render them sub-optimal in the cross-corpora abuse detection task. while the extraction of pivots, in the pivot-based models, is not optimal enough to capture the shared space across domains, the domain adversarial methods underperform substantially. the analysis of the masked language model fine-tuning of hatebert on the target corpus displayed improvements in general as compared to only fine-tuning hatebert over the source corpus, suggesting that it helps in adapting the model towards target-specific language variations. the overall performance of all the approaches, however, indicates that building robust and portable abuse detection models is a challenging problem, far from being solved. future work along the lines of domain adversarial training should explore methods which learn class boundaries that generalize well to the target corpora while performing alignment of the source and target representation spaces. such an alignment can be performed without target class labels by minimizing the intra-class domain discrepancy @xcite . pivot-based approaches should explore pivot extraction methods that account for higher-level semantics of abusive language across source and target corpora.