Dynamically Refined Regularization for Improving Cross-corpora Hate Speech Detection (2022)

Abstract: Hate speech classifiers exhibit substantial performance degradation when evaluated on datasets different from the source. This is due to learning spurious correlations between words that are not necessarily relevant to hateful language, and hate speech labels from the training corpus. Previous work has attempted to mitigate this problem by regularizing specific terms from pre-defined static dictionaries. While this has been demonstrated to improve the generalizability of classifiers, the coverage of such methods is limited and the dictionaries require regular manual updates from human experts. In this paper, we propose to automatically identify and reduce spurious correlations using attribution methods with dynamic refinement of the list of terms that need to be regularized during training. Our approach is flexible and improves the cross-corpora performance over previous work independently and in combination with pre-defined dictionaries.

## introduction the relative sparsity of hateful content in the real world requires crawling of many of the standard hate speech corpora through keyword-based sampling @xcite , rather than random sampling. thus, hate speech classifiers @xcite @xcite often learn spurious correlations from the training corpus @xcite leading to a substantial performance degradation when evaluated on a corpus with a different distribution @xcite @xcite @xcite . recent work has proposed regularization mechanisms to penalize spurious correlations by attempting to explain model predictions using feature attribution methods @xcite @xcite . these methods assign importance scores to input tokens that contribute more towards a particular prediction @xcite . for instance, @xcite penalize the attributions assigned to tokens contained in a manually curated dictionary consisting of group identifiers (e.g. women, jews) that are often known to be targets of hate. @xcite extract group identifiers manually from the top tokens indicated by a bag-of-words logistic regression model trained on the source corpus. however, regularizing only group identifiers limits the coverage of such approaches, and may not capture other forms of corpus-specific correlations learned by the classifier limiting its performance on a new corpus. moreover, such manually curated lists may not always remain up-to-date because new terms emerge frequently @xcite . while @xcite do not use such lists for refining models in different target-domains, their method still requires input from human annotators. in this paper, we hypothesize that the classification errors in a small annotated subset from the target can reveal spurious correlations between tokens and hate speech labels learned from the source (see ## conclusion we proposed a dynamic approach for automatic token extraction with regularization of the source model such that the spurious source specific correlations are reduced. our approach shows consistent cross-corpora performance improvements both independently and in combination with pre-defined tokens. future work includes applying our method on other cross-domain text classification tasks and exploring how explanation faithfulness can be improved in out-of-domain settings @xcite . mukund sundararajan, ankur taly, and qiqi yan. 2017. axiomatic attribution for deep networks. in proceedings of the 34th international conference on machine learning, icml 2017, sydney, nsw, australia, 6-11 august 2017, volume 70 of proceedings of machine learning research, pages 3319-3328. pmlr. steve durairaj swamy, anupam jamatia, and björn gambäck. 2019. studying generalisability across abusive language detection datasets. in proceedings of the 23rd conference on computational natural language learning (conll), pages 940-950, hong kong, china. association for computational linguistics. bertie vidgen, tristan thrush, zeerak waseem, and douwe kiela. 2021. learning from the worst: dynamically generated datasets to improve online hate detection. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 1667-1682, online. association for computational linguistics. zeerak waseem and dirk hovy. 2016. hateful symbols or hateful people? predictive features for hate speech detection on twitter. in proceedings of the naacl student research workshop, pages 88-93, san diego, california. association for computational linguistics. michael wiegand, josef ruppenhofer, and thomas kleinbauer. 2019. detection of abusive language: the problem of biased datasets. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 602-608, minneapolis, minnesota. association for computational linguistics. huihan yao, ying chen, qinyuan ye, xisen jin, and xiang ren. 2021. refining language models with compositional explanations. advances in neural information processing systems, 34. wenjie yin and arkaitz zubiaga. 2021. towards generalisable hate speech detection: a review on obstacles and solutions. arxiv preprint, abs/2102.08886.