Improving Generalizability in Implicitly Abusive Language Detection with Concept Activation Vectors (2022)

Abstract: Robustness of machine learning models on ever-changing real-world data is critical, especially for applications affecting human well-being such as content moderation. New kinds of abusive language continually emerge in online discussions in response to current events (e.g., COVID-19), and the deployed abuse detection systems should be updated regularly to remain accurate. In this paper, we show that general abusive language classifiers tend to be fairly reliable in detecting out-of-domain explicitly abusive utterances but fail to detect new types of more subtle, implicit abuse. Next, we propose an interpretability technique, based on the Testing Concept Activation Vector (TCAV) method from computer vision, to quantify the sensitivity of a trained model to the human-defined concepts of explicit and implicit abusive language, and use that to explain the generalizability of the model on new data, in this case, COVID-related anti-Asian hate speech. Extending this technique, we introduce a novel metric, Degree of Explicitness, for a single instance and show that the new metric is beneficial in suggesting out-of-domain unlabeled examples to effectively enrich the training data with informative, implicitly abusive texts.

## introduction when machine learning models are deployed in the real world, they must be constantly monitored for their robustness to new and changing input data. one area where this is particularly important is in abusive language detection @xcite @xcite . the content of online conversation is constantly changing in response to political and social events. new categories of abusive language emerge, encompassing topics and vocabularies unknown to previously trained classifiers. here, we tackle three main questions: how can a human user formalize new, relevant topics or concepts in text? how do we quantify the sensitivity of a trained classifier to these new concepts as they emerge? and how do we update the classifier so that it remains reliable? as a case study, we consider the rise of covidrelated anti-asian racism on social media. the covid-19 pandemic represented an entirely new and unexpected situation, generating new vocabulary (covid-19, coronavirus, social distancing, masking), new topics of conversation (dealing with isolation, working from home), and -unfortunately -new and renewed instances of hate speech directed towards asian communities. we imagine the case of an abusive language detection algorithm which had been deployed prior to the pandemic: what are the new types of abusive language that have emerged with the recent pandemic? to what extent can deployed classifiers generalize to this new data, and how can they be adapted? although social events can spark off a specific type of hate speech, they are rarely the root cause of the issue. often such hateful beliefs existed before the event, and are only magnified because of it @xcite . therefore, we expect that the classifier should detect this new variety of hate speech to some extent. an important factor in this study is whether the text expresses explicit or implicit abuse @xcite @xcite . explicit abuse refers to utterances that include direct insults or strong rudeness, often involving profanities, whereas implicit abuse involves more indirect and nuanced language. since understanding the offensive aspects of implicit abuse in our case study may require some knowledge of the context (i.e., the pandemic), we expect that the pretrained classifier will find these data especially difficult to handle. to examine a classifier's ability to handle new type of abusive text (without access to extensive labeled data), we propose a technique based on the testing concept activation vector (tcav) method from the interpretability literature in computer vision @xcite . tcav is used to explain whether a classifier associates a specific concept to a class label (e.g., the concept of stripes is associated with class zebra in image classification). similarly, we define implicit and explicit covidrelated anti-asian racism with a small set of humanchosen textual examples, and ask whether the pretrained classifier associates these concepts with the positive (abusive) class label. further, we ask whether sensitivity to humandefined concepts can direct data augmentation foot_0 to improve generalizations. intuitively, when updating a classifier, data enrichment should focus on adding examples of concepts to which the classifier is not yet sensitive. conventional active learning frameworks suggest examples with the lowest classification confidence as the most informative augmentation samples @xcite . however, deep neural networks' inability to provide reliable uncertainty estimates is one of the main barriers to adopting confidence-based sampling techniques @xcite . we suggest that, in the case of abuse detection, implicitly abusive examples are most informative for updating a general classifier. however, to the best of our knowledge, there is no quantitative metric that can measure the degree of explicitness of a candidate example, given a trained classifier. we extend the tcav technique to provide a "degree of explicitness" measure at the utterance level and use that for efficient data augmentation. the contributions of this work are as follows: ## datasets and data analysis we consider the following four english datasets, summarized in ## cross-dataset generalization we start by assessing the robustness of a generalpurpose abusive language classifier on a new domain of abusive language. specifically, we analyze the performance of classifiers trained on the wiki and founta datasets (expected to detect general toxicity and abuse) on covid-related anti-asian racism data. in addition, we want to assess the impact of the change of vocabulary on the generalizibility of the classifiers to implicit and explicit abuse in the new domain. we train binary roberta-based classifiers on the wiki, founta, ea and ch datasets (referred to hereafter as the wiki, founta, ea and ch classifiers), and test them on the ea as the mostly implicit covid-related dataset and ch as the mostly explicit covid-datasets count shared words ea -ch 50 covid-related (32%): ccp, 19, communist, pandemic, coronavirus, covid19, chinesevirus, infected, covid, chinese, chinavirus, corona, wuhanvirus, wuhan, china, virus hateful (0%) other (68%): racist, came, want, country, calling, come, does, spread, like, amp, media, eating, did, human, world, know, government, say, started, think, need, blame, evil, time, people, don, new, let, news, stop, countries, just, spreading, make wiki -founta 37 covid-related (0%) hateful (30%): *ss, b*tch, id*ot, n*ggas, d*ck, f*cking, f*ck, sh*t, hell, hate, stupid other (70%): oh, dont, want, way, going, come, does, like, look, life, did eat, sex, know, say, think, man, need, time, people, said, stop, really, just, make, tell founta -ea 19 covid-related (0%) hateful (0%) other (100%): racist, want, calling, come, does, like, did, world, know, say, think, need, time, people, trying, let, stop, just, make wiki -ea 15 covid-related (0%) hateful (0%) other (100%): people, want, did, say, think, good, need, come, does, stop, just, know, like, make, time founta -ch 35 covid-related (0%) hateful (23%): *ss, b*tch, f*cking, f*ck, sh*t, hate, stupid, f*cked other (77%): racist, want, way, going, calling, come, does, like, got, look, did, eat, world, know, say, think, man, trump, need, time, people, said, let, stop, really, just, make wiki -ch 33 covid-related (0%) hateful (27%): *ss, b*tch, f*cking, f*ck, sh*t, hate, stupid, shut, kill other (73%): want, way, going, come, does, like, look, did, eat, right, know, die, say, think, man, need, time, people, don, said, stop, really, just, make cross-dataset generalization is better when datasets share similar vocabulary. the classifiers trained on the ea and ch datasets perform better than all the classifiers trained on the pre-covid datasets (wiki and founta). interestingly, the performance of the ch classifier on the ea dataset is higher than the performance of all the general classifiers, despite the ch dataset being very small and containing mostly explicit abuse. this observation confirms that general classifiers need to be updated to learn the new vocabulary. general-purpose classifiers generalize better to explicit than implicit examples in the new domain. the wiki and founta classifiers, which have been exposed to large amounts of generally explicit abuse, perform well on the mostly explicit ch dataset, but experience difficulty with the covidspecific implicit abuse in the ea dataset. for example, the tweet 'the chinavirus is a biological attack initiated by china' is misclassified as non-abusive. we observe that wiki-exp performs relatively similar to the wiki classifier on ch, despite its small size (only 1,294 positive examples) but is worse than wiki classifier on ea. this means that the additional 35k instances (of which, 9k are positive examples) of the wiki compared to the wiki-exp, only moderately improve the classification of the implicit examples in the new domain. this observation indicates that generalization mostly occurs between the explicit type of the pre-covid abuse and the explicit type of the covid-related abuse. therefore, a general-purpose classifier should be specifically updated to learn implicit abuse in the new domain. ## sensitivity to implicit and explicit abuse to explain generalizability in section 3, we showed that when a new domain emerges, the change in vocabulary mostly affects the classification of implicitly expressed abuse. this observation is in line with findings by @xcite , and suggests that generalization should be evaluated on implicit and explicit abuse separately. however, due to complexities of annotation of abusive content, curating separate implicit and explicit test sets is too costly @xcite . instead, we propose to adapt the testing concept activation vector (tcav) algorithm, originally developed for image classification @xcite , to calculate the classifiers' sensitivity to explicit and implicit covid-related racism, using only a small set of examples. then, we show how these sensitivities can explain the generalizations observed in ## degree of explicitness here, we suggest that implicit examples are more informative (less redundant) for updating a general classifier and provide a quantitative metric to guide the data augmentation process. we extend the tcav methodology to estimate the degree of explicitness or doe of an utterance. we showed that the average tcav score of the positive class for the explicit concept is close to 1. doe is based on the idea that adding one example to an explicit concept will not affect its average tcav score (i.e., it will still be close to 1), if the added example is explicitly abusive. however, adding an implicit example presumably will change the direction of all cavs and reduce the sensitivity of the classifier to this modified concept. here, we modify equation 1 and calculate each cav by averaging the roberta representations of n υ -1 explicit concept examples, and the new utterance for which we want the degree of explicitness, x new , with representation r new . thus, we do not measure the sensitivity of this classifier to the explicit anti-asian and implicit ch concepts, since their concept examples are included in the training set of the classifier. we then calculate the average tcav score for each x new as its doe score. if the new utterance, x new , is explicitly abusive, υ p new will represent an explicit concept, and the average tcav score, i.e., mean(t cav c,p ) will remain close to 1. however, the less explicit the new example is, the more υ p new will diverge from representations of explicit abuse, and the average score will drop. we use n υ = 3 in the following experiments. ## data augmentation with doe score we now use the doe score to direct data augmentation. we consider a scenario where a general classifier should be re-trained with an augmented dataset to include emerging types of abusive language. as we showed, general classifiers are already sensitive to explicit abuse. therefore, we hypothesize that implicit examples are more beneficial for updating the classifier. here, we describe a novel doe-based augmentation approach and contrast it with the conventional process of choosing augmentation examples based on the classification confidence @xcite . we consider the general wiki classifier. our goal is to find a small but sufficient portion of the ea train set to augment the original wiki train set, so that the classifier is able to handle covid-related anti-asian hate speech. we calculate the doe and confidence scores for all the examples in the ea train set and add the n examples with the lowest scores to the original wiki train set. we vary n from 1k to 6k, with a 1k step. after the augmentation data size reaches 6k, the classifier performance on the original wiki test set drops substantially for both techniques. also, note that as the size of the augmentation dataset increases, the two methods converge to the same performance. ## related work the distinction between explicit and implicit abuse has been recognized as an important factor in abuse detection @xcite . @xcite showed that lexiconbased sampling strategies fail to collect implicit abuse and most of the annotated datasets are overwhelmed with explicit examples. @xcite showed that inter-annotation agreement is low when labeling the implicit abuse utterances, as sometimes specific knowledge is required in order to understand the implicit statements. for better detection of implicitly stated abuse, large annotated datasets with hierarchical annotations are needed @xcite , so that automatic detection systems can learn from a wide variety of such training examples. @xcite proposed propensity matching and adversarial learning to force the model to focus on signs of implicit bias. @xcite created a novel dataset for studying implicit abuse and presented a range of linguistic features for contrastive analysis of abusive content. we define explicitness as obvious rudeness and hateful language regardless of the social context and introduce a quantitative measure of explicitness from a trained classifier's point of view. data augmentation has been used to improve the robustness of abuse detection classifiers. to mitigate biases towards specific terms (e.g., identity terms), one strategy is to add benign examples con-taining the biased terms to the training data @xcite . other works combined multiple datasets to achieve better generalizations, using a set of probing instances @xcite , multi-task training @xcite , and domain adaptation @xcite . in contrast to these works, we take an interpretability-based approach and guide the data collection process by mapping the new data on the implicit vs. explicit spectrum. ## conclusion as real-world data evolves, we would like to be able to query a trained model to determine whether it generalizes to the new data, without the need for a large, annotated test set. we adopted the tcav algorithm to quantify the sensitivity of text classifiers to human-chosen concepts, defined with a small set of examples. we used this technique to compare the generalizations of abusive language classifiers, trained with pre-pandemic data, to explicit and implicit covid-related anti-asian racism. we then proposed a sensitivity-based data augmentation approach, to improve generalizability to emerging categories. we showed that in the case of abuse detection, the most informative examples are implicitly abusive utterances from the new category. our approach collects implicit augmentation examples and achieves higher generalization to the new category compared to confidence-based sampling. strategies for choosing the optimal set of concept examples should be explored in the future. while we examined abusive language detection as a case study, similar techniques can be applied to different nlp applications. for example, the tcav method could be used to measure the sensitivity of a sentiment analysis system to a new product, or a stance detection algorithm's sensitivity to an important new societal issue. as language evolves, methods of monitoring and explaining classifier behaviour over time will be essential.