Fortifying Toxic Speech Detectors Against Veiled Toxicity (2020)

Abstract: Modern toxic speech detectors are incompetent in recognizing disguised offensive language, such as adversarial attacks that deliberately avoid known toxic lexicons, or manifestations of implicit bias. Building a large annotated dataset for such veiled toxicity can be very expensive. In this work, we propose a framework aimed at fortifying existing toxic speech detectors without a large labeled corpus of veiled toxicity. Just a handful of probing examples are used to surface orders of magnitude more disguised offenses. We augment the toxic speech detector’s training data with these discovered offensive examples, thereby making it more robust to veiled toxicity while preserving its utility in detecting overt toxicity.

## toxic language in disguise toxic language has been recognized as a severe problem in the online social communities. while great efforts have been made to detect and prevent the spread of overt trolling, hate speech, abusive language, and toxic comments (schmidt and wiegand, 2017; fortuna and nunes, 2018), they often build upon lexicon-based approaches (waseem and hovy, 2016; davidson et al., 2017) and thus are ineffective at detecting forms of veiled toxicity; e.g., codewords (taylor et al., 2017), novel forms of offense (jain et al., 2018), and subtle and often unintentional manifestations of social bias such as microaggressions and condescension (breitfeller et al., 2019; wang and potts, 2019). in this work, we focus on disguised toxic language that is often undetected by existing tools. @xcite it can be innocuous on the surface lexicon-level, but as offensive as hate speech and lastingly harmful (sue et al., 2007; sue, 2010; nadal et al., 2014; jurgens et al., 2019). @xcite detecting veiled toxicity is hard: deep semantic analysis and large datasets are needed. but veiled offenses are not represented in existing toxicity datasets (waseem and hovy, 2016; davidson et al., 2017; founta et al., 2018) and building a new dataset is expensive: candidates for annotation cannot be filtered through lexicons and random sampling of social media posts will surface only a tiny fraction of relevant examples (breitfeller et al., 2019). moreover, since biased text is often unconscious and subjective, untrained annotators might mislabel it due to their own biases (breitfeller et al., 2019; field and tsvetkov, 2020). we propose a framework to surface veiled offenses and improve toxicity classifiers that are compromised in detecting them. it requires a small set of labeled probing examples to surface orders of magnitude more disguised offenses missed by the classifier, through interpretable ml techniques tracking the influence of training examples on classifier's predictions (koh and liang, 2017; pruthi et al., 2020). from an original classifier that detects veiled toxicity with an accuracy of 1%, we achieve up-to 51% accuracy in detecting veiled offenses while preserving the utility of the classifier in detecting overt offenses. to the best of our knowledge, our work is the first in making toxic speech detectors robust against veiled toxicity with almost no annotated data. ## identifying veiled toxicity a typical toxicity classifier c might fail to identify veiled offenses because they are not well represented among toxic examples in its training data d. moreover, the non-toxic portion of d might be polluted with (mislabeled) disguised offenses. at inference time, c might thus mislabel, for example, microaggressions as well as adversarial attacks deliberately avoiding known toxic lexicons. to make such compromised classifier c more robust, we propose a framework that does not require access to d. @xcite we start with a dataset d comprising examples that can be labeled as offensive or non-offensive. @xcite we build a student model c on d that approximates the behavior of c: @xmath0, x, c(x)), where x are instances from d . an effective c would mimic the behavior of c, including mislabeling disguised toxic examples x veiled as non-offensive. to address this issue, we surface the unknown x veiled out of the training set by probing the c with a small labeled held-out set of veiled offenses p, @xcite and tracing the model's decisions back to training examples in d . we hypothesize the 'influential' training examples that lead to c 's predictions on p are likely to be x veiled . one probing example can surface multiple influential examples in the training data. these influential examples can then be re-annotated, making the model more robust to future veiled toxicity. the key observation is that sampling on random a subset of non-offensive data to annotate it for disguised offensiveness is prohibitively expensive, as only a tiny fraction of annotated examples will be indeed offensive. our proposed approach increases the likelihood to identify disguised offensiveness by surfacing training examples that influence the decisions of c when tested on p. the key challenge is to devise a method to track examples influential to classifier's predictions; we discuss candidate approaches in the rest of this section. @xcite since hate speech training data is often proprietary and platform-specific (macavaney et al., 2019). @xcite here we assume a binary classifier for simplicity, but our setup is applicable to multi-class settings. @xcite although it could be very expensive to create largescale datasets for disguised offenses, it is feasible to collect a very small number (e.g., less than 100) of exemplars, using methods like user reporting, e.g., https://www. microaggressions.com/ . ## conclusion we propose a framework to robustify toxicity classifiers against veiled toxicity. through a few labeled probing examples, we can accurately surface orders of magnitude more disguised toxic messages missed by a compromised classifier, using interpretable ml techniques that track the influence of training examples on the probing examples. our framework, however, is not limited to toxicity detection. future work can explore how to enhance a sub-optimal model using the teacher-student setup for tasks that change across domains or over time, or in scenarios where the original model and data are restricted for privacy reasons. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, r'emi louf, morgan funtowicz, and jamie brew. 2019. huggingface's transformers: state-of-the-art natural language processing. arxiv, abs/1910.03771. chih-kuan yeh, joon sik kim, ian en-hsu yen, and pradeep ravikumar. 2018. representer point selection for explaining deep neural networks. in proc. neurips. jingbo zhu, huizhen wang, and eduard h. hovy. 2008. learning a stopping criterion for active learning for word sense disambiguation and text classification. in proc. ijcnlp.