Towards Building a Robust Toxicity Predictor (2023)

Abstract: Recent NLP literature pays little attention to the robustness of toxicity language predictors, while these systems are most likely to be used in adversarial contexts. This paper presents a novel adversarial attack, \texttt{ToxicTrap}, introducing small word-level perturbations to fool SOTA text classifiers to predict toxic text samples as benign. \texttt{ToxicTrap} exploits greedy based search strategies to enable fast and effective generation of toxic adversarial examples. Two novel goal function designs allow \texttt{ToxicTrap} to identify weaknesses in both multiclass and multilabel toxic language detectors. Our empirical results show that SOTA toxicity text classifiers are indeed vulnerable to the proposed attacks, attaining over 98\% attack success rates in multilabel cases. We also show how a vanilla adversarial training and its improved version can help increase robustness of a toxicity detector even against unseen attacks.

## introduction deep learning-based natural language processing (nlp) plays a crucial role in detecting toxic language content @xcite @xcite @xcite . toxic content often includes abusive language, hate speech, profanity or sexual content. recent methods have mostly leveraged transformer-based pre-trained language models @xcite and achieved high performance in detecting toxicity @xcite . however, directly deploying nlp models could be problematic for real-world toxicity detection. this is because toxicity filtering is mostly needed in security-relevant industries like gaming or social networks where models are constantly being challenged by social engineering and adversarial attacks. in this paper, we study the adversarial robustness of toxicity language predictors foot_0 and propose a new set of attacks, we call "toxictrap ". toxictrap generates targeted adversarial examples that fool a target model towards the benign predictions. our design is motivated by the fact that most toxicity classifiers are being deployed as api services and used for flagging out toxic samples. figure 1 shows one toxictrap adversarial example. the perturbed text replaces one word with its synonym and the resulting phrase fooled the transformer based detector into a failed detection @xcite . we propose novel goal functions to guide greedy word importance ranking to iteratively replace each word with small perturbations. samples generated by toxictrap are toxic, and can fool a victim toxicity predictor model to classify them as "benign" and not as any toxicity classes or labels. the proposed toxictrap attacks can pinpoint the robustness of both multiclass and multilabel toxicity nlp models. to the authors' best knowledge, this paper is the first work that introduces adversarial attacks foot_1 to fool multilabel nlp tasks. design multilabel toxictrap is challenging since coordinating multiple labels all at once and quantifying the attacking goals is tricky when aiming to multiple targeted and "toxicity prediction" interchangeably. ## method methods generating text adversarial examples introduce small perturbations in the input data checking if a target model's output changes significantly. these adversarial attacks help to identify if an nlp model is susceptible to word replacements, misspellings, or other variations that are commonly found in real-world data. for a given nlp classifier f : @xmath0, searching for an adversarial example x ′ from x is: here t (x, ∆x) denotes the transformations that perturb text x to x ′ . g(f, x ′ ) represents a goal function that defines the purpose of an attack, for instance like flipping the output. {c i (x, x ′ )} denotes a set of constraints that filters out undesirable x ′ to ensure that perturbed x ′ preserves the semantics and fluency of the original x. to solve equation (1), adversarial attack methods design search strategies 3 to transform x to x ′ via transformation t (x, ∆x), so that x ′ fools f by achieving the fooling goal g(f, x ′ ), and at the same time fulfilling a set of constraints {c i (x, x ′ )}. therefore designing adversarial attacks focus on 3 because brute force is not feasible considering the length and dictionary size of natural language text. designing four components: (1) goal function, (2) transformation, (3) search strategy, and (4) constraints between seed and its adversarial examples @xcite ). we propose a suite of toxictrap attacks to identify vulnerabilities in sota toxicity nlp detectors. toxictrap attacks focus on intentionally generating perturbed texts that contain the same highly abusive language as original toxic text, yet receive significantly lower toxicity scores and get predicted as "benign" by a target model. ## experiments we conducted a series of experiments covering three different toxicity classification tasks: binary, multilabel, and multiclass; over two different transformer architectures: distillbert and bert; and across two datasets: the large-scale wikipedia talk page dataset-jigsaw (jig, 2018) and the offensive tweet for hate speech detection dataset (davidson base toxicity models: our experiments work on three base models, including {jigsaw-bl , jigsaw-ml , htweet-mc } to cover three types of toxicity prediction tasks. see details in section a.5. implementation: we implement all of our toxictrap and toxictraps extend attacks using the nlp attack package textattack 5 @xcite . when generating adversarial examples, we only attack seed samples that are correctly predicted by a victim model. (adversarial attack does not make sense if the target model could not predict the seed sample correctly!). in our setup, this means we only use toxic seed samples when attacking three base models. this set up simulates real-world situations in which people intentionally create creative toxic examples to circumvent toxicity detection systems. evaluation metrics: we use attack success rate (@xmath1://github.com/qdata/textattack . to measure the runtime of each algorithm, we use the average number of queries to the victim model as a proxy. we also report the average percentage of words perturbed from an attack. in addition, for models trained with adversarial training, we report both, the model prediction performance and model robustness (by attacking robust model again). ## conclusion text toxicity prediction models are not designed to operate in the presence of adversaries. this paper proposes a suite of toxictrap attacks to identify weaknesses in sota toxicity language predictors that could potentially be exploited by attackers. we also evaluate how adversarial training improves model robustness across seen and unseen attacks. as next steps, we plan to investigate other strategies like virtual adversarial training, disentangled representation learning or generative methods and pinpoint how they will influence the robustness of toxicity predictors @xcite .