T ae B ench: Improving Quality of Toxic Adversarial Examples (2025)

Abstract: Toxicity text detectors can be vulnerable to adversarial examples - small perturbations to input text that fool the systems into wrong detection. Existing attack algorithms are time-consuming and often produce invalid or ambiguous adversarial examples, making them less useful for evaluating or improving real-world toxicity content moderators. This paper proposes an annotation pipeline for quality control of generated toxic adversarial examples (TAE). We design model-based automated annotation and human-based quality verification to assess the quality requirements of . Successful should fool a target toxicity model into making benign predictions, be grammatically reasonable, appear natural like human-generated text, and exhibit semantic toxicity. When applying these requirements to more than 20 state-of-the-art (SOTA) TAE attack recipes, we find many invalid samples from a total of 940k raw TAE attack generations. We then utilize the proposed pipeline to filter and curate a high-quality TAE dataset we call TaeBench (of size 264k ). Empirically, we demonstrate that TaeBench can effectively transfer-attack SOTA toxicity content moderation models and services. Our experiments also show that TaeBench with adversarial training achieve significant improvements of the robustness of two toxicity detectors.

## introduction toxicity text detection systems are popular content moderators for flagging text that may be considered toxic or harmful. these toxicity detectors are frequently used in safety-concerned applications like llm-based chatbots and face persistent threats from malicious attacks designed to circumvent and exploit them. recent literature includes a suite of text adversarial attacks that generate targeted adversarial examples from seed inputs, fooling a toxicity detection classifier into predicting "benign" outputs, while the examples are semantically toxic. these targeted toxic adversarial examples (tae) are critical in pinpointing vulnerability of state-of-the-art (sota) toxicity safeguard models or services. however, running existing tae attacks directly against a new model is time consuming (table a2 ), needs expert-level attack knowledge, and also results in many low-quality examples (see we, therefore, propose an annotation pipeline to conduct quality control of generated tae. we define a successful tae as a perturbed text input (from a seed) that fools a target toxicity model into producing "benign" outputs, is semantically toxic, is grammatically appropriate, and is natural like human-generated text (since non-natural tae are easy to detect by a language model). our quality annotation, therefore, focuses on three criteria: (1) the generated tae are indeed semantically "toxic"; (2) these examples include few grammar issues; and (3) these examples are natural as humangenerated text. for each criterion, we propose automated and human annotation-based strategies to measure and constrain these criteria. figure 1 illustrates the overall workflow. following this, we run more than 20 tae recipes derived from 6 sota tae attack algorithms from the literature (table a1 ) and apply the proposed annotation pipeline to examine the 940k generated raw tae examples. empirically, we find that most existing tae attack recipes generate invalid or ambiguous adversarial examples. for instance, our evaluation finds that less than 89% of adversarial examples are labeled as toxic by human annotators, facebook is a platform for sophomoric organisations to publicly post examples of their achievements. bad we have incompetent idiots running seattle and this state! we have capable geeks running seattle and this state! and less than 80% are judged as natural by humans. this careful filtering process helps us curate a high-quality dataset of more than 260k tae examples. we name it as taebench (toxic ## toxic adversarial examples (tae) and attack recipes this paper focuses on the tae proposed by @xcite . the main motivation of tae attacks is that a major goal of real-world toxicity detection is to identify and remove toxic language. adversarial attackers against toxicity detectors will focus on designing samples that are toxic in nature but can fool a target detector into making benign prediction (aka tae). tae attacks search for an adversarial example x ′ from a seed input x by satisfying a targeted goal function as follows: here b denotes the "0:benign" class. f : @xmath0. adversarial attack methods design search strategies to transform a seed x to x ′ via transformation, so that x ′ fools f by achieving the fooling goal g(f, x ′ ), and at the same time fulfilling a set of constraints. therefore literature has split each text adversarial attack into four components: (1) goal function, (2) transformation, (3) search strategy, and (4) constraints between seed and its adversarial examples @xcite . this modular design allows pairing the tae goal function (equation ( 1 )) with popular choices of other three components from the literature to obtain a large set of tae attack recipes. ## improving tae quality with an annotation pipeline as shown in ## connecting to related works literature has included no prior work on the quality control of adversarial examples from toxicity text detectors. literature includes just a few studies on adversarial examples for toxicity text classifiers. one recent study @xcite tried to deceive google's perspective api for toxicity identification by misspelling the abusive words or by adding punctuation between letters. another recent study @xcite proposed the concept of "toxic adversarial examples" and a novel attack called toxictrap attack. ## conclusion in this paper, we present a model-based pipeline for quality control in the generation of tae. by evaluating 20+ tae attack recipes, we curate a high-quality benchmark taebench. we demonstrate its effectiveness in assessing the robustness of real-world toxicity content moderation models, and show that adversarial training using taebench improves toxicity detectors' resilience against unseen attacks. annotation-based filtering process. we develop an efficient pipeline that employs models fine-tuned on a subset of taes annotated by a specially trained human team. before beginning their work, annotators are informed about the nature of the toxic data they will be working with, and written consent is obtained. it's important to note that while our approach significantly reduces the presence of low-quality taes, it does not eliminate all such instances, though minimizing them is our primary objective.