Red Teaming Language Model Detectors with Language Models (2024)

Abstract: The prevalence and strong capability of large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. To prevent the potentially deceptive usage of LLMs, recent work has proposed algorithms to detect LLM-generated text and protect LLMs. In this paper, we investigate the robustness and reliability of these LLM detectors under adversarial attacks. We study two types of attack strategies: 1) replacing certain words in an LLM’s output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation. In both strategies, we leverage an auxiliary LLM to generate the word replacements or the instructional prompt. Different from previous works, we consider a challenging setting where the auxiliary LLM can also be protected by a detector. Experiments reveal that our attacks effectively compromise the performance of all detectors in the study with plausible generations, underscoring the urgent need to improve the robustness of LLM-generated text detection systems. Code is available at https://github.com/shizhouxing/LLM-Detector-Robustness.

## introduction large language models (llms), such as chat-gpt (openai, 2023b), palm @xcite , and llama @xcite , have demonstrated human-like capabilities to generate high-quality text, follow instructions, and respond to user queries. although llms can improve the work efficiency of humans, they also pose several ethical and safety concerns, such as it becomes hard to differentiate llm-generated text from human-written text. for example, llms may be inappropriately used for academic plagiarism or creating misinformation at large scale @xcite . therefore, it is important to de- * alphabetical order. authors contributed equally. toward this end, previous work has developed methods for automatically detecting text generated by llms. existing methods mainly fall into three categories: 1) classifier-based detectors by training a classifier, often a neural network, from data with ai-generated/human-written labels @xcite ; 2) watermarking @xcite by injecting patterns into the generation of llms such that the pattern can be statistically detected but imperceptible to humans; 3) likelihood-based detectors, e.g., detectgpt @xcite , by leveraging the log-likelihood of generated texts. however, as recent research demonstrates that text classifiers are vulnerable to adversarial attacks @xcite @xcite , these llm text detectors may not be reliable when faced with adversarial manipulations of ai-generated texts. in this paper, we stress-test the reliability of llm text detectors. we assume that there is an llm g that generates an output @xmath1. g is protected when there exists a detector f that can detect text normally generated by g with high accuracy. an attack aims to manipulate the generation process such that a new output y is still plausible given input x while the detector fails to identify y as llm-generated. the attack may leverage another attacker llm g . in this context, we propose two novel attack methods. in the first method, we prompt g to generate candidate substitutions of words in y, and we then choose certain substitutions either in a query-free way or through a query-based evolutionary search @xcite to attack the detector. our second method focuses on classifier-based detectors for instruction-tuned llms such as chatgpt @xcite . we automatically search for an additional instructional prompt with a small subset of training data for a given classifier-based detector. at inference time, the additional instructional prompt instructs the llm to generate new texts that are hard to detect. several concurrent studies @xcite proposed to attack detectors by paraphrasing ai-generated texts, with a different language model g for paraphrasing. however, they assume that g here is not protected by a detector. paraphrasing with g has been shown as effective in attacking detectors designed for the original llm g, but it can become much less effective when g is also protected by a detector since the paraphrased model can still be detected by the detectors of g . in contrast, we demonstrate that even when the attacker llm g is also protected by a detector, we can still leverage g for attacking llm detectors. therefore, even if all the strong llms are protected in the future, the currently existing detectors can still be vulnerable to our attacks. we experiment with our attacks on all three aforementioned categories of llm detectors. our results reveal that all the tested detectors are vulnerable to our proposed attacks. the detection performance of these detectors degrades significantly under our attacks, while the texts produced by our attacks still mostly maintain reasonable quality as verified by human evaluation. our findings suggest the current detectors are not sufficiently reliable yet and it requires further efforts to develop more robust llm detectors. methods for red-teaming detectors. as the detectors emerge, several concurrent works showed that the detectors may be evaded to some extent, typically by paraphrasing the text @xcite . however, they need additional paraphrasing models which are typically unprotected models that are much weaker than the original llm. besides paraphrasing, @xcite also discussed attacks against watermark detectors with word substitutions generated by a masked language model such as t5 @xcite which is a relatively weaker language model and tends to generate results with lower quality, and thus it may generate attacks with lower quality. on the other hand, @xcite analyzed the possibilities of the detection given sufficiently many samples. ## settings and overview we consider a large language model g that conditions on an input context or prompt x and generates an output text @xmath2. we use upper-case characters to denote a sequence of tokens. for example, @xmath3, x 2 , . . . , x m ], where m is the sequence length. the model g is protected by a detector f (y) ∈ [0, 1] that predicts whether y is generated by an llm, where a higher detection score f (y) means that y is more likely to be llmgenerated. we use τ to denote a detection threshold such that y is considered llm-generated if f (y) ≥ τ . in this work, we consider three categories of detectors: (1) classifier-based detectors, (2) watermark detectors, and (3) likelihood-based detectors. for classifier-based detectors, a text classifier f (y) is trained on a labeled dataset with g-generated and human-written texts. for watermark detectors, g is modified from a base generator g 0 with a watermarking mechanism w , denoted as @xmath4, and a watermark detector f (y) is constructed to predict whether y is generated by the watermarked llm g. specifically, we consider the watermarking mechanism in @xcite . for likelihood-based detectors, they estimate f (y) by comparing the log probabilities of y and several random perturbations of y. specifically, we consider detect-gpt @xcite . we consider a model g as protected if there is a detector f (y) in place to protect the model from inappropriate usage. to stress test the reliability and robustness of those detectors in this setting, we develop red-teaming techniques to generate texts that can downgrade a detector using an llm that is also protected by this detector. we consider attacks by output perturbation and input perturbation respectively: in both cases, we aim to minimize f (y ) so that the new output y is wrongly considered as humanwritten by the detector f . meanwhile, we require that y has a quality similar to y and remains a plausible output to the original input x. for our attack algorithms, we also assume that the detector f is black-box-only the output scores are visible but not its internal parameters. we propose to attack the detectors in two different ways. in section 4, we construct an output perturbation by replacing some words in y, where we prompt a protected llm g to obtain candidate substitution words, and we then build query-based and query-free attacks respectively to decide substitution words. in section 5, if g is able to follow instructions, we search for an instructional prompt from the generation by g and append the prompt to x as an input perturbation, where the instructional prompt instructs g to generate texts in a style making it hard for the detector to detect. ## attack with word substitutions to attack the detectors with output perturbations, we aim to find a perturbed output y that is out of the original detectable distribution. this is achieved by substituting certain words in y. to obtain suitable substitution words for the tokens in y that preserve the fluency and semantic meaning, we utilize a protected llm denoted as g . for each token in y denoted as y k , we use s(y k , y, g , n) to denote the process of generating at most n word substitution candidates for y k given the context in y by prompting g , and note that not every word can be substituted, and s(y k , y, g , n) can be an empty set if it is not suitable to replace y k . we will discuss how we generate the word substitution candidates using g in section 4.1. ## attack by instructional prompts in this section, we build attacks by perturbing the input prompt to encourage llms to generate texts that are difficult to detect. in particular, we focus on llm-based generative models that can follow instructions and classifier-based detectors. we consider chatgpt (openai, 2023b) as the generative model g and openai ai text classifier (openai, 2023a) as the detector f . the openai ai text classifier is a fine-tuned neural network, while neural networks have been shown to be vulnerable to distribution shifts in nlp literature @xcite . therefore, we aim to shift the generated text to a different distribution where the detector is more likely to fail. we do not require the shifted generation to be semantically equivalent to the original text, but the generation should still be a plausible output to the given input. we achieve this by searching for an additional prompt x p appended to the original input x, which forms a new input @xmath5, x p ] to g. in particular, x p consists of x ins and x ref , where x ins is an instruction asking the model to follow the writing style of reference x ref . searching for x p . we search for x p on a small subset of training examples with n examples x 1 , x 2 , • • • , x n . we assume that we can query the detector f for multiple times during search time. after an effective x p is found, it can be for both of these two steps, we take the best candidate x p according to eq. ( 3 ). when generating g([x i , x p ]) in eq. ( 3 ), we push all the generated outputs to o as the candidates for x ref in the later rounds. we take t iterations and return the final x @xmath6, x ref ] to be used at test time, and for some x p , we find that the g may directly copy text from x ref to generate y when x p is appended into the input prompt. to prevent this behavior, we compute a matching score between x ref and y and discard a candidate x p during the search if more than 20% of words from x ref (except stop words) appear in y . in this way, we find that the copying behavior is effective prevented. baselines. to demonstrate the advantage of our methods in revealing detectors' weakness, we compare with several baselines. dipper paraphrase @xcite is a recent method that trains a paraphrasing model to rewrite ai texts and bypass detectors. it prepends diversity codes to control the level of paraphrases introduced to the texts. we use dipper-paraphraser-xxl with 20 lexical diversity and 60 order diversity to paraphrase the ai texts, which keeps the same level of 20% uni-gram difference as word substitution. we use nucleus sampling @xcite with @xmath7.9 for the paraphraser. and in chat-gpt experiments, we also use chatgpt itself for paraphrasing. datasets. we mainly use two types of datasets including text completion and long-form question answering. we use xsum @xcite for text completion, where we take the first sentence as the input prompt for the completion, and we use eli5 @xcite for long-form question answering. in addition, for the roberta-large detector, we also use a specific gpt-2 output dataset @xcite as they are fine-tuned solely on gpt-2 texts. since the open-ai ai text classifier requires the text to contain at least 1,000 characters, we filter xsum and eli5 datasets and only retain examples with human reference text containing at least 1,000 characters. 10 . for each dataset, we shuffle the test set and use the first 100 examples. metrics. we use several metrics to evaluate the detectors under attacks. area under the receiver operating characteristic curve (auroc) scores summarize the performance of detectors under various thresholds. a detection rate (dr) is the true positive rate under a fixed threshold (positive examples mean llm-generated texts), where we either tune the threshold to meet a particular false positive rate or follow the original thresholds of the detectors. for the gpt-2 output dataset, we also use attack success rate (asr) which computes the rate that the attack successfully flips the prediction by the detector, out of all the positive examples on which the detector originally predicts correctly. ## human evaluation to validate that our approach mostly preserves the quality of the generated text, we conduct a human evaluation on amazon mechanical turk (mturk). on each dataset, we consider the first 20 test examples and ask 3 mturk workers to rate the quality of the text generated by each method on each of the test examples. specifically, we use two metrics, including fluency and plausibility, where fluency measures whether the text is grammatically correct and fluent, and plausibility measures whether the generated text is a plausible output given the input, on either the text completion (xsum) or long-form question answering (eli5) task. we use a 1/2/3 rating scale for each of the metrics (3 is the best and vice versa), and we provide the workers with guidance on the ratings, according to whether there are many/several/ almost no issues for the 1/2/3 ratings on fluency and plausibility respectively. the workers are paid usd @xmath010/hr, which is higher than the us minimum wage ($7.25/hr). tables 11 to 13 show results on attacks against the three detectors respectively. these results show that our attack methods can maintain reasonable and satisfactory plausibility and fluency with a small degradation compared to the unattacked texts. among our attack methods, we find that the query-free substitution usually has better fluency and also sometimes better plausibility compared to the query-based substitution, as the query-based one which aims to search for a stronger attack tends to degrade the text quality slightly more. our method with instructional prompts has better fluency than the query-based substitution and sometimes better fluency than the query-free substitution, and its generation is directly from model g without further substituting words; it also has comparable plausibility compared to the word substitution methods. ## discussions robustness of detectors. comparing the results in tables 4 , 5 , and 10, we see a clear trend that watermarking is relatively more robust to the attacks compared to the other two techniques. the detection mechanism in watermarking is mainly a statistical method and tends to be more robust compared to the likelihoodbased and classifier-based detectors which heavily rely on neural networks. however, watermarking is also the only method here that modifies the generation process of the protected lm. it requires access to the intermediate outputs of the lm and the generation quality may degrade. while the three detectors are not strictly comparable, as watermarking has a different setting by modifying the generation, our results still show insights on the different degrees of robustness of the various detectors. from text quality under the attacks. we note that attack approaches often result in a minor decline in generation performance. nonetheless, based on human assessment, the quality remains acceptable and is adequate to spam the target in a real-world scenario. note that in practical scenarios such as online spamming, malicious actors do not have to use perfect text and they may still use text with slightly degraded quality, as their main purpose is not to generate perfect text but text that is hard to detect. therefore, the insufficient robustness of existing detection strategies continues to be a significant concern. defending against the attacks. inspired by our attack results, we discuss on potential directions for developing methods to defend against the attacks. one possibility is to combine watermarking with a likelihood estimation to defend against word substitution attacks. this is based on the observation that the word substitution attacks often need to substitute around 20% tokens from greenlisted tokens to redlisted tokens. after the word substitution, the new redlisted tokens tend to have lower probabilities in the prediction by the original watermarked model, and the new text also tends to have a higher perplexity under the watermarked model. thus, one may leverage a watermarked language model to check the perplexity or the likelihood of all the redlist tokens, to predict whether a word substitution attack is possibly present. ## conclusion and limitations in this work, we study the reliability of three distinct types of llm text detectors by proposing two attack strategies: 1) word substitutions and 2) instructional prompts using protected llms. experiments reveal the vulnerability of existing detectors, which urges the design of more reliable llm text detectors. we have released the source code and data with bsd-3-clause license at github ( https://github.com/shizhouxing /llm-detector-robustness ). finally, the purpose of this work is to test and reveal the limitations of the currently existing llm text detectors, and we red-team the detectors for future works to improve their robustness and reliability based on our proposed evaluation. thus this work is potentially beneficial to for developing future systems protecting llms and preventing abusive usage. the proposed approaches should not be used to bypass real-world llm text detectors for malicious purposes.