Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks (2024)

Abstract: The widespread use of large language models (LLMs) is increasing the demand for methods that detect machine-generated text to prevent misuse. The goal of our study is to stress test the detectors’ robustness to malicious attacks under realistic scenarios. We comprehensively study the robustness of popular machine-generated text detectors under attacks from diverse categories: editing, paraphrasing, co-generating, and prompting. Our attacks assume limited access to the generator LLMs, and we compare the performance of detectors on different attacks under different budget levels. Our experiments reveal that almost none of the existing detectors remain robust under all the attacks, and all detectors exhibit different loopholes. Averaging all detectors, the performance drops by 35% across all attacks. Further, we investigate the reasons behind these defects and propose initial out-of-the-box patches.

## introduction llms are becoming increasingly adopted in information seeking scenarios, assistive writing, translation, mental health support, and many more @xcite . their evolving capabilities to generate human-like and persuasive language raise wide concerns about misuse, e.g., deception, academic misconduct, and disinformation @xcite @xcite , and it becomes harder for humans to distinguish machine-generated texts (mgt) from human-written texts (hwt) @xcite . as a result, much recent work focus on automatic mgt detection to mitigate the risks @xcite @xcite . in this work, we focus on potential malicious attacks that attempt to deceive the detector using various attack strategies. existing works on this topic mostly focus on the robustness of specific detectors or particular attack methods. for example, @xcite specifically evaluate the token editing attack for model-based detectors, and @xcite consider the topic-shifting attack for metricbased detectors, etc. to the best of our knowledge, in the literature, there is no thorough comparative evaluation of robustness of machine-generated text detectors against malicious attacks, covering a wide range of detectors and attacks. with this goal, we study the robustness of 8 prevalent mgt detectors from 3 categories under 12 realistic attacks ( §6, we build a robustness leaderboard ( ## generators and detectors we select gpt-2 xl (1.5b) @xcite , gpt-j (6b) @xcite , and llama-2 (7b-hf) @xcite as the representative open-source generators, and text-davinci-003 (openai, 2022b) and gpt-4 (openai, 2023) as the closed-source generator representatives. all the generators shared similar results under attacks (appendix f.3). we select gpt-j (6b) as the default generator to show the results in §6 if unspecified (we empirically find metric-based detectors do not perform well on stronger generative lms even without attacks). the results of llama-2 and gpt-4 will be additionally shown in appendix f.3 and §6. for closed-source generators, some of the detectors can not be applied due to the requirement of white-box parameters. current mgt detectors could be classified into 3 high-level categories, as we introduce below. we include representative detectors from each category for our evaluation. a detailed introduction of the detectors is deferred to appendix b.2.1. metric-based detector relies on the inferred logprobability from the generator llm, and adopts a threshold for classification. 4 detectors for this type do not require any training. we include gltr @xcite , rank and logrank @xcite , and detectgpt @xcite as representative approaches in the category. fine-tuned detector is trained on a pretrained language model (plm) in a supervised method with a classification loss. we include openai detector @xcite , simpleai detector @xcite , and fine-tuned deberta as representative models in the category. watermark-based detector adds algorithmically detectable signatures into texts during generation. @xcite is a representative approach, which adds a token-level bias in the decoding stage (represented as watermark afterward). we follow the recommended configurations for most detectors. detailed hyperparameters are included in the appendix b.2.2. ## budget of attacks as stated in §2, to measure the perturbation level of attacks on the generated texts, we utilize a series of text generation evaluation metrics as the budget of attacks, covering syntactic-or semantic-level perturbation. a strong attack should induce large detection performance degradation with a relatively small budget. for the editing attacks, we use levenshtein edit distance @xcite as the major budget, which is the minimum number of single-character edits, including insertions, deletions, and substitutions. a larger distance represents a larger attack budget. additionally, we also record jaro similarity @xcite ). 5 to measure the quality of texts under the attacks that change the semantic meaning (e.g., prompting attacks and co-generating attacks), we utilize perplexity under llama-7b-hf @xcite and mauve @xcite . we use mauve to compare the distribution gap between mgts and hwts. mgts are used to estimate the model distribution, and hwts are used to estimate the target distribution (the setting is abbreviated as 'm2h'). lower perplexity or higher mauve (m2h) represents better quality and a smaller budget. for the attacks that do not change semantics meaning, e.g., paraphrasing, we use bertscore @xcite as the major metric for the budget. we utilize it to compare the similarity between mgts after the attack to mgts before the attack. in this scenario, attacked mgts are the candidates for bertscore, while unattacked mgts are the reference (the setting is abbreviated as 'a2b'). the bertscore we used is rescaled. a larger bertscore (a2b) value means a smaller budget in the attack. besides, we also record bartscore @xcite and cosine similarity, which shows equivalent results. see metrics for detector performance. the metrics we use to evaluate detection performance are binary classification metrics auc roc and tpr@fpr. auc roc is the area under the receiver operating characteristic curve. tpr@fpr is the true positive rate when the false positive rate is at a specific percentage. under our setting, it is equivalent to attack success rate (asr) @xcite 7 . we mainly show tpr@@xmath0, and tpr@@xmath1.2. we do not involve accuracy and f1-score because those metrics depend on the threshold setting for metric-based detectors, which could be biased in the comparison. 6 the x-ticks in format (zws) character editing is twice the ones in typo and homoglyph because the unicode is 2 bytes when computing edit distance. 7 under our settings, we define the success of the attack is to deceive detectors by classifying machine-generated text into human-written (described in §2). so, (1-tpr) and asr are both evaluating the equivalent thing. ## attacks and results in this section, we describe the attack methodologies and results divided by attack category. we view the degraded performance under attacks of various detectors as stumbling blocks to robust mgt detection. further, we analyze the defects and propose defense patches in each category to explore the potential of turning stumbling blocks into stepping stones. ## future work we propose ideas of enhancement for detectors in appendix d.1 and three stronger attack categories, namely sampling attacks, fine-tuning attacks, and human-involved attacks in appendix d.2. ## related work to the best of our knowledge, there is no existing thorough study on stress testing the robustness of machine-generated text detection under various attacks. some existing works of mgt detectors evaluate their robustness under some specific attacks. liu et al. (2022) to summarize, a thorough and comparative stress test on the robustness covering a wide range of detectors and attacks is lacking in the literature, which motivates our work. ## conclusion this study evaluates the robustness of 8 mgt detectors against 12 realistic attacks, revealing striking vulnerabilities. findings show that no detector consistently withstands all attacks, as some attack strategies severely compromise detection accuracy. among various detectors, watermarking is the most robust, followed by model-based detectors. we also suggest combining metric-and model-based detectors for better resilience. aiming at robust mgt detection, we call for awareness of vulnerability and the need for further methods.