Paper: Can AI-Generated Text be Reliably Detected? (ICLR-2024)

Reviewer comment (Experiments):
The experiments only use text passages from a single dataset, and tend to only evaluate a single model for each detector type. Evaluations on a wider range of datasets and detectors would greatly strengthen any generalizable claims regarding the proposed paraphrasing attack.

Author response:
Thank you for your comment. In our revised Appendix A, we add more results of our attacks with *all* the detectors on different domains – PubMedQA (a medical text dataset) and Kafkai (Deepfake text detection by Pu et al., based on reviewer JvAi’s suggestion) – and models (OPT-1.3B and GPT-2-Medium). Consistent with our previous results in the main paper, we are able to break all the detectors in all the new experimental settings that we consider.

Reviewer comment (Experiments):
Lack of baseline methods. In many of the experiments, the proposed attack is the only method being evaluated, have the authors compared their attack with similar attacks?

Author response:
Our work is the first to show the limitations of 4 different classes of detectors such as watermarking, retrieval-based, zero-shot, and trained detectors. To the best of our knowledge, we are not aware of other baseline attacks that break these detectors. As we discussed in our paper, previous works proposed weaker attacks via span replacements. However, Kitchenbauer et al. (2023) show that watermarking is robust to such attacks. Please find more details regarding our contributions in our Global response.

Reviewer comment (Evaluation):
I appreciate the human evaluation in the "Watermarked AI Text" section, however, this type of evaluation is missing in the other experimental sections. For example, Figure 5 claims a significant drop in detector accuracy with minimal degradation in text quality; however, it's unclear to me how significantly text quality degrades based on a perplexity score increase from 6.15 to 13.55.

Author response:
Thank you for this comment. As reviewer Ce9G notes, the results in Figure 5 were run on a shorter and smaller dataset when compared to the ones in the “Watermarked AI Text” section. Hence, it had a few very short text samples, which resulted in high perplexity scores. To be consistent with the “Watermarked AI Text” section (and obtain improved tradeoffs between our attack performance and text quality), we perform experiments on *all* the detectors with longer and larger datasets in the revision. 

We also modified Figure 5 in the revision based on the new dataset. As shown in the Figure, the perplexity only drops from 7.6 to 9.3 (measured using a larger OPT-13B model), while the detection rate degrades from 100% to below 60%, showing a clearer tradeoff between our attack performance and text quality. We agree with the reviewer that having a complimentary human evaluation can be insightful for all detectors. Unfortunately, performing such large-scale human studies with all the datasets and detectors would be expensive for us. However, we have provided several examples of paraphrased texts in Appendix B.2.  More experimental results are also provided in Appendix A.

Reviewer comment (Experiments):
The insight that smaller TV between AI-generated and human-generated text distributions leads to more difficult detection problems seems rather obvious. Although the authors show that more complex models can lead to smaller TV distances, the authors do not provide any empirical evidence that smaller TV distances actually lead to more difficult AI text detection.

Author response:
Our theoretical result (Theorem 1 and Figure 7) shows that as the total variation between human and AI-generated text decreases, the performance of even the best possible detector also decreases. This implies that smaller TV distances will lead to more difficult AI text detection.

Another way of understanding this phenomenon could be by considering the true and false positive rates (TPR and FPR). For a given FPR (say, 1%), the goal of a detector would be to maximize the TPR. However, the difference between TPR and FPR is bounded by the total variation. Thus, if the total variation decreases, the best-possible TPR also decreases, making detection harder. With this in mind, it is sufficient to empirically show that the TV decreases as AI models become more sophisticated.

Note that showing detection is difficult with respect to a particular detector does not imply the nonexistence of a better detector capable of distinguishing human and AI-generated texts. Our objective is to show a fundamental difficulty in detection. Thus, we first show that detection performance decreases with a decreasing TV (Theorem 1). Then, we provide empirical evidence that TV decreases as AI models become more sophisticated.

Reviewer comment (Evaluation):
There is no empirical runtime evaluation of the proposed attack.

Author response:
Thank you for the comment. We evaluate the time taken for our recursive paraphrase attacks (for 5 rounds) for AI passages of 300 tokens in length to be 36 seconds per passage. We add this to our revised Appendix A.1.

Reviewer comment (Presentation):
Figure 9 is not colorblind friendly.

Author response:
We have updated Figure 9 with a colorblind-friendly color palette.

Reviewer comment (Writing):
How is TV distance defined, and why is it difficult to compute for larger datasets?

Author response:
We use the definition of TV that measures the distance as half of the difference between the probability density functions, i.e., the TV between two distributions P and Q is given by:

TV(P, Q) = $\frac{1}{2}\sum_x |P(x) - Q(x)|$

The difficulty of computing this distance for text distributions does not come from the size of the datasets. Instead, it is due to the size of the sample space. The distribution of text sequences of n tokens from the token set $T$ has the sample space $T \times T \times T \times$ … (n times). The size of this space is $|T|^n$, which is exponential in the sequence length n. In order to compute TV, we need to estimate the probability density for each element in the sample space with a high degree of confidence. This significantly increases the number of samples needed to estimate the TV for larger sequence lengths.

Reviewer comment (Experiments):
How did the authors determine 5 rounds of paraphrasing to be sufficient?

Author response:
The number of recursion rounds is a hyperparameter, and we chose it to be 5 without any specific reason. More rounds of recursion might further deteriorate detection rates with further tradeoffs in text quality. Note that just 2 rounds of recursions are enough to degrade the watermark detection rates to below 50% in all the settings (revised Figure 10, Appendix A.1).

Reviewer comment (Experiments):
The biggest concern is with the para-phrasing task. Should be called the para-phrased text generated by any LLMs? Does it not destroy the inherent characteristics of the generated model? Have the original texts also been paraphrased by the paraphrases? What's the impact of para-phrasing genuine texts?

Author response:
> Have the original texts also been paraphrased by the paraphrases? What's the impact of para-phrasing genuine texts?

We only paraphrase the LLM generations and **not** the original human text. Paraphrases of original human text would be considered AI text since it is generated by a paraphrasing language model. Hence, we do not paraphrase them using a language model to retain them as genuine human texts.

---

> The biggest concern is with the para-phrasing task. Should be called the para-phrased text generated by any LLMs?

The paraphrased text should be considered as AI text by definition since it is output by a language model. For example, if a user queries a watermarked ChatGPT to paraphrase their essay, the paraphrased output will be watermarked and hence detected as AI-generated.

---

 > Does it not destroy the inherent characteristics of the generated model?

Yes, it might destroy the inherent characteristics of the generated model. Intuitively, this might be the reason why our attack works. For example, paraphrasing a watermarked text might remove the watermark patterns, destroying the inherent watermarked LLM text characteristics. This is what an attacker desires.

Reviewer comment (Novelty):
Ablation study of different paraphrasers? What is the contribution here? Do the authors have directly used the existing algorithms for phrasing? With the existence of several studies, the current paper leaves a low contribution concerning the sensitivity of LLM detectors.

Author response:
> What is the contribution here? Do the authors have directly used the existing algorithms for phrasing? With the existence of several studies, the current paper leaves a low contribution concerning the sensitivity of LLM detectors.

After receiving advice from the program chairs, we can note that a draft of our work had appeared on a public platform before the related papers were made public. To keep anonymity, we cannot reveal the name of the platform or give a link to it. Nevertheless, ours is the first work to comprehensively show the limitations of 4 different classes of text detectors – watermark-based, retrieval-based, zero-shot, and trained detectors. Though we use existing paraphraser models, we are the first to propose *recursive paraphrasing attacks* to effectively break the stronger retrieval-based detector by Krishna et al. (2023) and watermark-based detectors. We are the first to present *spoofing attacks* on text detectors that can potentially affect the reputation of LLM developers. We also present novel theoretical results that indicate that reliable AI text detection could get increasingly difficult as LLMs evolve. Please find our contribution details on page 3 (last paragraph).

---

> Ablation study of different paraphrasers? 

To our knowledge, DIPPER by Krishna et al. (2023) is the strongest open-source paraphraser that exists. We perform recursive paraphrase attacks only using DIPPER since weaker paraphrasers will generate texts of lower quality with recursion. We would like to highlight again that our empirical contributions here are the algorithms that we designed for the attacks (recursive paraphrasing and spoofing) and not the paraphraser model.

Reviewer comment (Experiments):
The experimental setting is weak. It is not clear how many samples have been used (2000 or 1000).

Author response:
> The experimental setting is weak. It is not clear how many samples have been used (2000 or 1000).

Thank you for the comment, we have revised the paper to make this clearer. We use a total of 2000 samples, 1000 per human and AI text classes.

We further perform more experiments on all the detectors with larger and longer samples in the revised Appendix A. We consider different domains – PubMedQA (a medical text dataset) and Kafkai (Deepfake text detection by Pu et al., based on reviewer JvAi) – and multiple target LLMs (OPT-1.3B and GPT-2-Medium). We consider 1000 to 2000 samples per experiment in all these settings. Consistent with our previous results, we are able to break all the detectors with a slight tradeoff in text quality.

Reviewer comment (Experiments):
Are these existing detectors trained using multiple augmentation strategies such as data of multiple LLMs and/or para-phrased samples?

Author response:
> Are these existing detectors trained using multiple augmentation strategies such as data of multiple LLMs and/or para-phrased samples?

To the best of our knowledge, the existing open-sourced trained detectors do not use augmented/paraphrased data for training. However, our Corollary 2 presents a fundamental tradeoff between type-1 and type-2 errors. It indicates that as detectors become more robust to paraphrases (by training on paraphrased samples), potentially more human passages will be wrongly flagged as AI text. Thus, reducing type-2 errors might lead to an increase in type-1 errors, which is not desirable.

Corollary 2 also indicates that as AI-paraphrasers become more human-like, this tradeoff will become more significant. Hence, the fundamentally hard task of text detection will become increasingly difficult as AI-paraphrasers evolve.

Reviewer comment (Experiments):
For retrieval-based detectors only 100 samples have been used reflecting inconsistency in the experimental setup. Figure 5 also shows the drastic drop in PPI value.

Author response:
> For retrieval-based detectors only 100 samples have been used reflecting inconsistency in the experimental setup. Figure 5 also shows the drastic drop in PPI value.

Thanks for your comment. We acknowledge that the results in Figure 5 were run on a shorter and smaller dataset when compared to the ones in the “Watermarked AI Text” section. Hence, it had a few very short text samples, which resulted in high perplexity scores. To be consistent with the “Watermarked AI Text” section (and obtain improved tradeoffs between our attack performance and text quality), we perform experiments on *all* the detectors with longer and larger datasets in the revision. 


We also modified Figure 5 in the revision based on the new dataset. As shown in the Figure, the perplexity only drops from 7.6 to 9.3 (measured using a larger OPT-13B model), while the detection rate degrades from 100% to below 60%, showing a clearer tradeoff between our attack performance and text quality.

Reviewer comment (Experiments):
A detailed experimental study is needed concerning experiments in sections 4 (i) and 4 (ii). Only 3-length tokens are used along with a single-layer LSTM network. Further, the decrement shown in Figure 9, is statistically significant?

Author response:
> A detailed experimental study is needed concerning experiments in sections 4 (i) and 4 (ii). Only 3-length tokens are used along with a single-layer LSTM network. Further, the decrement shown in Figure 9, is statistically significant? 

Thanks for the comment. For 4(i), we add the plots for varying sequence lengths in the Appendix (Figure 14b). For 4(ii), Figure 9 shows TV estimates with sequence lengths of 3, 4, and 5. We have modified Figure 9 in the revision with error bars. In all these settings, we consistently observe that the TV estimates reduce as the model gets bigger. We are performing more experiments with different sequence lengths, and vocabulary combinations. However, they are computationally expensive, and hence, we will update the manuscript if we finish our experiments before the discussion period ends. Otherwise, we will add these results to the final version of the paper.

Reviewer comment (Experiments):
The paper does not include testing on a diverse array of datasets like the M4 or Deepfake text detection, which encompasses Multi-generator, Multi-domain, and Multi-lingual data. Incorporating these datasets could provide a more comprehensive evaluation of the paraphrasing model's effectiveness across different text generation sources, domains, and languages.

Author response:
Thank you for your comment. In our revised Appendix A, we add more results of our attacks on different domains – PubMedQA (a medical text dataset) and Kafkai (Deepfake text detection by Pu et al., based on your review). We also evaluate the attacks on two target LLMs – OPT-1.3B and GPT-2-Medium. Consistent with our previous results in the main paper, we are able to break all the detectors in all the new experimental settings that we consider.

Reviewer comment (Evaluation):
Recursively paraphrased text could potentially suffer from semantic drift, where the meaning changes or degrades with each paraphrase iteration. How do you address the concern of maintaining semantic integrity and coherence with only perplexity metrics in text after multiple rounds of paraphrasing?

Author response:
As you rightly note, metrics in NLP, such as perplexity, have their limitations. To evaluate the semantic quality of our recursive paraphrasing framework, we perform MTurk human evaluations. As shown in Section 2.2 (Table 1) and Appendix B.1, the human evaluators scored 70% of our recursive paraphrases to have high-quality content preservation. 89% of the recursive paraphrases were scored to have high text quality or grammar.

Reviewer comment (Evaluation):
Could you elaborate on how perplexity (without semantic understanding) and other quality metrics have been validated to accurately reflect the readability and coherence of the paraphrased text?

Author response:
As you rightly note, metrics in NLP, such as perplexity, have their limitations. To evaluate the semantic quality of our recursive paraphrasing framework, we perform MTurk human evaluations. As shown in Section 2.2 (Table 1) and Appendix B.1, the human evaluators scored 70% of our recursive paraphrases to have high-quality content preservation. 89% of the recursive paraphrases were scored to have high text quality or grammar.

Reviewer comment (Evaluation):
Given that detection methods are constantly evolving, how might adaptive detectors, which are designed to learn and counteract paraphrasing patterns over time, impact the effectiveness of the DIPPER paraphrasing model?

Author response:
This is an interesting question that can be answered using Corollary 2 (Appendix C.2) in our paper. Here we discuss a fundamental tradeoff of AI text detection in the presence of paraphrasing. Corollary 2 indicates that if a detector becomes more robust to AI paraphrasing, type-I errors will increase, and more human passages will be wrongly flagged as AI text by the detector. This shows a tradeoff between type-1 and type-2 errors of AI text detectors.

Corollary 2 also indicates that as AI-paraphrasers become more human-like, this tradeoff will become more significant. Hence, the fundamentally hard task of text detection will become increasingly difficult as AI-paraphrasers evolve.