Paper: On the Possibilities of AI-Generated Text Detection: A Sample Complexity Analysis (ICLR-2024)

Reviewer comment (Theory):
The theoretical analysis in the paper has a nice overall takeaway, but I am not sure how helpful the bounds are for practitioners who are developing these algorithms / trying to detect AI-generated text. Specifically, how does one go about computing the variation between machine-generated text and human-written text (`TV(m,h)^n`) for some SoTA LLMs like ChatGPT or GPT4, perhaps in a restricted domain? This is needed in both proposition 1 (for AU-ROC upperbound calculation), and in Theorem 1 (for sample estimation). I wonder if the authors could walkthrough an example in their paper: calculate `TV(m,h)^n` for ChatGPT for different output lengths, and then provide the corresponding upperbounds for AU-ROC for different values of the output length. I wonder if the method used in the MAUVE text evaluation paper (https://arxiv.org/abs/2102.01454) may be helpful for this calculation, who propose a clever approximation for measuring the divergence between human/machine-written text.

Author response:
Thanks for this comment and we agree that it is indeed very important to discuss the significance and practical implications of our result. We attempt to provide certain insights of our results in Appendix A (remarks 1, 2, 3). Here we highlight how practitioners can leverage our results in designing reliable and more robust watermarks & detectors (currently susceptible to attacks) with information gain and human-machine distribution distance, and also discuss on the task-specific detectability with our results. We give two examples for simplicity

Watermark Design: Our analysis suggests adaptive watermark designs based on Chernoff information and sample complexity, enhancing robustness against attacks—a future research avenue.

Length Constraints in Exams: Our bounds inform optimal summary lengths for easier cheating detection, leveraging the Chernoff metric.

However, we agree that a more specific and detailed discussion on the practicality will be critical and we will add the same in our updated draft.

Reviewer comment (Theory):
The theoretical analysis bounds AU-ROC, but is it possible to derive a bound on the true-positive rate (TPR) for a fixed false-positive rate (FPR), instead of a bound on AU-ROC? This may be a more practical metric, since practitioners will want to optimize TPR for a fixed low FPR.

Author response:
This is a very interesting point. We agree that computing the total variation distance in exactness for a real detector for text generated from LLMs or humans is hard and hence we evaluate the performance w.r.t AUC similar to what was used in prior works on detection (Sadasivan et. al, Krishna et. al, Mitchell et al.). 

However, it is important to note that we do validate the AUC upper-bound (in exactness, missing from concurrent works) for the best detector with an n-gram bag of words based on discretized feature space in Figure 2a. Since, in a finite discretized feature space, TV distance can be computed using the definition [1] and then we plot the AUC upper bounds by varying the value of n in n-grams as in Figure 2a. 

On the other hand, the MAUVE metric leverages a modified KL/JS type divergence as in Equation 2 (Pillutla et al 2021), which can be an alternate measure of divergence replacing TV distance. Thanks for this very interesting suggestion and we agree that due to the piecewise constant approximation of the human (machine) distributions as done in Equation 3 (Mauve paper), we can estimate the KL divergence (or TV distance) under some approximations (depending on the clustering), and is a valid scope of future research. Additionally, one can also show theoretically by replacing TV with KL using Pinsker's inequality and obtaining the equivalent upper bound like ours in Theorem 1,2.

Reviewer comment (Theory):
I was wondering how the retrieval-based defense from Krishna et al. 2023 (https://arxiv.org/abs/2303.13408) fits in the theoretical analysis of this paper. While I appreciate the discussion in Appendix A, I think the analysis in both Sadasivan et al. 2023 and this paper does not directly apply to retrieval-based detection. If I understand the retrieval-based detection algorithm correctly, it does not leverage the difference in properties between `m(s)` and `h(s)`. In other words, I think retrieval-based detection will work with `h_1(s)` and `h_2(s)` as well, where `h_1` and `h_2` are two identically distributed human writers. Is it fair to say that in retrieval-based detection, the support (`S` in this paper's 3.1) of `m(s)` and `h(s)` is almost disjoint, and hence the `TV(m,h)` will be quite high / infinity? With a bigger database there may be a few collisions between `m(s)` and `h(s)`, so `TV(m,h)` will slightly drop?

Author response:
This is an extremely critical and important point and we want to thank the reviewer for highlighting this specifically. Indeed our theoretical and empirical results are much more general and not just restricted to the setting requiring iid samples. Our Theorem 2 extends to the more general non-iid setting where we relate the sample complexity with the sequence length. As the reviewer correctly pointed out, the majority of our experimental results (Figure 3a-f, 4, etc) are indeed for the non-iid settings and validate our theoretical findings. We absolutely agree with the reviewer that we need to emphasize more on Theorem 2 i.e. sample complexity with sequence length and will update our draft accordingly.

Reviewer comment (Novelty):
My main concern is that the results in this paper are general facts about sample complexity and not specific to detection of AI-generated text. For example, Theorem 1 is included here: https://github.com/ccanonne/probabilitydistributiontoolbox/blob/master/testing.pdf.

Author response:
**Response to Weakness 1** We thank the reviewer for the reference but there seems to be a confusion regarding the novelty of our work. We note that the upper bound in the Total variation distance of the distribution (product distribution) is deep-rooted in the fundamentals of Information theory more specifically Lecam's lemma (as clearly highlighted at multiple places in our paper such as in the start of Section 3.2, before Equation (9),  and Appendix B.1). We never claim the novelty regarding the same, whereas ***our main novelty*** lies in deriving a precise sample complexity bound in the context of AI-generated text detection for both iid and non-iid scenarios, which is novel.

***Our Results are for AI-generated Text Detection (as acknowledged by [Reviewer 6pNs](https://openreview.net/forum?id=oxEER3kZ9M&noteId=9zofG5ezim)):*** We illustrate our results precisely in the context of LLMs through several instances, provide key insights and also empirical evaluation ***in the text domain***. For example in Section 3.4, we develop the ***novel connection between non-iid sample complexity and increasing sequence length in generation***, illustrate the implication of $\sum_k s_k$ (Equation 16) and novel insights on dependence of topics/contexts present in the paragraph with sample complexity results and empirical evaluations with several SOTA generators and detectors. We want to highlight that our results and comparisons are inline with the recent results in the field of AI-generated text detection (Sadasivan et. al, Krishna et. al, Mitchel et al etc.)

***Clarification Regarding the use of the term 'multiple samples':*** Our result does not depend on the specific choice of the unit (for samples) to represent i.e., it can be a bag of words, sentences, or paragraphs, but rather emphasizes on the impact of product distribution on the sample complexity. ***To highlight the point we empirically demonstrate*** with several such instances like bag of words (Figure 2), sentences (Figure 3, 4), etc.

Reviewer comment (Novelty):
As this paper is outside of my area, I am less confident about the novelty of Theorem 2, but its formulation is not dependent on the domain of AI-generated text detection.

Author response:
**Response to Weakness 2** We ***respectfully disagree with the reviewer*** that our experiments replicate the known results. ***Our work is the first*** to show (with empirical experiments) a precise connection of the detection performance with the sample complexity (no. of samples (Theorem 1), length of sequence (Theorem 2)) in the context of LLMs, which has been a critical point of discussion in the recent times. We are aware of the concurrent works for this problem and cannot provide further details as it will violate anonymity. We will provide more information to the AC. 

Regarding the point on the controlled experiment, we believe we have the experimental results in similar settings for example in Figures 2b, 2c. Figure 2c (iid box plot) represents the scenario with pairwise samples (sequences) as input to the detector and Figure 2b represents the scenario with varying sequence lengths validating Theorem 1 and 2 respectively. It is evident that even with just pairwise samples, the detector performance is very good achieving an AUC of 95% which exactly validates our Theorem.

Reviewer comment (Writing):
It is unclear from the paper whether the “multiple samples” required by Theorem 2 are meant to be individual words, sentences, or documents.

Author response:
**Response to Weakness 1** We thank the reviewer for the reference but there seems to be a confusion regarding the novelty of our work. We note that the upper bound in the Total variation distance of the distribution (product distribution) is deep-rooted in the fundamentals of Information theory more specifically Lecam's lemma (as clearly highlighted at multiple places in our paper such as in the start of Section 3.2, before Equation (9),  and Appendix B.1). We never claim the novelty regarding the same, whereas ***our main novelty*** lies in deriving a precise sample complexity bound in the context of AI-generated text detection for both iid and non-iid scenarios, which is novel.

***Our Results are for AI-generated Text Detection (as acknowledged by [Reviewer 6pNs](https://openreview.net/forum?id=oxEER3kZ9M&noteId=9zofG5ezim)):*** We illustrate our results precisely in the context of LLMs through several instances, provide key insights and also empirical evaluation ***in the text domain***. For example in Section 3.4, we develop the ***novel connection between non-iid sample complexity and increasing sequence length in generation***, illustrate the implication of $\sum_k s_k$ (Equation 16) and novel insights on dependence of topics/contexts present in the paragraph with sample complexity results and empirical evaluations with several SOTA generators and detectors. We want to highlight that our results and comparisons are inline with the recent results in the field of AI-generated text detection (Sadasivan et. al, Krishna et. al, Mitchel et al etc.)

***Clarification Regarding the use of the term 'multiple samples':*** Our result does not depend on the specific choice of the unit (for samples) to represent i.e., it can be a bag of words, sentences, or paragraphs, but rather emphasizes on the impact of product distribution on the sample complexity. ***To highlight the point we empirically demonstrate*** with several such instances like bag of words (Figure 2), sentences (Figure 3, 4), etc.

Reviewer comment (Novelty):
The experiments also replicate known results, namely, that AI-generated text detectors perform better on longer sequences and that they perform worse with paraphrasing.

Author response:
**Response to Weakness 2** We ***respectfully disagree with the reviewer*** that our experiments replicate the known results. ***Our work is the first*** to show (with empirical experiments) a precise connection of the detection performance with the sample complexity (no. of samples (Theorem 1), length of sequence (Theorem 2)) in the context of LLMs, which has been a critical point of discussion in the recent times. We are aware of the concurrent works for this problem and cannot provide further details as it will violate anonymity. We will provide more information to the AC. 

Regarding the point on the controlled experiment, we believe we have the experimental results in similar settings for example in Figures 2b, 2c. Figure 2c (iid box plot) represents the scenario with pairwise samples (sequences) as input to the detector and Figure 2b represents the scenario with varying sequence lengths validating Theorem 1 and 2 respectively. It is evident that even with just pairwise samples, the detector performance is very good achieving an AUC of 95% which exactly validates our Theorem.

Reviewer comment (Experiments):
As far as I can tell, the main novel finding is the “pairwise with IID samples” condition (Figure 2c). It would be interesting to see a more controlled experiment here, which compares the performance of detectors on two IID documents of lengths X,Y versus one single document of length X+Y.

Author response:
**Response to Weakness 2** We ***respectfully disagree with the reviewer*** that our experiments replicate the known results. ***Our work is the first*** to show (with empirical experiments) a precise connection of the detection performance with the sample complexity (no. of samples (Theorem 1), length of sequence (Theorem 2)) in the context of LLMs, which has been a critical point of discussion in the recent times. We are aware of the concurrent works for this problem and cannot provide further details as it will violate anonymity. We will provide more information to the AC. 

Regarding the point on the controlled experiment, we believe we have the experimental results in similar settings for example in Figures 2b, 2c. Figure 2c (iid box plot) represents the scenario with pairwise samples (sequences) as input to the detector and Figure 2b represents the scenario with varying sequence lengths validating Theorem 1 and 2 respectively. It is evident that even with just pairwise samples, the detector performance is very good achieving an AUC of 95% which exactly validates our Theorem.

Reviewer comment (Reproducibility):
Could you provide additional details on the process used to generate Figure 2a? How were documents tokenized and how was AUC computed? I am also confused by the statement in Section 4.1 that this analysis “approaches sentence to paragraph level” while the maximum value of n seems to be 6 (which is shorter than most sentences, let alone paragraphs).

Author response:
**Response to Question 1:** To generate Figure 2a, we discretize the sequences (paragraphs) from the Xsum or Squad dataset using bag of words features varying the ngrams from n = 1 to 6. For each of the scenarios, we compute the AUC of the best detector using equation 13 (similar to Sadasivan et. al Figure 7). It is important to note that in the finite discretized feature space, TV distance can be computed using the definition [1] and then we plot the AUC upper bounds by varying the value of n in n-grams as in Figure 2a. The key message (via Figure 2a) is to demonstrate the significant improvement in AUC with increasing n-grams validating our Theorems.


***Regarding the statement in Section 4.1***, we meant to say that as the token length increases (word to sentence, and sentence to paragraph, and so on), it results in a significant increase in AUROC, hence better detectability as observed in Figure 2. Also, to clarify with ngrams = 6, it means considering bag of 6 words as each feature and this doesn't represent the number of features, which is different (num_features = 100). More specifically, it means that each feature is a collection of n = 6 words (almost like a sentence) and we can see a significant improvement in AUC of 95% (achieved by the best detector). We have also experimented with n = 7, 8, and 10 bag of words-based features and observed nearly 98% AUC achieved by the best detector. Hope this clears the confusion.

We will clarify this and update the discussion (providing more details about n-grams and features) in the revised version of our paper. 

We believe that our responses have addressed your concerns. It is our pleasure to provide additional details if needed. If your concerns have been resolved, we kindly request you to consider increasing the scores.

[1]. Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, Gert R. G. Lanckriet. On integral probability metrics, ϕ-divergences and binary classification, https://arxiv.org/abs/0901.2698

Reviewer comment (Evaluation):
Interesting method and conclusion; however not sure how much we can connect this with MGT detection. It seems like the conclusion is generally applicable. This means the proposed method itself is not necessarily a weakness, but the disconnection is.

Author response:
We are thankful that you find our formulation and methodology interesting. 

Regarding the generality of the results, we agree that certain theoretical results for example used in Eq 6, 7, and 9 have their roots in Information theory and Large deviation theory ***but a precise connection to the context of AI-generated text detection was missing from literature.*** Specifically, our sample complexity analysis for iid samples (Theorem 1) and non-iid samples (Theorem 2) are novel and provide a ***first step towards showing possibility*** results (in a theoretically rigorous manner) of MGT detection. 

We illustrate our results precisely in the context of LLMs through several instances, key insights, and empirical evaluation. For example in Section 3.4, we develop the connection between non-iid sample complexity and increasing sequence length in the generation, illustrate the implication of $\sum_k s_k$ (Equation 16), and novel insights on the dependence of topics/contexts present in the paragraph with sample complexity results and empirical evaluations with several SOTA generators and detectors.

Reviewer comment (Novelty):
Equation 7 is not surprising. As pointed out by Sadasivan et al. (2023), it is impossible to get a reliable detector (high TPR, e.g., 90% and lower FPR, e.g., 1%), when the overlap of two distributions is relatively small.

Author response:
We agree that Equation 7 trivially follows and it comes from an application of Lecam's Lemma from Information Theory which we have clearly cited (and not first appeared in Sadasivan et al. (2023)). But we respectfully point out that there seems to be a slight confusion here. Our point is not about Equation (7). We would like to highlight the **hidden possibility** part which starts around Equation (8) (which is different from Sadasivan et al. (2023)).

**Our Focus:** Our result in this paper focuses on Equation (9), which helps to conclude that even if the overlap of distribution in Equation (7), is small, increasing the samples/sequence length helps to increase the TPR because TV norm increases exponentially with respect to $n$ as highlighted in (12). This is rigorously connected with bounds developed in Theorem 1 (iid) and Theorem 2 (non-iid) cases.

Reviewer comment (Theory):
Furthermore, equation 7 does not depend on sample complexity. So, I am not sure I understand how increasing the number of examples can get around this issue pointed out by the previous comment.

Author response:
Equation 7 is mentioned to lay the foundation of our discussion but our results follow from the expression in Equation (9) which clearly depends upon the values of $n$ (samples/sequence length). We apologize for the confusion, we will highlight this fact in the revised version of our manuscript.

Reviewer comment (Evaluation):
More generally, the real challenge of detecting MGT is the lack of information on generative models. In practice, it’s hard to predict whether a collection of texts is from the same generation model. Furthermore, a gray area of this research question: how we should treat the texts generated by machines and edited by humans.

Author response:
This is a very interesting point. We agree that texts generated by machines and edited by humans is a very challenging setting and is very hard to detect in that case. However, ***our result provides a first step to rigorously show the possibility of detection*** where the text comes from either human/machine distribution. However, texts generated by machines and edited by humans are an extremely important and valid source of future research.

Reviewer comment (Writing):
What does “exponentially fast” mean in the caption of Figure 1?

Author response:
In Figure 1, the value of Total variation distance (best detector) increases at an exponential rate as a function of the number of samples, which is also evident from our theoretical analysis as shown in Equation 12. In equation 12, it is evident that as we increase n, TV exponentially approaches 1.

Reviewer comment (Experiments):
Are there any experiment results on non-IID data?

Author response:
***Yes, We want to highlight that all our experimental*** ablations including Figures 2 (a,b) 3 (a->f) ***are for non-iid scenarios only***, as you can see that the performance (AUC) is measured with increasing sequence length. This is exactly what we discuss in Theorem 2, when we increase the sequence length (which is equivalent to having more dependent samples) the detection performance varies, which is novel and not covered in past literature. 
We apologize if this point didn't come out clearly and will be updating the draft with a specific focus on this point as also highlighted by Reviewer 6pNs: "I feel like the authors are under-selling themselves in the title/abstract/introduction, by focusing on the "number of samples" rather than "sequence length"

Reviewer comment (Theory):
Maybe I missed something from the paper, but is there any sample complexity directly related to equation 15?

Author response:
Equation 15 represents the sample complexity results for the iid case (Theorem 1) which states that to achieve an AUROC of $\epsilon$, we will need $O(\frac{1}{\delta^2} \log \frac{1}{1 - \epsilon})$ samples (iid case, non-iid in Theorem 2). This precisely represents the sample complexity of detection i.e how the detection performance varies (in terms of AUC) with number of samples or sequence length. Hope this clears the confusion.