Paper: DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text (ICLR-2024)

Reviewer comment (Theory):
While the work addresses the pressing problem of understanding the structure of GPT/LLM and presents interesting observations, the authors seem explicit about not caring about execution quality. For example, Delta should be lower bounded but not upper bounded in order to justify using KL divergence/Total variation; this appears to be a fixable issue by providing the other side of the inequality.

Author response:
**We respectfully disagree with this claim**. Execution quality is our top priority: From TabIe 1, our method achieved SOTA results on 4 datasets and 3 models from OpenAI in both black- and white-box scenarios on 16/16 cases regarding AUROC and 13/16 cases regarding TPR.\
Table 3 also confirmed the superior detection quality of our methods on 2 advanced open-sourced models (LLaMa, GPT-NeoX), surpassing baselines in 8/8 cases on AUROC.\
Thus, we are confident that the execution quality is our top priority, and our experimental results demonstrate our method's much higher detection quality over baselines.

Reviewer comment (Experiments):
A major limitation is if we don't know which model is used to generate the text. In practice, one is often interested in determining whether a block of text is from some LLM at all, not specifically from a collection of 5–10 widely available models.

Author response:
We would like to clarify that **we are following the same task-setting as in previous work [1, 2]**: the requirements for known LLM are also the problem settings of the two most famous detection approaches: zero-shot detector DetectGPT [1] and Watermark [2].\
Additionally, as we wrote in the original submission on page 3, “Refer to Appendix B.4 for unknown source model”, **we actually included the experiments for detecting text from unknown sources**. To the best of our knowledge, there are no existing good solutions to detect text from unknown sources robustly. Our paper contributes to this challenging problem by using proxy models in Appendix B.4 to make a step forward. To be specific, we use two proxy models (OPT-125M and GPT2-124M) to estimate the relative probability difference and achieve good performance. So **this is also our unique contribution rather than weakness since previous baselines can not handle this**.\
Furthermore, on page 9, we have a paragraph titled “Model Sourcing” to tackle the problem of unknown LLM sources.  To be specific, we can detect the model source given some model candidates through pair-wise comparison by applying our method to those candidates and then rank the scores. Table 2 shows that our method can reliably detect model sources from a collection of 4 models with 85%-99% AUROC score. In comparison, none of the baselines can perform model sourcing. Therefore, **this is our additional unique contribution since previous baselines can not solve it.**

[1] Mitchell, Eric, et al. "Detectgpt: Zero-shot machine-generated text detection using probability curvature." (ICML 2023).\
[2] Kirchenbauer, John, et al. "A watermark for large language models." (ICML 2023).

Reviewer comment (Writing):
The current logic in the paper flows as "introduce llh gap → llh gap implies KL large → drop KL because it is hard to compute/estimate and return to llh gap"; this reasoning could be clearer, though the rebuttal text helped clarify the misunderstanding.

Author response:
Yes, exactly. Thanks for the suggestion! We will make this more understandable in our revision.

Reviewer comment (Evaluation):
It would be helpful to have more discussion on why KL estimation is harder than llh gap, since both relate human and machine distributions (e.g., llh gap requiring smaller samples from humans).

Author response:
Thanks for the quick feedback and suggestion!\
NO. The kl divergence is not a tester. This kl divergence is intractable for each input text since $\Delta$ can not be exactly computed, as you also mentioned. Instead, we use it as a theoretical guide to help us empirically detect text.\
As we mentioned, the $\Delta$ is relevant to different training mechanisms, data, model size, model architecture, alignment, etc. Thus, inherited from the DetectGPT hypothesis, we assume such a gap is large enough for us to distinguish between human and AI-written text. But in practice, we have to turn to alternative calculations, such as our proposed Bscore and Wscore, showing impressive performance over previous baselines, as reported in Table 1. Based on our experimental observation, like Figure 2, and the outperforming performance in Table 1, we believe our hypothesis is reasonable since it can explain our observed empirical results. For example, the detection effectiveness consistently improves as we increase the number of re-generations $N$ in Figure 4, validating our theoretical result in A.2 PRINCIPLED CHOICE OF K. On the other hand, if we can directly compute the kl divergence, we can perfectly detect any text without needing any detection algorithm. But it is impossible. Thus, we believe our method is solid and effective both empirically and theoretically compared with previous baselines. This is also admitted by reviewer 4FGJ: "The proposed method is simple and effective", reviewer UDX7: "The results are strong" and your kind comment "a simple and neat technique".\
So, we believe our theoretical and empirical results echo harmoniously, providing a timely solution to the urgent issue of text detection. This is particularly suitable for our ICLR submission track of societal considerations including fairness, safety, privacy.

Reviewer comment (Experiments):
Is it possible to fix the Delta?

Author response:
**Response:** We would like to first thank the reviewer for raising this interesting question.\
We believe our results are already very strong compared with baselines since our experiments validate the effectiveness of our method on both closed models like text-davinci-003, GPT-3.5-turbo, and GPT-4, as well as open-source models such as GPT-NeoX-20B and LLaMa-13B. Thus, it is evident that it holds for those representative models.\
However, in terms of “so long as an LM is trained through certain mechanism, it will always exhibit the likelihood gap hypothesis and the possibilities of fixing the Delta.”, we believe that it would be a complicated question because **it involves not only the training mechanism but also factors that might be relevant to the training data, model size, model architecture, and alignment, among others**. We would love to extend our results to more models in the future.\
Considering the popularity of those models, especially ChatGPT, we believe that our approach provides a timely solution to this problem. 

Besides, we also noticed that **on the OpenAI dev day, the former CEO of OpenAI announced [3] that they would give token logits access on GPT-4 API calls in the future. This will make our white-box detection method also applicable to GPT-4! Although we have not got access to GPT-4 output token logits, we believe this will only make our method more powerful in the future.**

[3] https://openai.com/blog/new-models-and-developer-products-announced-at-devday

Reviewer comment (Theory):
Is it possible to build a stronger result, for example, showing that so long as an LM is trained through a certain mechanism, it will always exhibit the likelihood gap hypothesis?

Author response:
**Response:** We would like to first thank the reviewer for raising this interesting question.\
We believe our results are already very strong compared with baselines since our experiments validate the effectiveness of our method on both closed models like text-davinci-003, GPT-3.5-turbo, and GPT-4, as well as open-source models such as GPT-NeoX-20B and LLaMa-13B. Thus, it is evident that it holds for those representative models.\
However, in terms of “so long as an LM is trained through certain mechanism, it will always exhibit the likelihood gap hypothesis and the possibilities of fixing the Delta.”, we believe that it would be a complicated question because **it involves not only the training mechanism but also factors that might be relevant to the training data, model size, model architecture, and alignment, among others**. We would love to extend our results to more models in the future.\
Considering the popularity of those models, especially ChatGPT, we believe that our approach provides a timely solution to this problem. 

Besides, we also noticed that **on the OpenAI dev day, the former CEO of OpenAI announced [3] that they would give token logits access on GPT-4 API calls in the future. This will make our white-box detection method also applicable to GPT-4! Although we have not got access to GPT-4 output token logits, we believe this will only make our method more powerful in the future.**

[3] https://openai.com/blog/new-models-and-developer-products-announced-at-devday

Reviewer comment (Theory):
The entire framework is based on the likelihood-gap hypothesis. What is the basis for this assumption? Is this formally acknowledged somewhere else in the literature or verified empirically in this paper (e.g. from Fig 2)?

Author response:
We would like to thank the reviewer for the positive feedback on our work. Below are our responses.

1. The entire framework is based on the likelihood-gap hypothesis. What is the basis for this assumption? Is this formally acknowledged somewhere else in the literature or verified empirically in this paper (e.g. from Fig 2)?

**Response**: We have both theoretical and experimental bases for this hypothesis.\
**Theoretically**: In the baseline work of DetectGPT [1] published on ICML 2023, they also have a likelihood-gap hypothesis (on page 3, section 4): the perturbation discrepancy distance of the machine is positive and large, while it is close to 0 for humans, which means that there is always a positive gap between the human and machine. Therefore, our hypothesis is inherited from theirs.

**Experimentally**: It is also empirically verified by Fig 2, where we can see a clear gap between those two distributions of human and machine. Additionally, our comprehensive experiments in Tab 1 further confirm it by showing that our method achieves SOTA results on 4 datasets and 3 models from OpenAI in both black- and white-box scenarios on 16/16 cases regarding AUROC and 13/16 cases regarding TPR. 

[1] Mitchell, Eric, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. "Detectgpt: Zero-shot machine-generated text detection using probability curvature." (ICML 2023).

Reviewer comment (Experiments):
The paper mentions that is able to provide explanations and evidence in the form of n-grams overlaps. Would be good to provide some examples.

Author response:
2. The paper mentions that is able to provide explanations and evidence in the form of n-grams overlaps. Would be good to provide some examples.

**Response**: **We indeed provided examples in our original submission** since this is the unique advantage of our detector over other baselines. On page 8, we had a whole paragraph titled ‘Explainability’ to talk about the explanations and evidence. In Appendix D, we also had the additional tables 14, 15, and 16 for explanations and evidence.\
Specifically, one example in Figure 1 shows that the typical overlapped n-grams are: 
The original $y_0$ contains the following sentence: “le analyses that focus on specific neighborhoods or regions may reveal disparities that are not apparent in larger-scale analyses. Therefore, it is important to consider the scale of analysis when”. Among 20 re-generations, we found the following two typical examples with significant overlapped n-grams:\
In $y_1$, the overlapped n-gram is “le analyses that focus on specific neighborhoods or regions may reveal disparities that are not apparent in larger-scale analyses“\
In $y_5$, the overlapped n-gram is “may reveal disparities that are not apparent in larger-scale analyses. Therefore, it is important to consider the scale of analysis when”\
We hope this example of overlapped n-grams helps. More examples can be found in Appendix D.

Additionally, we also noticed that on **the OpenAI dev day, the former CEO of OpenAI announced [2] that they would give token logits access on GPT-4 API calls in the future. This will make our white-box detection method also applicable to GPT-4! Although we have not got access to GPT-4 output token logits, we believe this will only make our method more powerful in the future.**

[2] https://openai.com/blog/new-models-and-developer-products-announced-at-devday

Reviewer comment (Theory):
While the proposed method is effective, it is not clear how certain terms in the score function are selected—e.g., the f(n) function in BScore. Can authors comment more on that?

Author response:
For BScore, we aim to make it simple and effective. The basic idea is to assign higher weights to overlapped N-grams with larger N since it would be more unusual for the machine to generate overlapped N-grams with larger N. As we wrote, “In practice, we set f(n)=n log(n), … More comparisons on parameter sensitivity can be found in Appendix B.” In Appendix B.8, we demonstrate the effects of 6 different weight functions on additional validation sets and found that f(n)=nlog(n) works well across 2 models and 3 datasets. For other terms like the starting or ending N-grams, we follow the same design principle to ensure it is simple and effective.
Thus, we keep the current choices since they already work well across comprehensive experiments, serving as a strong baseline for future work.

Reviewer comment (Reproducibility):
Can authors explain a bit more on how P(Y_0 | X) is calculated when Y_0 is human-written text? Does text-davinci-003 allow such calculation?

Author response:
Yes, text-davinci-003 allows such calculations. When Y_0 is human-written text, we could get the logits over Y_0 given X by changing the settings in the OpenAI API: set the max_tokens = 0, n= 1, logprobs=5, echo=True. According to the official documentation [1], 

max_tokens = 0: The maximum number of tokens to generate in the completion.

n= 1: How many completions to generate for each prompt.

logprobs=5: Include the log probabilities on the logprobs of the 5 most likely tokens, as well the chosen tokens.

echo=True: Echo back the prompt in addition to the completion 

Thus, by the above settings, we were able to get all the token probabilities over Y_0 and X on text-davinci-003.

We also noticed that **on the OpenAI dev day, the former CEO of OpenAI announced [2] that they would give token logits access on GPT-4 API calls in the future. This will make our white-box detection method also applicable to GPT-4! Although we have not got access to GPT-4 output token logits, we believe this will only make our method more powerful in the future**

[1]  https://platform.openai.com/docs/api-reference

[2] https://openai.com/blog/new-models-and-developer-products-announced-at-devday