Paper: Efficient Detection of LLM-generated Texts with a Bayesian Surrogate Model (ICLR-2024)

Reviewer comment (Theory):
One point has not been discussed is the variance of the $ \log p_{\theta} (\tilde{\mathbf{X}})$  (also please use uppercase letter to denote the random variable for the mathematical rigorousness). The proposed method replaces $ \log p_{\theta} (\tilde{\mathbf{X}})$ with a a surrogate function $f$, such that numerous text examples can be used to get the empirical estimation of $E\left[f\left(\tilde{\mathbf{X}}\right)\right]$. On one hand, the variance of such an empirical estimation would be reduced as surrogate function $f$ can be inexpensively accessed; on the other hand, there is bias between $E\left[ \log p_{\theta} (\tilde{\mathbf{X}})\right]$ and $E\left[f\left(\tilde{\mathbf{X}}\right)\right]$. I was hoping to see the analysis of the impact of the variance and bias on the performance of the method, but failed to found them in both methodology and experimental sections.

Author response:
Thanks for the constructive comment. 

The probabilistic density function $p_\theta$ is **deterministic**. Thus, the mentioned “variance of the $\log p_\theta$” must be w.r.t. the perturbation distribution $q$. However, as detailed in (Mitchell et al., 2023), only $E_{q(x)} [\log p_\theta (x)]$ connects to the probabilistic curvature, the variance $Var_{q(x)}[\log p_\theta (x)]$ is meaningless.

Instead, as suggested by the reviewer, we should care about the variance of **the empirical estimates** of $E_{q(x)} [\log p_\theta (x)]$ and $E_{q(x)} [f(x)]$. For the former, the randomness stems from the data sampling process, and its effect is reflected by the error bars in Figure 2. For the latter, the randomness mainly stems from the samples used to assess $f$ at last, because the typical samples are identified in a nearly deterministic way (except for the initialization). As we use 200 samples to assess $f$, the variance caused by this part is minimal.

To evaluate the bias between $E_{q(x)} [\log p_\theta (x)]$ and $E_{q(x)} [f(x)]$, we can regard our current estimation of $E_{q(x)} [f(x)]$ using 200 samples as accurate and also use 200 samples to estimate $E_{q(x)} [\log p_\theta (x)]$. In our experience, the latter can usually lead to a detection AUROC of more than 98%. In comparison, the detection AUROC of the former with only 10 queries to the source LLM is less than 96%. Thus, the bias is significant. However, it is certain that we can reduce such a bias by incorporating more queries to the source LLMs for $E_{q(x)} [f(x)]$. This is echoed by the results in Section 4.1: once including enough queries, using $E_{q(x)} [f(x)]$ can catch up with the performance of using $E_{q(x)} [\log p_\theta (x)]$.

---

Thanks for your further clarification. We make the following comments:

1) **Variance of $\frac{1}{N}\sum_{i=1}^N f(X_i)$.** 

It is minimal as we set $N=200$ (i.e., use 200 rephrased texts around the original text to compute an average score). We have performed an empirical study regarding this. Using the setting corresponding to the results in Figure 2a, at the query time of 16 for estimating $f$, we compute the variance of $\frac{1}{N}\sum_{i=1}^N f(X_i)$ over 3 runs for 5 randomly selected texts from the dataset and obtain 0.0006, 0.0003, 0.0011, 0.0013, and 0.0011. In comparison, the mean of $\frac{1}{N}\sum_{i=1}^N f(X_i)$ is -2.2376, -2.5655, -2.4450, -2.2474, and -2.5945 respectively. As demonstrated, the mean is orders of magnitude larger than the variance. We will add more detailed results in the revision. 

2) **Bias of $\frac{1}{N}\sum_{i=1}^N f(X_i)$.** 

As $\frac{1}{N}\sum_{i=1}^N f(X_i)$ is an unbiased estimator of $\mathbb{E}_{q(x)} f(x)$, we speculate the mentioned bias corresponds to the gap between 

$\frac{1}{N} \sum_{i=1}^N f(X_i) $ and $ \frac{1}{N}\sum_{i=1}^N \log p_\theta (X_i) $. 

In our experience, using $200$ samples to estimate the latter usually leads to a detection AUROC of more than 98%. In comparison, the detection AUROC of the former with 10 queries to the source LLM $p_\theta$ for estimating $f$ is less than 96%. Thus, the bias is significant. However, it is certain that we can reduce such a bias by incorporating more queries to the source LLMs for estimating the function $f$. This is echoed by the results in Section 4.1: once including enough queries for estimating $f$, using $\frac{1}{N} \sum_{i=1}^N f(X_i) $ can catch up with the performance of using $\frac{1}{N}\sum_{i=1}^N \log p_\theta (X_i) $.

Reviewer comment (Experiments):
The evaluation is limited in that only 2 LLMs are utilized, one of which is GPT-2, which is more than 4 years old. The other model used is Llama-1 (65B), although the Llama-2 set of models has been released for quite some time now. Since there are many different open-source LLMs (for obtaining logits in the white-box setting) available with different parameter sizes, a comprehensive evaluation across many more LLMs — such as Llama-2, Guanaco, Vicuna, Falcon, MPT, ChatGLM, etc. — should be undertaken. Even for the experiments of Section 4.3, only smaller models such as GPT-J and GPT-Neo are used. Considering only 2 models (one of which is GPT-2) is not sufficient for evaluating the current performance of the proposed method.

Author response:
Thanks. We first clarify that our existing experiment settings align with DetectGPT (Mitchell et al., 2023). Our current empirical comparisons are fair, having proven our contribution to improving the query efficiency of log-curvature-based LLM-generated text detection. As acknowledged by Reviewer 5xFn, our method “show clear performance improvements over DetectGPT”, and by Reviewer 2wLg, our method ”maintain high prediction performance”. So, we clarify that the lack of the mentioned experiments is not a fundamental limitation of this paper.

Besides, LLaMA-65B is not a trivial LLM, as shown in ([LLaMA 2 paper](https://arxiv.org/pdf/2307.09288.pdf), Table 3), LLaMA-65B can consistently beat LLaMA 2 of 7B, 13B, and 34B across a series of benchmarks (e.g., Commonsense Reasoning, World Knowledge, Reading Comprehension, Math, MMLU, BBH, AGI Eval). As the community of LLM is developing very rapidly, we don’t have enough energy to catch up with all the recent models and *undertake a comprehensive evaluation across many more LLMs: for example, Llama-2, Guanaco, Vicuna, Falcon, MPT, ChatGLM, etc*. 

Comparing the results on LLaMA-65B to those on GPT-2, we can see that detecting larger or more advanced models is more challenging for both DetectGPT and our method. This implies that more queries are needed to complete the detection on  LLM with a high parameter size. To further verify these claims, we have started a set of experiments with the series of LLaMA models on the XSum dataset and will offer the corresponding results in subsequent updates (if the running finishes before the end of the rebuttal period) or the final version. 

As for black-box detection, our cross-evaluation is sufficient to demonstrate that our method outperforms DetectGPT in black-box scenarios, using the same setting as the DetectGPT paper. We will experiment with the mentioned API-based black-box models in the next version.

Reviewer comment (Experiments):
There is no analysis of trends with respect to parameter size. An obvious question is whether, as the models considered have an increasing number of parameters, the proposed method becomes less efficacious, regardless of query size. Compared to DetectGPT, how many queries would be required to successfully detect text generated by a more advanced model such as Llama2-70B? Experiments using LLMs that have different parameter size versions (e.g., Llama 7B, 13B, 70B) could provide performance curves for different LLM size classes.

Author response:
Thank you for your patience regarding the results of the LLaMA2 series. Below, you will find the results for XSum:
   
  |   ||DetectGPT| | |Our method | |
|---|---|---|---|---|---|---|
|Query time| 5 | 10 |15 | 5| 10 |15|
|LLaMA2-7B|0.682|0.719|0.728 |0.704|0.732|0.747|
|LLaMA2-13B|0.649|0.707 |0.712| 0.664|0.722|0.729|
|LLaMA2-34B|0.633|0.702|0.708| 0.662|0.715|0.719|

Due to time constraints, we only experimented with up to 15 queries to the source LLM. As shown, both DetectGPT and our method exhibit lower detection AUROC scores as the model size increases. However, our method, using 10 queries, outperforms the DetectGPT baseline using 15 queries. This demonstrates the query efficiency of our method in the LLaMA2 case. It is also worth noting that the LLaMA2 models are indeed more challenging to detect compared to GPT-2.

In the next version, we will investigate if allocating a higher query budget and employing a stronger perturbation model can yield improved AUROC scores. We will also strengthen our cross evaluation of the black-box models. We thank the reviewer again for the constructive feedback!

Reviewer comment (Experiments):
There is a lack of analysis on black-box models. The mismatch setting of Section 4.3 should be augmented with a black-box setting where state-of-the-art black-box LLMs such as GPT-3.5, GPT-4, PaLM-2, Claude, etc., are analyzed. If the goal of the paper is to truly ensure that LLM-generated text is detected, the evaluation should ideally include these models via proxy models. As these LLMs are the easiest to use via a user interface, LLM-generated text is most likely to stem from these sources. Incorporating evaluation along these lines would be beneficial.

Author response:
Thanks. We first clarify that our existing experiment settings align with DetectGPT (Mitchell et al., 2023). Our current empirical comparisons are fair, having proven our contribution to improving the query efficiency of log-curvature-based LLM-generated text detection. As acknowledged by Reviewer 5xFn, our method “show clear performance improvements over DetectGPT”, and by Reviewer 2wLg, our method ”maintain high prediction performance”. So, we clarify that the lack of the mentioned experiments is not a fundamental limitation of this paper.

Besides, LLaMA-65B is not a trivial LLM, as shown in ([LLaMA 2 paper](https://arxiv.org/pdf/2307.09288.pdf), Table 3), LLaMA-65B can consistently beat LLaMA 2 of 7B, 13B, and 34B across a series of benchmarks (e.g., Commonsense Reasoning, World Knowledge, Reading Comprehension, Math, MMLU, BBH, AGI Eval). As the community of LLM is developing very rapidly, we don’t have enough energy to catch up with all the recent models and *undertake a comprehensive evaluation across many more LLMs: for example, Llama-2, Guanaco, Vicuna, Falcon, MPT, ChatGLM, etc*. 

Comparing the results on LLaMA-65B to those on GPT-2, we can see that detecting larger or more advanced models is more challenging for both DetectGPT and our method. This implies that more queries are needed to complete the detection on  LLM with a high parameter size. To further verify these claims, we have started a set of experiments with the series of LLaMA models on the XSum dataset and will offer the corresponding results in subsequent updates (if the running finishes before the end of the rebuttal period) or the final version. 

As for black-box detection, our cross-evaluation is sufficient to demonstrate that our method outperforms DetectGPT in black-box scenarios, using the same setting as the DetectGPT paper. We will experiment with the mentioned API-based black-box models in the next version.

Reviewer comment (Experiments):
There is an overall lack of experimental rigor. Some claims made in the paper are hand-wavy and lack sufficient rigor. For example, on page 7 (Section 4.1 end), the authors state that even under high query budgets, their approach remains effective and comparable to DetectGPT; however, only the WritingPrompts data is used with GPT-2. To support such a claim, more experiments should be conducted over all datasets and all LLMs, presented as a table or figure, from which adequate conclusions can be drawn.

Author response:
We would like to clarify that the reason for the method not performing well against newer LLMs is not due to its inherent limitations. Rather, it is because the number of queries used in our experiments was limited due to time constraints. Besides, the paper focuses on demonstrating the superiority of our method over DetectGPT in the low-budget regime. However, it is worth noting that we can improve the AUROC by using a larger number of queries, as demonstrated in the DetectGPT paper. In the next version, we will include experiments with a higher query count to address this aspect.

Reviewer comment (Writing):
While a minor issue, the paper contains many typos and grammatical errors, which should be corrected in revision. For example: page 7: "bedget" → "budget", and page 4: "typicity" means something unrelated to statistics and ML.

Author response:
Thanks for pointing out this, and we have revised the paper accordingly.

Reviewer comment (Experiments):
Why have the authors not considered more LLMs, especially of different parameter size classes, in experiments?

Author response:
Thanks. We first clarify that our existing experiment settings align with DetectGPT (Mitchell et al., 2023). Our current empirical comparisons are fair, having proven our contribution to improving the query efficiency of log-curvature-based LLM-generated text detection. As acknowledged by Reviewer 5xFn, our method “show clear performance improvements over DetectGPT”, and by Reviewer 2wLg, our method ”maintain high prediction performance”. So, we clarify that the lack of the mentioned experiments is not a fundamental limitation of this paper.

Besides, LLaMA-65B is not a trivial LLM, as shown in ([LLaMA 2 paper](https://arxiv.org/pdf/2307.09288.pdf), Table 3), LLaMA-65B can consistently beat LLaMA 2 of 7B, 13B, and 34B across a series of benchmarks (e.g., Commonsense Reasoning, World Knowledge, Reading Comprehension, Math, MMLU, BBH, AGI Eval). As the community of LLM is developing very rapidly, we don’t have enough energy to catch up with all the recent models and *undertake a comprehensive evaluation across many more LLMs: for example, Llama-2, Guanaco, Vicuna, Falcon, MPT, ChatGLM, etc*. 

Comparing the results on LLaMA-65B to those on GPT-2, we can see that detecting larger or more advanced models is more challenging for both DetectGPT and our method. This implies that more queries are needed to complete the detection on  LLM with a high parameter size. To further verify these claims, we have started a set of experiments with the series of LLaMA models on the XSum dataset and will offer the corresponding results in subsequent updates (if the running finishes before the end of the rebuttal period) or the final version. 

As for black-box detection, our cross-evaluation is sufficient to demonstrate that our method outperforms DetectGPT in black-box scenarios, using the same setting as the DetectGPT paper. We will experiment with the mentioned API-based black-box models in the next version.

Reviewer comment (Evaluation):
Is there any limiting factor for evaluation of black-box LLMs available only via APIs?

Author response:
As for black-box detection, our cross-evaluation is sufficient to demonstrate that our method outperforms DetectGPT in black-box scenarios, using the same setting as the DetectGPT paper. We will experiment with the mentioned API-based black-box models in the next version.