Paper: Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text (ICLR-2024)

Reviewer comment (Evaluation):
In section 4.2 it is argued that TPR@FPR is a better metric than F1 score or AOU, and particularly TPR at 0.01% of FPR. However TPR at 0.01 FPR is never used in the results presented afterwards and mainly F1, AUC, precision and recall are used in most of the figures. Only in figures 4 and 6 TPR vs FPR is plot, but without specifically analyzing TPR at 0.01% FPR as claimed in section 4.2. Related to this, in the abstract it is claimed that "On news documents Binoculars detect 95% of synthetic samples at a false positive rate of 0.01%". In the results only figure 4 for the student essay dataset seems to show a result similar to this one, but it is very specific to only one of the datasets used in the experiments.

Author response:
__1. Usage of TPR @ low-FPR across experiments__
- In the metrics section, we discuss how detectors’ performance should be evaluated under a low-false-positive regime (which can be realized by many metrics) since type 1 errors are significantly more costly than type 2 in downstream tasks. We also argue that typically reported metrics like standalone AUC values, as in other baselines, don’t translate into real-life performance sufficiency (as shown in Table 2: AUCs are uncorrelated with TPR at low FPR). Below is our rationale for metrics in figures:
  - Figure 1 reports TPR @ 0.01% FPR and Figure 2 reports F1-Score on the identical dataset as Figure 1 (ChatGPT benchmark by [1])
  - Figure 3 is updated to have TPR @ 0.01% FPR at the varying sizes of samples (# of tokens)
  - Figure 4 reports TPR observed at different FPR levels (on log scale) for LLaMA-2-13B generations over 3 datasets
  - Figure 5, 6, and 7 reports how Binoculars maintain high precision (with varying recall) over multiple generators, languages and domains respectively.
  - Figure 9's dataset contains only machine-generated samples and hence only reports an error rate.

__2. The abstract claim of detecting news documents Binoculars detect 95% of synthetic samples at a false positive rate of 0.01%__
- This figure is from the first column in Figure 1 (News, ChatGPT dataset [1]). We updated the abstract to replace the previously rounded number to 94.92%.

Reviewer comment (Evaluation):
Evaluation should be done in a more systematic and coherent way among different datasets to be able to better compare the performance of the proposed method with SoA. As long as it is possible the same methods should be used for comparison. However, the set of methods used to compare with in figures 2, 4, 5 and 6 are different, and even methods used in figure 6 are not mentioned or discussed in the text.

Author response:
__4. Performance comparison with state-of-the-art and using the same methods should be used for comparison__
- Our emphasis is directed toward baselines that function within post hoc, out-of-domain (zero-order), and black-box detection scenarios. We use state-of-the-art Ghostbuster (Verma et al., 2023), the commercially deployed GPTZero, and DetectGPT to compare detection performance over various datasets in Section 4.
- Performance Benchmark: As per above, on all benchmark datasets in section 4, we use these baselines (which include the ChatGPT benchmark dataset competing baseline in [1]). To align our experiments better, we have updated the figure as mentioned above in this note. We thank you for your feedback on this.
- Reliability: In Section 5, we evaluate the reliability of Binoculars in various settings that constitute edge cases and interesting behaviors of our detector. With the exception of Figure 9 (modified prompting strategies), we report these numbers as absolute to aide understanding of Binoculars specifically. 

__5. Methods in Figure 6 are not mentioned or discussed in the text__
- Thank you for this feedback. We have added a paragraph in section 5.1 for these methods.

Reviewer comment (Experiments):
The selection of the methods used to compare with is not clearly motivated. In section 2 many methods are described and categorized in different categories. It would be nice to have a systematic comparison with more methods of each of the categories defined in section 2.

Author response:
__6. The selection of the methods used to compare with is not motivated.__
- We updated the draft to motivate our choice of baselines (also mentioned in the previous point) (section 2). To reiterate, we focus on evaluating baselines designed for post hoc, out-of-domain (zero-order), and black-box detection scenarios. which includes state-of-the-art Ghostbusters (Verma et al., 2023), the commercially deployed GPTZero, and DetectGPT.

Reviewer comment (Experiments):
I miss an ablation study analyzing some of the choices made in the design of the method. For instance, the contribution of using the proposed detection score vs. simple perplexity or the effect of the LLMs used both to compute perplexity and to generate the text to be classified. Is there any difference in performance (positive or negative) if the observer LLM is the same LLM used to generate the text under analysis?

Author response:
__7. Ablation study to ascertain the contribution of using the proposed detection score vs. simple perplexity__
- Due to the page limit we have this study in Figure 12 + Table 3 in Appendix A. We compare PPL, xPPL, and Binoculars (Falcon and LLaMA variants) performance over the ChatGPT dataset from 1]. We see how Binoculars perform better than its components to detect machine text.

## Rebuttal 2 (ID: HFqf7FXRhw)
__8. The difference in performance (positive or negative) if the observer LLM is the same LLM used to generate the text under analysis__
- Figure 13 includes performance when using Falcon-7B (which powers the Binoculars score) for sample generation. We do not see a remarkable change in performance or a visible trend when using Falcon-7 B-produced text.

Reviewer comment (Experiments):
There is no comparison of the results obtained in the Orca dataset with results obtained by other methods.

Author response:
__12. There is no comparison of the results obtained in the Orca dataset with results obtained by other methods__
- We run other methods and append baselines in Figure 9 to show our method is more robust than other methods against different system prompting strategies.

Reviewer comment (Writing):
A minor comment with respect equation (3): shouldn't it be log(M_2(T(s))_i) instead of log(M_2(T(s))_j)?

Author response:
__13. Typo in formulation/notation.__
- We thank you for the detailed feedback and have updated to fix all pointers.

Reviewer comment (Novelty):
In section 2, it could be clearer how the proposed method relates to other PPL-based ones.

Author response:
__1. In section 2, it could be clearer how the proposed method relates to other PPL-based ones__
- One difference between Binoculars and some other perplexity-based detectors is in the number of forward passes required to compute a discriminative score. For example, DetectGPT is built on a similar motivation -- that perplexity alone is insufficient for classification -- but to overcome the weakness in the signal, DetectGPT utilizes the perplexity of several perturbations to the input. In addition to the overhead of generating perturbations, it also requires computing many perplexities and thus it can be far more computationally expensive. So there may be various ways to extract meaningful information from perplexity, ours is less computationally demanding than some others.

Reviewer comment (Evaluation):
In Table 2, we see that the performance can vary a lot depending on the chosen LLM for scoring: what makes a good LLM for the method? Why other choices reach lower TPR?

Author response:
__2. In Table 2, we see that the performance can vary a lot depending on the chosen LLM for scoring: what makes a good LLM for the method? Why other choices reach lower TPR?__
- This is more because of how stringent TPR @ 0.01% FPR as a performance metric is than instability of performance. We updated the table 2 to include TPR @ 0.1% FPR. F1-Score and AUC. As you may see, the performances are "stable" through the lens of F1-Score and AUC - which are fine metrics to use when the costs of type 1 and 2 errors are uniform. 
- However, in problems with the severely skewed cost of error towards FP (like detection), metrics like (vanilla) F1-score and AUC don’t provide downstream safety in the deployment of these systems. 
- We also note (in section 4.2) “that AUC scores are often uncorrelated with TRP@FPR when the FPR is below 1%.” (as seen in the updated Table 2).
- Moreover, we see saturation of performance using Binoculars (at least in its current format) irrespective of the scorers used. However, with the surge in open-source LMs, we do not claim an exhaustive search of scoring models to be used.

Reviewer comment (Experiments):
The point about similarity of LLMs is interesting. What would happen if the LM (evaluated or chosen for scoring) is less similar, e.g. different training set, different kind of model, etc.

Author response:
__3. What would happen if the LM (evaluated or chosen for scoring) is less similar, e.g. different training set, different kind of model, etc.__
- Please note, by our x-PPL definition, we have an implicit constraint of having to use two models to share a tokenizer, and most available open-source models have a high degree of training set overlap (eg. Falcon, Llama families, etc.). 
- The cross-perplexity measures the degree to which two models' next-token distributions overlap. We suspect cross-ppl from two models trained on dissimilar datasets would provide more noise than the signal we want to detect (see updated text section 3.3 for motivation of x-PPL). 
- The similar behavior of the transformers model is what we depend on for cross-ppl and Binoculars score definition. Thank you for mentioning this important point. We expanded the definition section to mention the impact of different training sets and will be happy to provide any more information required. 
- In the ablation experiment (in Figure 17), we see how x-PPL and PPl alone aren't enough for reliable detection performance.

Reviewer comment (Writing):
The point made at the end of section 5.1 is interesting but it is not so clear what should be concluded from that remark. It also begs the question of how would the training set of the LLMs (mostly written by humans I guess) be detected.

Author response:
__4. How would the training set of the LLMs (mostly written by humans I guess) be detected__
- Training samples are a super-set of memorized strings in Section 5.3 (i.e. strings that were part of training but may or may not be re-generated by LLM). 
- For a perplexity-based detector, memorized/famous text is more difficult to classify in comparison to samples from larger training sets (since the famous text is expected to be in the training set multiple times). 
- Also, the classification of such samples is domain-specific since both humans and machines can be deemed to have produced it in different domains (eg. plagiarism detection v/s  removal of LLM-generated text from a training corpus). 
- With this understanding and performance on memorized text, we believe that our detector would score samples from the training set towards the human side of the threshold and should be leveraged accordingly in downstream tasks.

Reviewer comment (Experiments):
The point made in section 5.4 (randomization) is interesting as well. It would be interesting to see if the detector could be fooled by randomly changing "some" words in the generated text. The remark made in section 5.1 about the desirability of outcome is relevant here too (= how to consider human-edited machine-generated text).

Author response:
__5. Can the detector be fooled by randomly changing "some" words in the generated text?__
- Our focus is on detecting machine-generated text in typical usage scenarios. We specifically exclude consideration of explicit efforts to bypass detection, such as adversarial perturbed text detection using Binoculars (or any variant). 
- However somewhat related, in section 5.3, we observe that Binoculars are insensitive to grammar-corrective changes in human-generated text (same accuracy for either text). Specifically, in Figure 8 we see a comparison of the two distributions of Binoculars.
- Adversarial evasion/attack on Binoculars is left for future work.

Reviewer comment (Experiments):
The point of section 5.2 is interesting and we could also wonder how robust this detection method is. For example, is it easy to tweak existing LLMs to fool the detector (other than prompt), or during training, how easy would it be to integrate the detection score to fool the detector.

Author response:
__6. Is it easy to tweak existing LLMs to fool the detector (other than prompt), or during training, how easy would it be to integrate the detection score to fool the detector?__
- In this work, we concentrate on detecting unadulterated text from language models and leave the adversarial attack on Binoculars to future work. 
- As such [1] discusses the theoretical limits of detection and highlights that with enough paraphrasing, detection can be impossible.
- For example, LMs can be hard-prompted, soft-prompted, or even fine-tuned to produce text that evades detection or particular detector.
- There can be several such attack models used to bypass detection which we leave for future work.

Reviewer comment (Experiments):
The experiments and comparisons in Figure 5 are crucial, but the authors only compared with Ghostbuster and the analyses are too brief.

Author response:
__1. Only Ghostbusters used as a baseline in open-source language models text detection (LLaMA-2-13B)__
- We add baseline experiments in Figure 4. The performances on Falcon-7B generations can be found in Figure 13 in Appendix A3.

Reviewer comment (Experiments):
According to the ablation study in A.1 in the APPENDIX, the authors actually only conducted experiments using the models from the Falcon and Llama-2 families. Why other types of open-source or closed-source large language models (such as ChatGPT, GPT-4 and Baichuan 2) are not adopted?

Author response:
__2. Only models from Falcon and Llama-2 families used for computing x-PPL__
- Thanks for noting this. This is because of the following:
- Implicit constraint of x-PPL computation which requires two models to have identical tokenizer (we mention in section 3.1)
- As per our intuition (and motivation, section 3.3), our detector's performance is dependent on having two _similar_ (by training set) models for computing x-PPL and would only cause a marginal change in performance across different scoring model pairs. We show this in the updated Table 2 in Appendix A1, how F1-Score and AUC metrics (which weigh type 1 and type 2 error equally) are stable across multiple pairs of scoring models only resulting in a marginal change in the performance of the detector. Please note, that we do not claim our scoring models search to be exhaustive.

Reviewer comment (Experiments):
The authors claim their method is a zero-shot detector, but they "optimize and fix the detection threshold globally using these datasets". The fact that the used pre-trained LLMs are not retrained/fine-tuned does not imply the proposed method is zero-shot, as they optimize the detection threshold for the target task with (in most of the cases) some in-domain data.

Author response:
__1. Zero-shot Detection Claim.__
- We would like to clarify that we do not optimize the threshold for the target task (in all cases/experiments), but use a global threshold obtained from Binoculars. As mentioned in section 4.3: since our global threshold is tuned on a ChatGPT benchmark dataset (News, Creative Writing, Essay) by [1] (Ghostbusters method), we make an exception "to be sure that we meet the OOD criteria, we do not include the ChatGPT datasets in the threshold determination, and only use samples from CC News, CNN, and PubMed (generated via LLaMA and Falcon) to determine threshold to do predictions/evaluation on ChatGPT benchmark dataset (Figure 1, 2, & 3). For open-source datasets (CC News, CNN, and PubMed), we report AUC plots for all methods that are threshold-invariant for all methods (Figure 4). With this OOD threshold, we demonstrate that reliably detects texts generated by different LMs and from different domains.

Reviewer comment (Experiments):
Several of the baseline models are not explained in the text: Zero-shot (in Figure 1) and Roberta, LR GLTR, Stylistic, Nela (in Figure 6).

Author response:
__2. Roberta, LR GLTR, Stylistic, Nela not explained.__
- Thank you for pointing out, that we have added a note in the M4 Datasets note in section 5.1

Reviewer comment (Experiments):
The comparison with baselines is not consistent across experiments. In some Figures/Tables the proposed method is compared with a set of baselines while in others the baselines are different. The choice of baselines seems quite trivial and makes it difficult to assess the contribution of the proposed method.

Author response:
__3. Choice of baselines and across experiments__
- In this work, our emphasis is directed toward baselines that function within post hoc, out-of-domain (zero-order), and black-box detection scenarios. We use state-of-the-art Ghostbusters [1], GPTZero, and DetectGPT to compare detection performance over various datasets in section 4.
- In section 5, we evaluate the reliability of Binoculars in various settings that constitute edge cases and
interesting behaviors of our detector (multi-domain, languages, non-native English text, etc.)
- We updated the draft to mention our choice of baselines and rationale towards the end of Section 2.

Reviewer comment (Experiments):
For the Orca Dataset, no baseline results are provided. The same applies to all experiments in section 5. Not having any baseline results on these experiments makes it very difficult to quantify the quality of the proposed solution.

Author response:
__6. Baselines in Orca Dataset__
- We run other methods to append baselines in Figure 9 to show our method is more robust than other methods.

Reviewer comment (Evaluation):
When showing FPR/TPR plots, it is unclear how these plots are generated. Are they obtained by changing the detection threshold? It is also unclear why the authors claim that a 0.01% FPR threshold is “the smallest threshold we can comprehensively evaluate with our limited compute resources.”

Author response:
__10. When showing FPR/TPR plots, how are these plots generated? by changing the detection threshold? I do not understand why you say 0.01% FPR threshold is "The smallest threshold we can comprehensively evaluate with our limited compute resources."__
- Yes, for each method we move the threshold to achieve 0.01% FPR and report the TPR figures to compare performance in low false positive regime. 
- The FPR rate we can achieve on a log scale is a function of the number of samples. For all our experiments, we use a balanced dataset (50-50 human and LM samples). At 0.01% FPR, we would need 10k samples per class, and at 0.001% we would need 100k samples per class.

Reviewer comment (Writing):
The formulation/notation of the Binoculars score is confusing because $L$ (in eq. 2 and 3) is not defined anywhere in the text. It is assumed to be the sentence length.

Author response:
__7. Inconsistency/typos in formulation/notation.__
- We thank you for the detailed feedback and have updated to fix all pointers.

Reviewer comment (Writing):
In eq. 3 it would be better to use $\overrightarrow{x}$ (as defined before) instead of $T(s)$, as it would make the equation more readable.

Author response:
__7. Inconsistency/typos in formulation/notation.__
- We thank you for the detailed feedback and have updated to fix all pointers.

Reviewer comment (Theory):
If not mistaken, the measure expressed in eq. 3 is the $log XPPL$ and not $XPPL$. This interpretation seems consistent with the $log PPL$ definition in eq. 2.

Author response:
__7. Inconsistency/typos in formulation/notation.__
- We thank you for the detailed feedback and have updated to fix all pointers.

Reviewer comment (Evaluation):
Since cross-perplexity ($XPPL$) is not a standard term in NLP or machine learning, there should be more discussion on its interpretation and its effects on the proposed Binoculars score. This includes: what are the upper/lower bounds of B? What happens when M_1 and M_2 are the same pre-trained model? What if they are trained in two totally different domains? While section 3.2 (Capybara problem) discusses the case of “hand-crafted prompts”, other aspects of the Binoculars score should be discussed as well.

Author response:
__9. Use of scoring models trained on different domains and other aspects of Binoculars.__
- Please note, by our x-PPL definition, we have an implicit constraint of having to use two models to share a tokenizer, and most available open-source models have a high degree of training set overlap (eg. Falcon, Llama family). 
- The cross-perplexity measures the degree to which two models' next-token distributions overlap. We suspect cross-ppl from two models trained on dissimilar datasets would provide more noise than the signal we want to detect (see updated text section 3.3 for motivation of x-PPL). The similar behavior of the transformers model is what we depend on for cross-ppl and Binoculars score definition. Thank you for mentioning this important point. We expanded the definition section to mention the impact of different training sets and will be happy to provide any more information required. 
- In the ablation experiment (in Figure 17), we see how x-PPL and PPl alone aren't enough for reliable detection performance.
- We are happy to answer any further or follow-up questions on this.