Paper: Few-Shot Detection of Machine-Generated Text using Style Representations (ICLR-2024)

Reviewer comment (Experiments):
*No experiments measuring robustness to paraphrasing attacks*: It is now well-established that AI-generated text detectors are very brittle against paraphrasing attacks [4, 5], including DetectGPT, watermarks and GPTZero. How robust is the proposed few-shot style classifier against paraphrasing attacks like DIPPER [4, 5]? Given the vulnerability to paraphrasing, it is critical to evaluate AI-generated detectors on this dimension. Moreover, prior work [6] has shown that paraphrasing strips out stylistic elements from text. This might make the proposed approach even more vulnerable to paraphrasing attacks.

Author response:
Thank you for the interesting question. The proposed framework can account for paraphrasing attacks in two ways:

1. The few-shot example may itself consist of a handful of examples of paraphrased text, in which case we can identify further instances of paraphrased LLM text as usual.
2. If the test setting features both LLM text and paraphrased LLM text, our multi-LLM detection approach may be applied in which, for example, a few-shot sample of GPT-4 and paraphrased GPT-4 are used to build the LLM detector.

We expect the second approach to provide additional robustness since an attacker may vary the degree of paraphrasing. To validate this hypothesis, we conduct an additional experiment in which DIPPER (Krishna et al., 2023), https://arxiv.org/pdf/2303.13408.pdf is used to paraphrase LLM text.

We test two versions of UAR, one where the support examples come strictly from the original LLM, and another that mimics the Multi-LLM scenario in that we provide UAR with one support example of the original LLM and another of the paraphrased LLM. For both experiments, we vary the proportion of queries that are paraphrased from 0% to 100% and set N=5.

|                 |      0.0 |     0.25 |      0.5 |     0.75 |      1.0 |
|:----------------|---------:|---------:|---------:|---------:|---------:|
| UAR (Reddit 5M) | **0.90446**  | **0.910341** | 0.86506  | 0.797051 | 0.706332 |
| UAR Multi-LLM   | **0.90446**  | 0.907958 | **0.890054** | **0.881983** | **0.886131** |
| PROTONET        | 0.871244 | 0.865504 | 0.81867  | 0.752616 | 0.666963 |

We observe that both UAR (Reddit 5M) and PROTONET suffer as the proportion of paraphrased queries increases. However, including the paraphrased LLM as a support example (UAR Multi-LLM) ameliorates the drop in performance.

Reviewer comment (Experiments):
After reading the paper, I was a bit confused by the first sentence in Section 3.1, "The experiments in this paper deal with both the supervised and few-shot learning regimes". Where exactly were the experiments in the fully supervised regime? Please correct me if I'm wrong, but to the best of my understanding, the model was trained on data from "Amply Available and Cheap" LLMs (like GPT2), and then tested on harder LLMs (like ChatGPT) using few-shot style matching on this trained model.

Author response:
We agree that this should be clarified in the main text and have made appropriate revisions. To confirm, the “zero-shot” baseline, namely “RoBERTa (zero-shot)”, employs a supervised training regime. We argue in the paper that such supervised zero-shot approaches are fundamentally limited due to distribution shifts in real-world data (e.g., introduction of new LLM or new domain relative to training data), which is why we emphasize few-shot approaches and baselines. We show the brittleness of the supervised training regime in Appendix A, where we evaluate what happens to supervised detectors as the testing distribution shifts.

Reviewer comment (Novelty):
In the end, the paper boils down to applying a few methods that learn stylistic and semantic representations to classify a document as human- or LLM-written. There is no new method, and the contribution lies in applying existing methods and creating a training and evaluation corpus for AAC and LWD LLMs (what the authors do not mention as a contribution, and these datasets will not be made available according to the paper).

Author response:
> the contribution lies in applying existing methods and creating a training and evaluation corpus for AAC and LWD LLMs (what the authors do not mention as a contribution, and these datasets will not be made available according to the paper).

We do intend to release all the datasets mentioned in the paper (the reproducibility statement has been updated to clarify this point). Regarding the novelty of the approach, to the best of our knowledge, few-shot detection of machine-generated text using style representations is a new method:

* We are the first to systematically explore few-shot machine detection. We argue that this is necessary in light of real-world distribution shifts to obtain a robust machine-text detector. This is a non-trivial departure from prior work in machine-text detection and offers a practical way to tackle distribution shifts pervasive in most settings where one may be interested in detecting machine-generated text.
* We are the first to find that representations of writing style estimated solely from human-written text (i.e., negative examples) can outperform state of the art few-shot machine-text detection methods, even though those baseline methods are trained using additional positive examples of machine-generated texts (Table 2). We argue that this is a significant finding in light of the proliferation of LLMs, since it is impractical to repeatedly train new supervised classifiers on the basis of the latest LLMs. Our approach opens the door to building “training-free” detectors for the latest LLMs that are tailored to a variety of downstream settings (e.g., education, science, social media).

We also explore several simple but important adaptations of existing style representations for our machine-text detection task, rather than simply applying them “off-the-shelf”, including training on significantly more data, training on combinations of machine-generated and human-generated data, and applying them to several detection settings (e.g., multi-LLM).

Reviewer comment (Experiments):
The experiments miss some details, and it is unclear why the proposed UAR is trained on different datasets than the other methods.

Author response:
> The experiments miss some details, and it is unclear why the proposed UAR is trained on different datasets than the other methods.

The proposed approach is trained on different data than the non-stylistic baselines because it does not require examples of machine generated text. This is a key finding of our work: representations trained only on human-authored text (e.g., Reddit, Twitter, StackExchange) yield stylistic representations that can effectively discriminate between human and machine authors.

Reviewer comment (Reproducibility):
How is the similarity score between support and query samples calculated?

Author response:
> How is the similarity score between support and query samples calculated?

We compute the cosine similarity between the support embedding and the query embedding as outlined in the first paragraph of Section 3.2.

Reviewer comment (Experiments):
Section 4.1: Where is the Amazon dataset coming from? How is it used? Also, in the multi-domain variation?

Author response:
> Section 4.1: Where is the Amazon dataset coming from? How is it used? Also, in the multi-domain variation? 

As stated in the reproducibility statement, the Amazon dataset comes from: https://nijianmo.github.io/amazon/index.html#files. It is strictly used to generate LWD data that is used for evaluation. We do not use it as training data.

Reviewer comment (Evaluation):
Section 4.3: The metric being reported is AUC'. This is rather uncommon. Why not report the normal AUC? Maybe all methods look too similar, then? Also, instead of introducing a new AUC' the authors could have resorted to something more in line with other works, e.g., something similar to FPR95 but reflecting the importance of a low false positive rate. Using the reasoning from section 4.3, a TPR95 (also not in the literature like AUC', but would be more in line with FPR95).

Author response:
> Section 4.3: The metric being reported is AUC'. This is rather uncommon. Why not report the normal AUC? Maybe all methods look too similar, then? Also, instead of introducing a new AUC' the authors could have resorted to something more in line with other works, E.g., something similar to FPR95 but reflecting the importance of a low false positive rate. Using the reasoning from section 4.3, a TPR95 (also not in the literature like AUC', but would be more in line with FPR95.).

We choose to emphasize AUC’ since we focus on an operating point with a low false alarm rate, which we argue is necessary for real-world applications of machine-text detectors. Our findings are consistent with other metrics at other operating points. Below (and also in the updated draft in Appendix H, Table 15), we show results when we truncated every evaluation document to the nearest sentence boundary before the thirty-second token:

 | **Method** | **Training Dataset** | **AUC** | **AUC'** | **FPR@95TPR** | 
 |     :----:    |     :----:    |     :----:    |     :----:    |     :----:    | 
 | **Few-Shot Methods** |  |  |  | 
 | LUAR  |  Reddit(5M)  |  **0.9884** | 0.9094 |  **0.0533** | 
 | LUAR  |  Reddit (5M), Twitter, StackExchange  |  **0.9884** |  **0.9118**  | 0.0543 | 
 | LUAR  |  AAC, Reddit (politics)  | 0.8913 | 0.7308 | 0.3394 | 
 | CISR  |  Reddit (hard neg/hard pos)  | 0.9608 | 0.7973 | 0.144 | 
 | RoBERTa (ProtoNet)  |  AAC, Reddit (politics)  | 0.9791 | 0.9062 | 0.0934 | 
 | RoBERTa (MAML)  |  AAC, Reddit (politics)  | 0.6686 | 0.5141 | 0.6555 | 
 | SBERT  |  SNLI, MultiNLI  | 0.9673 | 0.8087 | 0.1448 | 
 | **Zero-Shot Methods** |  |  |  |  | 
 | RoBERTa (Zero shot)  |  AAC, Reddit (politics)  | 0.7238 | 0.5295 | 0.5369 | 
 | OpenAI Detector  |  WebText, GPT2-XL  | 0.6933 | 0.5559 | 0.7698 | 
 | Rank (GPT2-XL)  |  BookCorpus, WebText  | 0.7301 | 0.5423 | 0.6341 | 
 | LogRank (GPT2-XL)  |  BookCorpus, WebText  | 0.9107 | 0.6395 | 0.2209 | 
 | Entropy (GPT2-XL)  |  BookCorpus, WebText  | 0.2083 | 0.4977 | 0.9667 | 
 | Random |  | 0.50 | 0.05 | -- | 

When evaluating using smaller writing samples and looking at standard metrics, the proposed style-based approaches continue to outperform both few-shot baselines and standard zero-shot baselines.

Reviewer comment (Experiments):
Section 4.3: Missing details in the experiment section: Size of training and evaluation corpora, which LLMs are analyzed. It would help if the authors also mentioned that AUC' is normalized.

Author response:
> Missing details in the experiment section: Size of training and evaluation corpora, which LLMs are analyzed.

The details for each of the datasets are mentioned in the Appendix due to the page limit; we agree that having this in the main text would be helpful. Appendix C shows the statistics of the datasets used to train UAR, while Appendix E covers the details of creating the AAC and LWD datasets as well as their respective statistics and the details of the M4 dataset. We evaluate all our approaches on unseen domains and unseen LLM’s including Llama-2, ChatGPT, GPT-4, Dollyv1, Dollyv2, Davinci, FlanT5, Cohere, and BloomZ.

> It would help if the authors also mentioned that AUC' is normalized.

We have clarified this point in paper.

Reviewer comment (Experiments):
Section 4.4: The paper mentions that Reddit is the main training corpus and, therefore, not used for evaluation. This contradicts section 4.1, where Reddit is also in the evaluation corpus. So, how is Reddit used?

Author response:
> Section 4.4: The paper mentions that Reddit is the main training corpus and, therefore, not used for evaluation. This contradicts section 4.1, where Reddit is also in the evaluation corpus. So, how is Reddit used? Table 2 shows different methods trained on different training corpora. Since the paper emphasizes that UAR is THE method to perform few-shot LLM text detection, why are the baselines not trained on the same training corpora?

We made an error in Section 4.1 when we mentioned that Reddit was used as an evaluation corpora. The evaluation datasets are Amazon, M4 PeerRead, M4 ArXiv, M4 WikiHow, and M4 Wikipedia. The Reddit corpora is used strictly for training, which enables us to evaluate robustness to distribution shift.

For baselines which require AAC data (MAML, Protonet, and RoBERTa), we train on a politics-only version of the Reddit dataset. We restrict the topic to politics to ensure that these baselines learn features that are agnostic to the domain. In Appendix B, we perform ablations with ProtoNet where we show that controlling the domain is essential to achieving good results. 

As mentioned earlier, the proposed style-based representation can be trained using only negative (human-written) examples, and as a result we can train on more datasets such as StackExchange.

Reviewer comment (Experiments):
Table 2 shows different methods trained on different training corpora. Since the paper emphasizes that UAR is THE method to perform few-shot LLM text detection, why are the baselines not trained on the same training corpora?

Author response:
> Section 4.4: The paper mentions that Reddit is the main training corpus and, therefore, not used for evaluation. This contradicts section 4.1, where Reddit is also in the evaluation corpus. So, how is Reddit used? Table 2 shows different methods trained on different training corpora. Since the paper emphasizes that UAR is THE method to perform few-shot LLM text detection, why are the baselines not trained on the same training corpora?

We made an error in Section 4.1 when we mentioned that Reddit was used as an evaluation corpora. The evaluation datasets are Amazon, M4 PeerRead, M4 ArXiv, M4 WikiHow, and M4 Wikipedia. The Reddit corpora is used strictly for training, which enables us to evaluate robustness to distribution shift.

For baselines which require AAC data (MAML, Protonet, and RoBERTa), we train on a politics-only version of the Reddit dataset. We restrict the topic to politics to ensure that these baselines learn features that are agnostic to the domain. In Appendix B, we perform ablations with ProtoNet where we show that controlling the domain is essential to achieving good results. 

As mentioned earlier, the proposed style-based representation can be trained using only negative (human-written) examples, and as a result we can train on more datasets such as StackExchange.

Reviewer comment (Experiments):
What is the difference between Reddit (5M) and Reddit (used for SBERT) in Table 2?

Author response:
> What is the difference between Reddit (5M) and Reddit (used for SBERT) in Table 2

Thanks for catching this; the original text is inaccurate in this case. We used an off-the-shelf SBERT, specifically paraphrase-distilroberta-base-v1 which is actually trained on a variety of domains, including some of our domains intended to be held-out (such as Wikipedia). This is strictly to the advantage of the SBERT baseline since it suffers less from distribution shifts, while we control the training data of our stylistic representations to ensure the held-out evaluation domains do introduce distribution shifts. In other words, our baseline is given an unfair advantage relative to our proposed methods, yet we still outperform it, which we attribute to the stylistic training objective.

Reviewer comment (Presentation):
Why is Twitter from Table 2 not mentioned in Table 1?

Author response:
>  Why is Twitter from Table 2 not mentioned in Table 1? 

Twitter is only used as a training corpus for the Multi-LLM UAR. This is the reason it was not mentioned in Table 1 in the original draft.

Reviewer comment (Experiments):
How do the authors explain that the proposed UAR (based on SBERT) is still much better than SBERT? The authors emphasize that SBERT is unsuitable as it is trained to capture semantics. So why not use a "normal" BERT in UAR?

Author response:
> How do the authors explain that the proposed UAR (based on SBERT) is still much better than SBERT? The authors emphasize that SBERT is unsuitable as it is trained to capture semantics. So why not use a "normal" BERT in UAR?

The proposed UAR was trained to capture the stylistic differences that are agnostic to the topic being written about. SBERT was trained to capture the semantic similarity, hence its embedding intrinsically captures the topic of a piece of text. The stylistic features of LLMs remain quite stable across the various domains considered, hence the effectiveness of the UAR in capturing the differences in writing styles between different LLMs. The reason to initialize UAR with SBERT over BERT is simply that SBERT was also trained with a contrastive objective, although in practice we suspect that initializing from BERT would perform equally well since the UAR training data is very large (hundreds of millions of documents) and hence the initialization is less critical.

Reviewer comment (Experiments):
Section 4.5: The text mentions "other UAR variants." Are the Multi-LLM and Multi-domain models UAR variants? This should be clarified. Also, again, the same problem as in the other sections: why not train the other methods with this setup?

Author response:
> Section 4.5: The text mentions "other UAR variants." Are the Multi-LLM and Multi-domain models UAR variants? This should be clarified. Also, again, the same problem as in the other sections: why not train the other methods with this setup?

Yes, they are UAR variants, which was mentioned in Section 3.2 of the original paper under the “Multi-domain” and “Multi-LLM” headers.

Reviewer comment (Experiments):
The approach assumes that documents generated from amply available and cheap (AAC) models are available at training time, which may not always be the case in practice.

Author response:
First of all, as mentioned in our response to all reviewers, we do intend to release the datasets we created and have clarified this point in our reproducibility statement. We note that such datasets are already amply available publically, and in fact, as mentioned in S4.1, the publicly available M4 dataset comprises a portion of our evaluation. For this reason we disagree with the reviewer’s assertion that AAC data may be unavailable in practice, unless the reviewer was referring to AAC data in low-resource languages, in which case we agree but would argue that there are fewer LLM of concern in such languages. In any case, for the sake of reproducing our results exactly, we agree that the datasets we generated must be released.

Reviewer comment (Evaluation):
The authors should consider additional qualitative and quantitative metrics to evaluate the specification tasks. The paper adopts only AUROC as the metric. However, others such as F1 and TPR are more reasonable to use. It is important to consider additional metrics to ensure consistency of performance especially in leveraging these algorithms for a product use case. The models could be compared based on a perplexity score-based approach to help find the disambiguation between human and ChatGPT-generated text. Here is a reference paper: https://arxiv.org/pdf/2301.13852.pdf

Author response:
Regarding reporting additional standard metrics, we report additional experiments in Appendix I on smaller writing samples (making the detection problem harder), where we include standard metrics. These results are reproduced below for convenience. 
 | **Method** | **Training Dataset** | **AUC** | **AUC'** | **FPR@95TPR** | 
 |     :----:    |     :----:    |     :----:    |     :----:    |     :----:    | 
 | **Few-Shot Methods** |  |  |  | 
 | LUAR  |  Reddit(5M)  |  **0.9884** | 0.9094 |  **0.0533** | 
 | LUAR  |  Reddit (5M), Twitter, StackExchange  |  **0.9884** |  **0.9118**  | 0.0543 | 
 | LUAR  |  AAC, Reddit (politics)  | 0.8913 | 0.7308 | 0.3394 | 
 | CISR  |  Reddit (hard neg/hard pos)  | 0.9608 | 0.7973 | 0.144 | 
 | RoBERTa (ProtoNet)  |  AAC, Reddit (politics)  | 0.9791 | 0.9062 | 0.0934 | 
 | RoBERTa (MAML)  |  AAC, Reddit (politics)  | 0.6686 | 0.5141 | 0.6555 | 
 | SBERT  |  SNLI, MultiNLI  | 0.9673 | 0.8087 | 0.1448 | 
 | **Zero-Shot Methods** |  |  |  |  | 
 | RoBERTa (Zero shot)  |  AAC, Reddit (politics)  | 0.7238 | 0.5295 | 0.5369 | 
 | OpenAI Detector  |  WebText, GPT2-XL  | 0.6933 | 0.5559 | 0.7698 | 
 | Rank (GPT2-XL)  |  BookCorpus, WebText  | 0.7301 | 0.5423 | 0.6341 | 
 | LogRank (GPT2-XL)  |  BookCorpus, WebText  | 0.9107 | 0.6395 | 0.2209 | 
 | Entropy (GPT2-XL)  |  BookCorpus, WebText  | 0.2083 | 0.4977 | 0.9667 | 
 | Random |  | 0.50 | 0.05 | -- |

Reviewer comment (Experiments):
The approach may not be effective against more advanced language models that can better mimic human authorship.

Author response:
The reviewer’s concern about the robustness of our approach to future, more fluent language models was precisely the motivation for our research methodology, which simulates precisely this situation. Indeed, we chose to evaluate documents by LWD models, which produce more fluent text than the AAC models used at training time for some proposed variations of style representations, and were developed in the future relative to those models. We also note that the proposed variation of style representations not availing of any machine generated data still detects LWD models better than baseline models in many cases (see Table 1 in the revised paper, or the reproduced table above), some of which are trained with AAC.

Reviewer comment (Evaluation):
The exact setting is not clear, and the presentation of the results under a single value (AUC) makes it hard to assess the impact of this work. In particular, it would be good to see some more examples to get a better intuition.

Author response:
Our previous use of the AUC’ metric may have insinuated that the approach is only relevant for settings requiring low false positive rates. However, this is not the case, as demonstrated in the additional results reported in Appendix H and reproduced below. Here, we vary the writing sample size to yield a harder detection problem, and additionally report the standard AUC and FPR@95TPR metrics. Note that the proposed style representation methods outperform all competing methods for these standard metrics, including the additional zero-shot baselines.
 | **Method** | **Training Dataset** | **AUC** | **AUC'** | **FPR@95TPR** | 
 |     :----:    |     :----:    |     :----:    |     :----:    |     :----:    | 
 | **Few-Shot Methods** |  |  |  | 
 | LUAR  |  Reddit(5M)  |  **0.9884** | 0.9094 |  **0.0533** | 
 | LUAR  |  Reddit (5M), Twitter, StackExchange  |  **0.9884** |  **0.9118**  | 0.0543 | 
 | LUAR  |  AAC, Reddit (politics)  | 0.8913 | 0.7308 | 0.3394 | 
 | CISR  |  Reddit (hard neg/hard pos)  | 0.9608 | 0.7973 | 0.144 | 
 | RoBERTa (ProtoNet)  |  AAC, Reddit (politics)  | 0.9791 | 0.9062 | 0.0934 | 
 | RoBERTa (MAML)  |  AAC, Reddit (politics)  | 0.6686 | 0.5141 | 0.6555 | 
 | SBERT  |  SNLI, MultiNLI  | 0.9673 | 0.8087 | 0.1448 | 
 | **Zero-Shot Methods** |  |  |  |  | 
 | RoBERTa (Zero shot)  |  AAC, Reddit (politics)  | 0.7238 | 0.5295 | 0.5369 | 
 | OpenAI Detector  |  WebText, GPT2-XL  | 0.6933 | 0.5559 | 0.7698 | 
 | Rank (GPT2-XL)  |  BookCorpus, WebText  | 0.7301 | 0.5423 | 0.6341 | 
 | LogRank (GPT2-XL)  |  BookCorpus, WebText  | 0.9107 | 0.6395 | 0.2209 | 
 | Entropy (GPT2-XL)  |  BookCorpus, WebText  | 0.2083 | 0.4977 | 0.9667 | 
 | Random |  | 0.50 | 0.05 | -- | 

Note that Rank, LogRank, and Entropy are standard methods from the literature:
* Rank - https://arxiv.org/pdf/1906.04043.pdf
* LogRank - https://arxiv.org/ftp/arxiv/papers/1908/1908.09203.pdf
* Entropy - https://aclanthology.org/2020.acl-main.164.pdf

Reviewer comment (Evaluation):
The evaluation examples seem easily detectable because the LLM mimics the provided persona so well as to insist on it in the generated text (`...as a <persona>, I...`). This might be the reason for the comparably high scores, as opposed to previous work, which generated non–persona-guided text instead.

Author response:
We thank the reviewer for their keen observation about potential artifacts of our prompting scheme appearing in the LM-generated documents. This artifact occurs only with LM-generated documents in the Reddit domain, which were not used in any experiments except the one reported in Appendix A, where they were used for evaluation rather than training. Therefore the artifact does not help any model to detect machine-generated documents. We have clarified where this generated data is employed in the revised paper to avoid confusion.

Reviewer comment (Novelty):
The field of detecting machine-generated text is still nascent, but this paper seems to miss some relevant references:  
- Daphne Ippolito, Daniel Duckworth, Chris CallisonBurch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled.  
- Antonis Maronikolakis, Mark Stevenson, and Hinrich Schutze. 2020. Transformers are better than humans at identifying generated text.  
- Matthias Gallé, Jos Rozen, Germán Kruszewski, and Hady Elsahar. 2021. Unsupervised and Distributional Detection of Machine-Generated Text.

Author response:
We also thank the reviewer for the references which we have included in the Related Work section of the revised paper.