Paper: On the Generalization of Training-based ChatGPT Detection Methods (ICLR-2024)

Reviewer comment (Novelty):
ChatGPT Direction seems to be not well motivated. It needs a why, not just a what and a how.

Author response:
**1. What is "ChatGPT Direction'' and how is it motivated?**

To have a further explanation on "ChatGPT Direction'', it would be helpful for us to first recall the concept about "Irrelevant Direction''.  In Section 4 of our paper, we provide evidence showing that the detection models could learn "**irrelevant features**'' for prediction. For example, in Section 4.2, we show the possibility of a model to use "**length**'' as a deterministic feature to make prediction. As an evidence, Figure 5 (c) shows a detection model could have a high risk to predict short ChatGPT texts to be human written, because its training dataset has more short human written texts. However, since ChatGPT can be utilized to generate longer or shorter texts, the lengths should not be treated as the principal features for prediction. In our paper, we define the "**irrelevant direction**'' which contains these irrelevant features. Our analysis in Section 4.1 also finds that the collected ChatGPT generated texts from certain prompts could possibly differ from human texts in this "irrelevant direction'', which cause them have a poor HC-Alignment. In this way, the trained detection model can make predictions based on irrelevant features and have a poor generalization.


Orthogonal to the irrelevant direction, we define the  "**ChatGPT direction**'' which is the direction that contains the principal features to distinguish human and ChatGPT texts. From our experiments, we demonstrate if one can prevent the detection model to learn irrelevant features, the models can have a better generalization, which validate the existence of principal features. Specifically: In Figure 5(b) in Section 4.2, if we train on ChatGPT texts with similar length distribution as human texts, the model can correctly predict both short and long texts. Based on these empirical findings, we verify the model can also learn principal features for prediction with better generalization. Thus, we are motivated to define this "ChatGPT Direction'' which contains these principal features. In our theoretical analysis in Section 4.3, the ChatGPT direction is the direction that determines the ground truth label (ChatGPT generated or human written) of data samples, and it is orthogonal to the irrelevant direction.

Reviewer comment (Novelty):
This work exclusively discusses the train-based methods, which are smaller in scope.

Author response:
**2. Why we focus on the training-based detection methods?**


We are happy to discuss why we believe it is of great importance to have a specified and comprehensive study for training-based detection methods. First, the training-based methods have high "**in-distribution**'' performance. For example, in our empirical study in Table 3 in the paper, we compare the training-based methods with other methods beyond training-based methods, including "similarity-based'' methods and "score-based'' methods. From the result, we found that training-based methods can provide much more promising detection performance compared to other methods. Second, the training-based methods have high "**applicability**'' for ChatGPT detection and LLM-generated text detection in general. Notably, some well-known LLM-generated detection methods [1-2] are not applicable to black-box LLMs like ChatGPT. In specifics, the method [1] relies on the knowledge of model prediction score, and the method [2] needs to manipulate the text sampling process. However, the training-based methods only need to collect the human texts and LLM generated texts. This process is more general and applicable to all LLMs. Given the superior performance and wide applicability, we believe it is worth a comprehensive study especially for training based methods. Notably, in addition to the analysis, our collected dataset, HC-Var, is one of the most comprehensive and versatile dataset for the research of ChatGPT detection until now, which can also support the evaluation for various kind of detection techniques. 


**Reference**

[1] DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature, Mitchell et al, ICML 2023

[2] A Watermark for Large Language Models, Kirchenbauer et al, ICML 2023

Reviewer comment (Writing):
Could you elaborate on ChatGPT Direction? How is it motivated?

Author response:
**1. What is "ChatGPT Direction'' and how is it motivated?**

To have a further explanation on "ChatGPT Direction'', it would be helpful for us to first recall the concept about "Irrelevant Direction''.  In Section 4 of our paper, we provide evidence showing that the detection models could learn "**irrelevant features**'' for prediction. For example, in Section 4.2, we show the possibility of a model to use "**length**'' as a deterministic feature to make prediction. As an evidence, Figure 5 (c) shows a detection model could have a high risk to predict short ChatGPT texts to be human written, because its training dataset has more short human written texts. However, since ChatGPT can be utilized to generate longer or shorter texts, the lengths should not be treated as the principal features for prediction. In our paper, we define the "**irrelevant direction**'' which contains these irrelevant features. Our analysis in Section 4.1 also finds that the collected ChatGPT generated texts from certain prompts could possibly differ from human texts in this "irrelevant direction'', which cause them have a poor HC-Alignment. In this way, the trained detection model can make predictions based on irrelevant features and have a poor generalization.


Orthogonal to the irrelevant direction, we define the  "**ChatGPT direction**'' which is the direction that contains the principal features to distinguish human and ChatGPT texts. From our experiments, we demonstrate if one can prevent the detection model to learn irrelevant features, the models can have a better generalization, which validate the existence of principal features. Specifically: In Figure 5(b) in Section 4.2, if we train on ChatGPT texts with similar length distribution as human texts, the model can correctly predict both short and long texts. Based on these empirical findings, we verify the model can also learn principal features for prediction with better generalization. Thus, we are motivated to define this "ChatGPT Direction'' which contains these principal features. In our theoretical analysis in Section 4.3, the ChatGPT direction is the direction that determines the ground truth label (ChatGPT generated or human written) of data samples, and it is orthogonal to the irrelevant direction.

Reviewer comment (Experiments):
The experiments are limited to CHATGPT, we do not know whether these conclusions still hold in GPT4 or other open-source LLMs.

Author response:
**2. Whether the findings are valid for other LLMs?**

We follow the reviewer's suggestion to validate our major conclusions in other LLMs beyond ChatGPT, including GPT-4, Llama-2 (7b-chat) and PaLM2. We include the detailed results in Appendix E.1 in our revised submission. Overall, we can draw observations consistent to our conclusions in the paper. Below, we are happy to discuss some of these experimental results.
1. **Generalization to unseen prompts.** In our paper, we claim: training with prompts with higher HC-Alignment can have better generalization to unseen prompts. In Figure 26 in Appendix E.1, we calculate the HC-Alignment of different prompts generated by different models. In Figure 27, we also get a similar finding which is the prompts with higher HC-Alignment can have better generalization. Notably, an interesting finding is that the HC-Alignment of different prompts are not the same in different LLMs. For example, in the task QA, the prompt P3 has highest HC-Alignment in ChatGPT. However, in other models like Llama-2 and PaLM2, the prompt P2 has the highest HC-Alignment. 

2. **Generalization to unseen topics / domain.** We conduct a similar experiment especially under Llama-2 generated texts to study the detection model's generalization to unseen topics in QA. The table below reports the result for Llama-2 where the experimental setup resembles Table 7 in our paper. From our result in Llama-2, we can draw similar conclusions: the model can have a compromised performance when testing on samples from unseen topics, which is denoted as ``no transfer'' in table below. However, if we apply transfer learning, such as fine-tuning (FT) or linear probing (LP) with a few samples, we can significantly improve the detection performance in the target domain. 

|              |   |       | finance |       |   |   |   |       | history |       |   |   |   |       | medical |       |   |   |   |       | science |       |
|--------------|---|-------|---------|-------|---|---|---|-------|---------|-------|---|---|---|-------|---------|-------|---|---|---|-------|---------|-------|
|              |   | h->f  | m->f    | s->f  |   |   |   | f->h  | m->h    | s->h  |   |   |   | f->m  | h->m    | s->m  |   |   |   | f->s  | h->s    | m->s  |
| No Transfer  |   | 0.979 | 0.841   | 0.980 |   |   |   | 0.977 | 0.902   | 0.994 |   |   |   | 0.762 | 0.885   | 0.796 |   |   |   | 0.975 | 0.991   | 0.881 |
| LP-5         |   | 0.975 | 0.898   | 0.975 |   |   |   | 0.977 | 0.926   | 0.995 |   |   |   | 0.923 | 0.955   | 0.965 |   |   |   | 0.979 | 0.990   | 0.910 |
| FT-5         |   | 0.952 | 0.921   | 0.911 |   |   |   | 0.974 | 0.942   | 0.958 |   |   |   | 0.873 | 0.956   | 0.887 |   |   |   | 0.952 | 0.962   | 0.930 |
| LP-Scratch-5 |   | 0.725 | 0.725   | 0.725 |   |   |   | 0.702 | 0.702   | 0.702 |   |   |   | 0.792 | 0.792   | 0.792 |   |   |   | 0.751 | 0.751   | 0.751 |
| FT-Scratch-5 |   | 0.821 | 0.821   | 0.821 |   |   |   | 0.740 | 0.740   | 0.740 |   |   |   | 0.734 | 0.734   | 0.734 |   |   |   | 0.689 | 0.689   | 0.689 |

Reviewer comment (Novelty):
Most findings seem obvious.

Author response:
**3. Whether the finding is obvious?**

The findings of this paper are established through investigation based on our collected dataset HC-Var. Thus, we would first highlight the contribution of our dataset. As mentioned in Section 3 in our paper, compared to other datasets for ChatGPT detection, our dataset is the first one to consider the factors including the prompts and lengths during data collection. Notably, before our dataset, one of the most investigated datasets for ChatGPT detection, HC-3 [1], only considers one prompt to inquire ChatGPT outputs and also neglects the impact of lengths. Therefore, our collected dataset is a good supplementary to the literature of existing works.

Our findings can be drawn thanks to the diversity of different factors that are provided by our dataset. 
Therefore, we respectively disagree that our findings are obvious. In fact, our findings are novel compared to existing works. Especially, through our analysis on the generalization across different prompts, we disclose the pitfall of only collecting samples far away from human data. This is not adequately addressed by previous works. Moreover, each of our findings has been subjected to rigorous examination, including using numerical metrics such as MAUVE [2], visualization, and theories. These findings can also be treated as preliminary insights which pave the way for further research.

 **Reference:**

[1] How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection, Guo et al, 2023.

[2] MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers, Pillutla, et al, 2021

Reviewer comment (Experiments):
How are these findings valid to other LLMs?

Author response:
**2. Whether the findings are valid for other LLMs?**

We follow the reviewer's suggestion to validate our major conclusions in other LLMs beyond ChatGPT, including GPT-4, Llama-2 (7b-chat) and PaLM2. We include the detailed results in Appendix E.1 in our revised submission. Overall, we can draw observations consistent to our conclusions in the paper. Below, we are happy to discuss some of these experimental results.
1. **Generalization to unseen prompts.** In our paper, we claim: training with prompts with higher HC-Alignment can have better generalization to unseen prompts. In Figure 26 in Appendix E.1, we calculate the HC-Alignment of different prompts generated by different models. In Figure 27, we also get a similar finding which is the prompts with higher HC-Alignment can have better generalization. Notably, an interesting finding is that the HC-Alignment of different prompts are not the same in different LLMs. For example, in the task QA, the prompt P3 has highest HC-Alignment in ChatGPT. However, in other models like Llama-2 and PaLM2, the prompt P2 has the highest HC-Alignment. 

2. **Generalization to unseen topics / domain.** We conduct a similar experiment especially under Llama-2 generated texts to study the detection model's generalization to unseen topics in QA. The table below reports the result for Llama-2 where the experimental setup resembles Table 7 in our paper. From our result in Llama-2, we can draw similar conclusions: the model can have a compromised performance when testing on samples from unseen topics, which is denoted as ``no transfer'' in table below. However, if we apply transfer learning, such as fine-tuning (FT) or linear probing (LP) with a few samples, we can significantly improve the detection performance in the target domain. 

|              |   |       | finance |       |   |   |   |       | history |       |   |   |   |       | medical |       |   |   |   |       | science |       |
|--------------|---|-------|---------|-------|---|---|---|-------|---------|-------|---|---|---|-------|---------|-------|---|---|---|-------|---------|-------|
|              |   | h->f  | m->f    | s->f  |   |   |   | f->h  | m->h    | s->h  |   |   |   | f->m  | h->m    | s->m  |   |   |   | f->s  | h->s    | m->s  |
| No Transfer  |   | 0.979 | 0.841   | 0.980 |   |   |   | 0.977 | 0.902   | 0.994 |   |   |   | 0.762 | 0.885   | 0.796 |   |   |   | 0.975 | 0.991   | 0.881 |
| LP-5         |   | 0.975 | 0.898   | 0.975 |   |   |   | 0.977 | 0.926   | 0.995 |   |   |   | 0.923 | 0.955   | 0.965 |   |   |   | 0.979 | 0.990   | 0.910 |
| FT-5         |   | 0.952 | 0.921   | 0.911 |   |   |   | 0.974 | 0.942   | 0.958 |   |   |   | 0.873 | 0.956   | 0.887 |   |   |   | 0.952 | 0.962   | 0.930 |
| LP-Scratch-5 |   | 0.725 | 0.725   | 0.725 |   |   |   | 0.702 | 0.702   | 0.702 |   |   |   | 0.792 | 0.792   | 0.792 |   |   |   | 0.751 | 0.751   | 0.751 |
| FT-Scratch-5 |   | 0.821 | 0.821   | 0.821 |   |   |   | 0.740 | 0.740   | 0.740 |   |   |   | 0.734 | 0.734   | 0.734 |   |   |   | 0.689 | 0.689   | 0.689 |