On the Generalization of Training-based C hat GPT Detection Methods (2024)

Abstract: Large language models, such as ChatGPT, achieve amazing performance on various language processing tasks. However, they can also be exploited for improper purposes such as plagiarism or misinformation dissemination. Thus, there is an urgent need to detect the texts generated by LLMs. One type of most studied methods trains classification models to distinguish LLM texts from human texts. However, existing studies demonstrate the trained models may suffer from distribution shifts (during test), i.e., they are ineffective to predict the generated texts from unseen language tasks or topics which are not collected during training. In this work, we focus on ChatGPT as a representative model, and we conduct a comprehensive investigation on these methods’ generalization behaviors under distribution shift caused by a wide range of factors, including prompts, text lengths, topics, and language tasks. To achieve this goal, we first collect a new dataset with human and ChatGPT texts, and then we conduct extensive studies on the collected dataset. Our studies unveil insightful findings that provide guidance for future methodologies and data collection strategies for LLM detection.

## introduction large language models (llms) have demonstrated a great versatility to handle diverse language tasks, including question answering @xcite , creative writing @xcite and personal assistance @xcite . meanwhile, it also gives rise to an urgent need for detecting llm generated texts from human written texts to regulate the proper use of llms. for example, they can be misused to accomplish the tasks such as producing fake news or generating fake reviews @xcite , leading to public deception. similarly, they can be also used for plagiarism, offending people's intellectual property @xcite , which can cause severe negative consequences to society. in this paper, we focus one type of the most popular methods @xcite , which propose to train classification models to distinguish human and llm generated texts, called "training-based methods". these methods do not require the full knowledge of the parameter of the source llm, which enables them to detect contents from black-box models like chatgpt. however, recent studies @xcite demonstrated that these training-based methods tend to be overfitted to their training data distribution. for instance, @xcite show that a roberta classification model @xcite trained on hc-3 dataset @xcite for detecting chatgpt answered questions will exhibit a notable accuracy decrease if it is tested on some specific topics (i.e., finance and medicine). @xcite also find the detection models trained on hc-3 struggle to detect chatgpt written news or scientific paper abstracts. therefore, we are motivated to have a systematical study on the generalization behavior of these training-based methods. to achieve this goal, we focus on chatgpt (gpt-3.5-turbo) as a representative model, and we consider the impacts from four types of distribution shifts (between training and test distribution). specifically, in addition to the topic and task shift as observed in previous works, we consider two other factors which are not identified or adequately discussed, including: • prompts to inquire chatgpt outputs: a user can have various prompts to obtain chatgpt outputs. for example, when asking chatgpt to write a movie review, a user can ask "write a review for the @xmath0. alternatively, they can let chatgpt give comments to the movie via asking it to complete a dialogue that reflects the preference of the talkers to this movie (see section 3 for details). in section 4.1, we find the texts from different prompts could have significant distributional difference. in practice, because only a limited number of training data can be collected, the training data cannot fully cover the distribution of test data. thus, it is critical to deeply understand the detection models' generalization behaviors when the distribution shifts occur. to achieve this goal, we first collect a new text dataset, named hc-var (human chatgpt texts with variety), which contains human texts and chatgpt outputs by considering multiple types of variety, including prompts, lengths, topics and language tasks (see section 3). facilitated with hc-var, we conduct comprehensive analysis on the models' generalization, when facing the aforementioned distribution shifts. via extensive experiments, we draw key conclusions: • in section 4, we analyze the generalization under distribution shift due to the change of prompts and length. we identify one factor that can hurt the detection models' generalization: the trained classification models tend to overfit to some "irrelevant features" which are not principal for detection. this overfitting issue can be originated from the "incautious and insufficient" data collection process, which collects chatgpt texts that are distinct from human texts in these "irrelevant features". in section 4.3, we conduct a theoretical analysis to deeply understand the consequence of this phenomenon. ## related works in this section, we introduce background knowledge about existing methods for chatgpt generated text detection, as well as other detection methods for open-source language models. we also discuss existing research findings about the generalization of chatgpt detection methods. ## preliminary in this section, we first introduce the details of our proposed dataset, hc-var: human and chatgpt texts with variety. then we discuss the general experimental setups and evaluation metrics used in the paper. next, we conduct a preliminary comparison on existing methods under the "in-distribution" setting, before we discuss their generalization. ## generalization across topic & tasks in this section, we discuss the circumstances that the models can face texts from unforeseen language tasks or topics. overall, our claim in section 5.1 shows that the trained models could have high errors on unseen topics & tasks (which is consistent to previous works such as @xcite ). however, in section 5.2, we also observe that these models can still provide valuable information which helps fast adaptation. notably, here we only provide the results for task-level generalization, and we leave the topic-level study in appendix b, where we draw similar conclusions. ## conclusion in this paper, we conduct a comprehensive analysis on the generalization behavior of training-based chatgpt detection methods. we first show the trained models could overfit to irrelevant features which leads poor generalization. however, we also show the trained models can usually provide useful information that helps fast adaption to unseen topics and language tasks.