How Reliable Are AI -Generated-Text Detectors? An Assessment Framework Using Evasive Soft Prompts (2023)

Abstract: In recent years, there has been a rapid proliferation of AI-generated text, primarily driven by the release of powerful pre-trained language models (PLMs). To address the issue of misuse associated with AI-generated text, various high-performing detectors have been developed, including the OpenAI detector and the Stanford DetectGPT. In our study, we ask how reliable these detectors are. We answer the question by designing a novel approach that can prompt any PLM to generate text that evades these high-performing detectors. The proposed approach suggests a universal evasive prompt, a novel type of soft prompt, which guides PLMs in producing “human-like” text that can mislead the detectors. The novel universal evasive prompt is achieved in two steps: First, we create an evasive soft prompt tailored to a specific PLM through prompt tuning; and then, we leverage the transferability of soft prompts to transfer the learned evasive soft prompt from one PLM to another. Employing multiple PLMs in various writing tasks, we conduct extensive experiments to evaluate the efficacy of the evasive soft prompts in their evasion of state-of-the-art detectors.

## introduction recent advances in transformer-based pre-trained language models (plms), such as palm @xcite , gpt 3.5 @xcite , and gpt 4 (openai, 2023), have greatly improved natural language generation (nlg) capabilities. as a result, there is a proliferation of highly compelling ai-generated text across various writing tasks, including summarization, essay writing, academic and scientific writing, and journalism. while ai-generated text can be impressive, it also brings potential risks of misuse, such as academic fraud and the dissemination of ai-generated misinformation @xcite . consequently, to combat the misuse associated with human written write a news article ... .... ## methodology figure 2 illustrates the process of generating the universal evasive soft prompt, which involves two main steps: evasive soft prompt learning and evasive soft prompt transfer. in the first step, we learn an evasive soft prompt for a specific frozen plm (source plm -p lm s ). the second step is the evasive soft prompt transfer, which involves transferring the learned soft prompt to a frozen target plm (p lm t ). in the subsequent sections, we comprehensively discuss these two steps. ## experiment setup here we describes the experimental settings used to validate our framework, including the ai generators (plms), writing tasks (datasets), detection setup, baselines, and the implementation details of escape to support reproducibility. ## conclusion in this paper, we investigated the reliability of current state-of-the-art ai-generated-text detectors by introducing a novel universal evasive soft prompt framework. our framework, referred to as es-cape, was designed to learn an evasive soft prompt capable of efficiently guiding any plm in generating text that can effectively deceive the detectors. through experiments conducted on various prominent plms across different writing tasks, our results reveal the unreliability of existing aigenerated-text detectors when confronted with text generated by plms guided by evasive soft prompts. in future research, exploring the potential benefits of adversarial training in developing more robust detectors to combat the proposed evasive soft prompts would be an intriguing avenue to pursue.