SPADE : Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detection (2025)

Abstract: The increasing capability of large language models (LLMs) to generate synthetic content has heightened concerns about their misuse, driving the development of Machine-Generated Text (MGT) detection models. However, these detectors face significant challenges due to the lack of high-quality synthetic datasets for training. To address this issue, we propose SPADE, a structured framework for detecting synthetic dialogues using prompt-based adversarial samples. Our proposed methods yield 14 new dialogue datasets, which we benchmark against eight MGT detection models. The results demonstrate improved generalization performance when utilizing a mixed dataset produced by proposed augmentation frameworks, offering a practical approach to enhancing LLM application security. Considering that real-world agents lack knowledge of future opponent utterances, we simulate online dialogue detection and examine the relationship between chat history length and detection accuracy. Our open-source datasets can be downloaded.

## introduction large language models (llms) are increasingly deployed in conversational systems, but their accessibility also enables adversaries to launch automated attacks. for instance, in online customer service chatrooms, attackers may use llms to launch prompt injection attacks that spread misinformation, or flood the system with realistic but excessive queries, leading to denial-of-service outcomes @xcite . these scenarios highlight the pressing need for high-performance machine-generated * both authors contributed equally to this research. ## data augmentation framework this section details the proposed training-free dialogue data augmentation frameworks designed to generate high-quality synthetic dialogues, from bona fide human-generated dialogues. these frameworks fall into two main categories: partial-chatbot data augmentation and full-chatbot data augmentation. figure 1 outlines the abstract construction process of each framework. appendix a.1 provides example dialogues and complete prompts. ## dialogue data construction this section outlines the preprocessing of the full-human dataset, synthetic data generation, and quality assurance. ## offline dialogue detection this section focuses on offline dialogue detection, where a single-stage mgt detection model is applied to all user responses in a dialogue simultaneously. we evaluated the performance of our augmented datasets using eight models for mgt detection. ## online dialogue detection in an online dialogue setting, both the customer service system and the user can access the chat history but lack knowledge of future utterances. this dynamic introduces a new challenge: detecting chatbot users based on incomplete chat history, with updates occurring as new user utterances are progressively received. unlike common mgt detection, which receives a passage as input, a new challenge is to detect the chatbot user based on the incomplete chat history and update the detection as user utterances are progressively received. the proposed e2e convo. framework addresses this challenge by emulating an online environment with turn-based interaction between a system and a user. to conduct online dialogue detection on the generated dataset, we use the high-performing models in the offline dialogue detection setting (section 5.4, including mlp, logr, and roberta). experiments were conducted on the e2e convo., par., and g2d datasets to explore performance trends across different prompting structures. models were initially trained on complete dialogues, and during detection, the input dialogues were progressively cropped to varying lengths while maintaining the same binary classification labels as the full dialogues. we tested the models on dialogues cropped to 𝑘 ∈ {1, 2, 3, 4} user utterances, as dialogues with less than k user utterances are excluded from testing datasets, and many dialogues terminate after 4 utterances. on both feature-based models (mlp and logr) and a plm-based model (roberta), figure 3 reveals a positive correlation between detection accuracy and the number of user utterances across all datasets. as shown in ## conclusion the scarcity of high-quality datasets for mgt detection in customer service remains challenging. this paper introduced five structured data augmentation frameworks to reduce the costs of traditional dialogue collection, using prompting techniques to efficiently generate synthetic dialogues. we derived 14 new datasets and evaluating these across eight mgt detection models, we found that training on a mix of these datasets significantly improves generalization. simulated online dialogue detection showed that longer chat histories enhance detection accuracy. given the rising threats of misused llms in real-world systems, spade contributes a practical and extensible methodology for improving llm safety.