DAMAGE : Detecting Adversarially Modified AI Generated Text (2025)

Abstract: AI humanizers are a new class of online software tools meant to paraphrase and rewrite AI-generated text in a way that allows them to evade AI detection software. We study 19 AI humanizer and paraphrasing tools and qualitatively assess their effects and faithfulness in preserving the meaning of the original text. We show that many existing AI detectors fail to detect humanized text. Finally, we demonstrate a robust model that can detect humanized AI text while maintaining a low false positive rate using a data-centric augmentation approach. We attack our own detector, training our own fine-tuned model optimized against our detector’s predictions, and show that our detector’s cross-humanizer generalization is sufficient to remain robust to this attack.

## introduction the ability of large language models such as chat-gpt @xcite to generate realistic and fluent text has spurred the need for ai text detection software. commercial methods, such as turnitin, gptzero, originality, and pangram labs have emerged, as well as open-source research methods, such as detectgpt @xcite , binoculars @xcite , and many more. however, both researchers and practitioners alike have found these solutions to be fragile. a study from google research @xcite found that a paraphrasing text-to-text model (a variant of t5) was able to effectively rewrite aigenerated text in a way that could preserve the meaning of the original text but largely evade ai detection algorithms. this finding gave rise to an explosion of new ai "humanizer" tools appearing online. these tools promise to bypass ai detection tools by rewriting ai-generated text. they are primarily marketed at students, who can use these tools to effectively cheat on writing assignments by plagiarizing from large language models without getting caught. other humanizers target their product towards seo marketers, who may generate hundreds of blog posts using ai and apply humanizers to evade ai detection by search engine algorithms. in this work, we attempt to comprehensively study these ai humanizers: what they are doing, and whether it is possible to identify humanized ai-generated text. our main contributions are as follows. ## experiments in our experiments, we seek to answer the question of whether a deep learning based ai text classifier is capable of detecting humanized ai-generated text. first, we narrow our scope to l1 humanizers. we do this because their subtle changes are the hardest to detect by eye and because they have the highest levels of fluency, making them most relevant in real-world adversarial attacks. we train two models: one model is unaware of humanized text, and one model contains a small amount of humanized text from a variety of humanizers. we describe the methodology and training procedure for training these models here. ## detector-specific adversarial humanization in this paper, we study commercial online humanizers meant to generally evade ai detectors, but we also study the directly adversarial scenario: when a humanizer is directly optimized against a particular detector. to do so, we train our own humanizer using the gpt-4o fine-tuning api and measure the detector's robustness to ai-generated text passed through the adversarial fine-tuned model. ## conclusion in this study, we attempted to both qualitatively and quantitatively study humanizer tools and their effects on individual writing samples and the robustness of ai detectors against them. we showed it is possible to train an effective deep learning classifier that is robust to most humanizers, and ran a proof-of-concept to show that some degree of robustness even extends to a humanizer that is adversarially optimized against our specific detector. figures 6 and 7 demontrate jailbreaks that reveal the system prompt for the underlying paraphraser llm. figure 6 : this humanizer is an llm that is instructed to paraphrase "in a conversational tone".