Benchmarking AI Text Detection: Assessing Detectors Against New Datasets, Evasion Tactics, and Enhanced LLM s (2025)

Abstract: The rapid advancement of Large Language Models (LLMs), such as GPT-4, has sparked concerns regarding academic misconduct, misinformation, and the erosion of originality. Despite the growing number of AI detection tools, their effectiveness is often undermined by sophisticated evasion tactics and the continuous evolution of LLMs. This research benchmarks the performance of leading AI detectors, including OpenAI Detector, RADAR, and ArguGPT, across a variety of text domains, evaded content, and text generated by cutting-edge LLMs. Our experiments reveal that current detection models show considerable unreliability in real-world scenarios, particularly when tested against diverse data domains and novel evasion strategies. The study underscores the need for enhanced robustness in detection systems and provides valuable insights into areas of improvement for these models. Additionally, this work lays the groundwork for future research by offering a comprehensive evaluation of existing detectors under challenging conditions, fostering a deeper understanding of their limitations. The experimental code and datasets are publicly available for further benchmarking on Github.

## introduction llms such as gpt-3 have achieved strong performance on several tasks that require on-the-fly reasoning or domain adaptation, such as translation, question answering, and representing text in an intelligent, presentable form (brown, 2020). in addition, more advanced llms such as gpt-4 have achieved human-level performance on academic benchmarks, such as passing a simulated bar examination by scoring around the top 10% test takers @xcite . on top of that, openai released openai o1-preview, a new series of reasoning models claiming that these models perform on the level of phd students for challenging benchmark tasks such as graduate-level google-proof q&a (gpqa) benchmark in subjects like physics, biology and chemistry @xcite . these events highlight the exceptional capabilities of llms in academia and their fast-paced evolution. with exceptional capabilities, llms have also brought several threats like academic misconduct @xcite , such as students submitting assignments, passing examinations, misinformation @xcite , lack of creativity @xcite , and other several ethical concerns @xcite . to overcome these threats, efficient detection of texts generated from llms is necessary. much research has been done to build highly effective and robust llm-generated text detection algorithms. a training-based classifier, zero-shot detection, watermarking, and adversarial learning approach are some approaches used to create models that detect llm-generated text. the training-based classifier approaches treat the problem as a binary classification problem, and labelled datasets are trained on several algorithms to build models such as argugpt @xcite , ghostbuster @xcite and roberta-base-openai-detector @xcite . a zero-shot learning approach that allows a model to handle tasks it hasn't been explicitly trained for by using its existing general knowledge has also been used to develop detectors such as binoculars @xcite , fast-detectgpt @xcite , dna-gpt @xcite , and gltr @xcite . similarly, watermarking techniques by which subtle, identifiable patterns are embedded into the generated content, making it easier to recognise as ai-generated, has resulted in models such as postmark @xcite , waterfall @xcite , waterjudge @xcite , and watermax @xcite . other approaches, such as adversarial learning, which enhances the robustness of llm-generated text detection algorithms by exposing the detector to an evasion example while training, have been introduced, resulting in models like radar @xcite and outfox @xcite . along with developing llm detectors, several evasion techniques, such as paraphrasing and synonym replacement, have also been developed. these evasion techniques are applied after generating text from llms so that the detector cannot identify the text as ai-generated. these evasion techniques may be as simple as adding a single space randomly before a comma @xcite , deleting an article randomly in a sentence, inserting misspellings randomly in a sentence @xcite , and replacing some random characters with homoglyph characters @xcite . there are also more complex techniques such as paraphrasing @xcite , word substitution @xcite , sentence substitution @xcite and prompting @xcite (using instructions to generate human-written text while generating text) also have been successful in fooling ai detectors. thus, it becomes crucial to benchmark existing open-source, state-of-the-art ai detectors against these evasion strategies. one gap in developing efficient ai detectors is the reliance on subsets of a single dataset for training and testing, compromising the models' robustness @xcite . such detectors claim high accuracy. however, they typically fail when tested in real settings with very different data from the training and testing set. out-of-distribution testing of the existing open-source state-of-the-art detectors is crucial @xcite . in addition, it is essential to benchmark the existing ai detectors with new generators in the space. the rapid growth in the development of new llms with exceptional learning capabilities, along with the increasing number of parameters, has brought up new concepts such as reasoning @xcite , coherent and cohesive long-form text generation @xcite , and multilingual and cross-domain capabilities @xcite . this leads to the research question of whether the existing state-of-the-art ai detectors are up to date and capable of detecting text from new llms with such capabilities or not. thus, benchmarking such detectors against recent powerful llms is vital. this paper's main contribution is benchmarking existing ai detectors against different datasets (text from various domains, text created using evasion techniques, and text generated by the latest llms such as gpt4 o from openai and command r+ from cohere). this benchmarking allows for an in-depth analysis of the different detectors' performance in the context of their general approach to ai-generated text detection and the various types of datasets used. the rest of the paper is organised as follows. section 2 discusses the previous benchmarking research done for llm-generated text detection. section 3 explains the methodology used to perform the benchmarking experiment. section 4 highlights the results obtained from the experiment. section 5 discusses the analysis of the results obtained. section 7 finally presents the conclusions obtained from the experiment. ## literature review as the race for the development of robust ai detectors and the development of evasion strategies to fool ai detectors goes on, along with the development of even more powerful llms, research has been conducted to test the efficiency of existing llm-generated text detection algorithms developed so far. initially, the benchmarking experiments used human-written and ai-generated text with no further modifications. chaka @xcite did a comprehensive review of 17 published articles on testing ai detectors. the author found that the machine-generated text used in testing in those research papers was from chatgpt-3.5 and chatgpt-4 @xcite . madelyn a. et al. @xcite tested three ai detector tools, openai's ai classifer, content at scale, and originality.ai, on human-written scientific and ai-generated articles, which are not modified. the tool originality.ai achieved 100% accuracies in this testing @xcite . however, people may not just copypaste the text entirely from llms and may try to modify the text. weber-wulff et al. @xcite tested ai detection tools on three types of ai-generated text: ai-generated text, aigenerated text with subsequent human edits, and ai-generated text with subsequent machine paraphrasing and found those detectors were biased in classifying ai-generated text as human-written. similarly, elkhatat et al. @xcite ) also tested five ai detection tools on human-written control responses and saw an increase in false positives in the case of human-written control responses. these experiments against types of text observations (human-written, ai-generated and human editing on ai-generated texts) show that ai detectors are easily evaded with few further human edits on ai-generated texts. when evasion techniques are applied to aigenerated texts, ai detectors fail to perform well. krishna et al. @xcite developed an 11b parameter paraphrase generation model called dipper, which successfully evaded detectors such as watermarking, gptzero, detectgpt, and ope-nai's text classifier and further proposed a retrieval method which detected 80% to 97% of paraphrased generations across various settings, while only 1% of human-written text was mistakenly flagged as ai-generated. however, after 4 months, sadasivan et al. @xcite again introduced the recursive paraphrasing attack, which degraded the accuracy of several watermarking-based, zero-shotbased, neural-network-based and retrieval-based detectors. the adversarial learning approach has been introduced to develop recent ai-generated text detectors such as radar @xcite and outfox @xcite . there seems to be a gap in benchmarking these recently developed models on different scenarios. the challenge with building ai detectors is these detectors need to be able to generalise on unseen domain text @xcite . for example, existing ai detectors may fail when tested on text generated from recent llms released after the ai detector's release. dugan et al. @xcite tested neural and metrics-based ai detectors on ai-generated texts from 11 different llms. they found that the performance of these ai detectors varies according to the llms used for generating the text. however, with the release of even more powerful llms with reasoning capabilities like gpt-4o, it remains to be seen how these detection algorithms perform on the text generated from such new llms. ## methodology the benchmarking of existing open-source llmgenerated text detection algorithms was done on three significant aspects: out-of-distribution along with multiple domain testing, evasion applied dataset testing and new llm-generated text dataset testing. the data flow in the benchmarking experiment has been represented in fig 1 . initially, sampling was performed from two datasets, i.e., the m4 dataset and the hc3 dataset, resulting in a data subset of 3,000 ai-generated text observations and 3,000 ai-generated text observations. the 3,000 ai-generated observations were further edited using the six evasion strategies, and 3,000 evasionapplied ai-generated text observations were created for each evasion strategy. additionally, 1,000 new ai-generated text observations were generated from each of the recent llms, i.e. gpt-4 o and command r plus, using the same prompt used to create them previously. because of computational constraints, the number of observations was limited. the complete testing dataset representing multiple datasets, multiple evasion strategies, and multiple generators was passed to the ai-generated text detection algorithms, and the benchmarking results were obtained. ## experiments and results this section presents the results of the benchmarking experiment on different edge cases such as multiple domain data, multiple evasion applied data and multiple llms generated data. the testing of a model developed on another dataset rather than testing on the test set of its dataset is referred to as out-of-distribution testing. the benchmarking dataset used in this experiment was not used to train and test these models. such out-of-distribution testing of the ai detectors was done by passing the text sample observations to the models. prediction probability was obtained for each of the predictions. we set the threshold value to 0.5 to classify it as human-written or aigenerated. out-of-distribution set testing was performed with text observations from the m4 and hc3 datasets. we calculated the accuracy of the binary classification evaluation metrics, f1-score, false negatives, false positives, precision, and recall. the results obtained are shown in the detectors were benchmarked on different evasion strategies. the results from the benchmarking against the evasion strategies are shown in ## discussion and analysis this section discusses and analyses the experiment's results to report key insights. the discussion is organised according to the edge cases, i.e., out-ofdistribution and multi-domain, evasion strategies, and recent llms. we mainly analyse the false negatives (fn), which gives the number of aigenerated text observations that were misclassified as human-written, representing the inefficiency of the ai-detector tools. ## limitations the benchmarking experiment is done on three popular open-source ai text detection models. however, the framework and the datasets can be used for testing other ai text detection models also. six of the basic evasion strategies have been applied to generate data samples representing evasion applied to ai-generated text. this can be further enhanced by employing other additional evasion strategies such as adversarial prompting @xcite , substitution-based in-context example optimization method (sico) @xcite , self-color testing-based substitution @xcite , and reinforcement learning @xcite . ## conclusion the research highlighted significant critical challenges in detecting llm-generated text. the existing state-of-the-art algorithms for detecting text generated from llms could perform better when tested on text generated from other domains, llmgenerated text on which evasion techniques have been applied and text generated from recent llms. this leads to the conclusion that these algorithms cannot be fully relied upon and used in university assignment checkers and research publications checkers. from the results and analysis of the benchmarking against the evasion techniques, we can observe that even simple techniques, such as deleting a random article or misspelling a random word on ai-generated text, can bypass existing ai detectors. similarly, from the results and analysis from multi-ple benchmarking experiments, it can be concluded that training on diverse ai-generated text, including evasion techniques, domains, and outputs from various llms, improves detector robustness. furthermore, the knowledge extracted from the critical analysis of the models serves as the baseline for future researchers trying to build robust ai-generated text detection algorithms. training models representing a wide variety of data (multiple domains, multiple evasion techniques being applied, and multiple generators) may lead to the development of more efficient detectors. likewise, training models with an adversarial learning approach that aims to train the model in different adversarial attacks and scenarios also seems promising. the benchmarking in this research validates that the problem is far from solved. the knowledge gained from the critical analysis of the results concerning different approaches will help shape the further development of algorithms that can solve the problem more robustly. with the contribution of knowledge extracted from the experiment and thorough analysis of the results obtained, we aim to develop more robust ai detectors.