Paraphrasing Attack Resilience of Various Machine-Generated Text Detection Methods (2025)

Abstract: The recent large-scale emergence of LLMs has left an open space for dealing with their consequences, such as plagiarism or the spread of false information on the Internet. Coupling this with the rise of AI detector bypassing tools, reliable machine-generated text detection is in increasingly high demand. We investigate the paraphrasing attack resilience of various machine-generated text detection methods, evaluating three approaches: fine-tuned RoBERTa, Binoculars, and text feature analysis, along with their ensembles using Random Forest classifiers. We discovered that Binoculars-inclusive ensembles yield the strongest results, but they also suffer the most significant losses during attacks. In this paper, we present the dichotomy of performance versus resilience in the world of AI text detection, which complicates the current perception of reliability among state-of-the-art techniques.

## introduction the widespread use of llms can be precarious when left unchecked, with the consequences ranging from intellectual dishonesty to the spread of fake news on social media. @xcite found that ai chatbots can easily produce both realistic-looking academic results and a polished manuscript that may well be accepted to a conference and published. since scientific research, especially medical, is often falsified, the emergence of such a possibility opens up a dangerous playing field @xcite . it was found that 14% of scientists were aware of colleagues who falsified results, whereas 72% of scientists knew of colleagues who engaged in questionable research practices @xcite . more incidents of ai being used in the case of fake news spreading on the internet can be found in the ethics statement. what is particularly concerning about this is that humans have been found to perform rather poorly on manual detection of ai-written text. in particular, human performance has shown to be only marginally better than random classification @xcite . in fact, in a study involving over 130 subjects, @xcite found that participants were only able to correctly identify ai-generated text with an accuracy rate of 24%. concerning the use of ai in academia, @xcite conducted an experiment where participants were to identify whether abstracts for academic papers were written by chatgpt or a human. they found that only 68% of the ai-detected abstracts were correctly classified. such a precedent makes a strong case for the necessity of precise automated ai text detection mechanisms. with the emergence of freely accessible sites such as zerogpt, detectgpt, and quillbot, bypassing attacks have been developed against these technologies. methods which are commonly used include automated paraphrasing tools, prompt engineering, and the calculated addition of errors into ai-generated text @xcite . it has been generally shown that the use of these methods decreases the efficacy of the detection tool; however, we aim to put together a more comprehensive analysis of the leading ai detection methods against bypassing attacks. in this paper, we will focus on paraphrasing attacks. the leading state-of-the-art detectors can be categorized into two paradigms, those using trainingbased and training-free mechanisms @xcite . most training-free approaches rely on statistical feature analysis and commonly look at perplexity, log probability, and n-grams @xcite . although training-based models have been widely leading, a recently developed methodology -binoculars -proves successful in a zeroshot context, which stands out over multiple metrics @xcite . this approach is developed further in the related work section. trainingbased methods largely rely on transformer models, namely roberta @xcite , a maskedbased model, easily fine-tunable for downstream tasks such as text classification. the methods we stacked to develop our own model include binoculars, roberta, and text feature analysis, which we justify due to their leading benchmarks (detailed in related work). ## data and methodology to track the progress on machine-generated text detection, we use the materials of the competition on detecting ai generated content @coling 2025 task 1: binary machine-generated text detection @xcite . it is an aggregation of other datasets that have been studied before, such as m4gt. the experiments in the following sections are based on the testing dataset that the final leaderboard used. all models use the training dataset, which is described in appendix a.0.1. first, we chose to fine-tune roberta for ai text detection because it provided a substantial improvement in the model's ability to understand nuanced language differences. in essence, we added a final layer of size 2 for binary classification. it is also a well-tested approach in machine-generated text detection @xcite . we performed finetuning over a subset (12k entries) of the training set provided by the workshop. the hyperparameters are learning @xmath0, batch @xmath1, @xmath2, and training @xmath3, 000, with a train/test split of 0.8. second, we also test the binoculars method, which reaches high accuracy and low false-positive rates over multiple llm tested on, without relying on training data. binoculars uses two closely related llms -'tiiuae/falcon-7b', 'tiiuae/falcon-7b-instruct' -to calculate cross-perplexity, meaning perplexity is calculated using the log perplexity of text generated by one llm and the next-token prediction of another. third, we measured several document metrics that are related to ai detection. we selected 5 text markers: average word length, lexical diversity, punctuation frequency (corizzo and leal-arenas, 2023), standard deviation of sentence length, and stopword ratio @xcite . the selection of features was based on the entropy values from the random forests. we combined the features extracted from each approach into a single vector for each text sample and fed it to the random forests model that acts like a meta-learner. this vector includes the prediction probabilities from the fine-tuned roberta model as well as the predicted labels, the crossperplexity scores from binoculars, and the five document metrics we selected (fig 1 ). in the following sections, we will show the performance of all 7 different stackings of the models. since we had limited resources, we manually chose 201 random entries from the evaluation dataset, with the ai label, and fed them to the high-performance ai text detector bypasser gptinf. we concatenated these paraphrased entries with 201 randomly selected human-written entries from the same evaluation dataset. gptinf claims to bypass all ai detectors, including turnitin ai detector, gptzero, zerogpt, and gptradar. the dataset is published now on huggingface at 'antebe1/paraphrased_ai_text'. although the algorithm used by gptinf is not publicized, their website states that it works by paraphrasing the inputted text-removing common phrasing and diversifying sentence structure by varying the wording, grammar, and ordering of words used (gpt). to calculate the confidence interval (ci) for the f1 score on the full dataset, we used a bootstrapping approach (9000 out of 73k). to verify whether the differences between modules tested on were significant, we ran 21 pair-wise mc-nemar statistical tests ( ## conclusion we believe that the tradeoff between performance and resilience is significant enough to become a leading theme in the ai-detection community. for example, the reported high performance of binoculars on flagging machine-generated text has suffered the most drastic loss under paraphrasing attacks. under our testing framework, we also reaffirmed the significant information gain provided by the stacking of multiple detectors.