How Sampling Affects the Detectability of Machine-written texts: A Comprehensive Study (2025)

Abstract: As texts generated by Large Language Models (LLMs) are ever more common and often indistinguishable from human-written content, research on automatic text detection has attracted growing attention. Many recent detectors report near-perfect accuracy, often boasting AUROC scores above 99%. However, these claims typically assume fixed generation settings, leaving open the question of how robust such systems are to changes in decoding strategies. In this work, we systematically examine how sampling-based decoding impacts detectability, with a focus on how subtle variations in a model’s (sub)word-level distribution affect detection performance. We find that even minor adjustments to decoding parameters - such as temperature, top-p, or nucleus sampling - can severely impair detector accuracy, with AUROC dropping from near-perfect levels to 1% in some settings. Our findings expose critical blind spots in current detection methods and emphasize the need for more comprehensive evaluation protocols. To facilitate future research, we release a large-scale dataset encompassing 37 decoding configurations, along with our code and evaluation framework https://github.com/BaggerOfWords/Sampling-and-Detection .

## introduction texts generated with large language models (llms) have become almost indiscernible from human-written samples @xcite @xcite , raising concerns regarding their use for deception or harassment, among others misuses @xcite @xcite . to answer such threats, the task of artificial text detection (atd) has become increasingly popular, spurring competitions such as the voight-kampff task @pan24 @xcite or the machine-generated text detection task @col-ing25 @xcite . with top-@xmath0.7 : with repetition-@xmath1.05 ## key takeaways observation n o 1. the detection of machinewritten texts is not fully solved, even in supervised settings. as shown by the last experiment in § 6.2, human samples that are "out-of-domain" (news articles from another source than the training data) are poorly identified by a strong roberta classifier. in addition, it seems that some generation strategies, notably using repetition-penalties, need to be paid special attention, when preparing the training data for the detector. observation n o 2. existing benchmarks do not sufficiently represent all decoding strategies. if changing one parameter value (e.g., the temperature) seriously derails detectors, it is likely that they would also struggle to identify more elaborate generation techniques such as minimum bayes risk (mbr) decoding @xcite or monte carlo tree search (mcts) @xcite . observation n o 3. two state-of-the-art unsupervised detection systems need their detector models to assign very similar probabilities to artificial texts; failures to do so will cause on overdetection of natural texts ( ## conclusion in this work, we have presented a systematic study of how sampling-based decoding strategies affect the detectability of texts produced by large language models. considering 37 different configurations-varying multiple generation parameters, we have shown that even modest adjustments in the (sub)word-level generation distribution can undermine state-of-the-art detectors, observing auroc drops from near-perfect down to null simply by tuning the repetition-penalty. our analysis sheds light on the underlying failure mechanisms: different sampling techniques induce distinctive distributional signatures (e.g., changes in token entropy or frequency tails) that current detectors are not able to recognize. this suggests that robustness in detection is still an open issue, both for supervised and unsupervised methods. by exposing blind spots in current systems, we aim to steer the field toward more reliable and trustworthy methods for distinguishing human and machine-generated text, an ability that is bound to become essential in an era of fluent and accessible generative models. we'd like to thank the reviewers for their feedback and especially reviewer xqub for his insightful comments, helping us improve and refine unclear parts of our paper. chatgpt was used to write comments and a readme in the released code, in order to make sure it is understandable and re-usable by everyone. this work was performed using hpc resources from genci-idris (grant 2023-ad011014903r1).