E vo B ench: Towards Real-world LLM -Generated Text Detection Benchmarking for Evolving Large Language Models (2025)

Abstract: With the widespread of Large Language Models (LLMs), there has been an increasing need to detect LLM-generated texts, prompting extensive research in this area. However, existing detection methods mainly evaluate on static benchmarks, which neglect the evolving nature of LLMs. Relying on existing static benchmarks could create a misleading sense of security, overestimating the real-world effectiveness of detection methods.To bridge this gap, we introduce EvoBench, a dynamic benchmark considering a new dimension of generalization across continuously evolving LLMs.EvoBench categorizes the evolving LLMs into (1) updates over time and (2) developments like finetuning and pruning, covering 7 LLM families and their 29 evolving versions. To measure the generalization across evolving LLMs, we introduce a new EMG (Evolving Model Generalization) metric. Our evaluation of 14 detection methods on EvoBench reveals that they all struggle to maintain generalization when confronted with evolving LLMs. To mitigate the generalization problems, we further propose improvement strategies. For zero-shot detectors, we propose pruning the scoring model to extract shared features. For supervised detectors, we also propose a practical training strategy.Our research sheds light on critical challenges in real-world LLM-generated text detection and represents a significant step toward practical applications.

## introduction large language models (llms), such as chat-gpt (openai, 2022b), claude (anthropic, 2024), and llama @xcite , have demonstrated remarkable capabilities in natural language evobench figure 1: current benchmarks @xcite @xcite primarily focus on specific versions of llms and neglect vast ecosystem of evolving llms beneath the surface, including updates over time or developments through fine-tuning or pruning. the figure primarily includes llms before jan 2025. the academic community has carried out extensive research to detect llm-generated text effectively @xcite @xcite . current methods can be categorized into supervised methods @xcite @xcite , 2024b; @xcite and zeroshot methods @xcite @xcite @xcite . supervised methods are typically trained with a binary classifier to distinguish between texts generated by llms and those created by humans, while zeroshot methods primarily rely on statistical features gathered from pre-trained large language models. although these methods show strong performance on existing benchmarks @xcite @xcite , they often fall short in real-world applications as current benchmarks neglect the evolving nature of llms @xcite @xcite @xcite . as illustrated in figure 1 , existing benchmarks include a limited versions of llms, much like observers seeing only the tip of an iceberg while overlooking the vast evolving llms hidden beneath the surface. in real-world applications, llms are continuously evolving via regular updates, fine tuning @xcite , or pruning @xcite , all of which affect llms' output @xcite @xcite , thereby impacting the detection performance. figure 2 illustrates that the detection performance of current detection methods suffers up to 25% drop as llms evolve. 1 therefore, relying on current static benchmarks for evaluation could create a misleading sense of security, leading the research community to overestimate the real-world effectiveness of detection methods. therefore, it is crucial to enhance existing benchmarks to better capture the ongoing evolving llms. in this paper, we propose evobench, a dynamic benchmark that extends traditional benchmarking to account for the evolving llms. evobench aims to provide a more accurate evaluation of detection methods by incorporating two dimensions of evolving llms: updates and developments, as shown in figure 1 . updates refer to changes made by the llm publishers, including llm updates, i.e., gpt-4o undergoes updates approximately every 3 months foot_2 . on the other hand, developments refer to optimizations made by llm developers for specific application scenarios, such as fine-tuning and pruning. evobench introduces a new dimension of generalization, enabling the evaluation of detection methods across evolving llms, covering 7 widely used llms and their 29 evolving versions. using evobench, we evaluate 14 widely used detection methods and find that all struggle to adapt to the evolving nature of llms. to intuitive quantify the generalization across evolving llms, we introduce the emg (evolving model generalization) metric. results reveal that the performance of 14 current detection methods, including widely used fast-detectgpt @xcite and radar @xcite detectors, significantly drop when faced with evolving versions, as reflected by low emg values, while some achieve high auroc (0.91) scores on individual models foot_3 . this phenomenon highlights the limitations of current detection methods in generalization capabilities. to mitigate this challenge, we explore two complementary strategies targeting zero-shot and supervised detection methods, respectively. for zeroshot detectors, we propose pruning the scoring model to extract shared features that enhance detection generalization. this strategy is not limited to fast-detectgpt @xcite , which we adopt as a representative case, but is applicable to a broader class of zero-shot methods. our experiments demonstrate that this approach consistently improves performance across a range of detectors-including likelihood, rank, logrank, fast-detectgpt, and biscope-when using sheared-llama as the scoring model in detecting text generated by developments of llama-2. for supervised detectors, we propose a practical training strategy aimed at enhancing generalization: including training data from the initial version of an llm family can significantly improve performance on its subsequent updates. overall, evobench serves as a complementary benchmark that focuses on the evolving nature of llms, offering a more accurate framework for evaluating the real-world applicability of detection methods. while it does not claim to be a one-sizefits-all solution, we envision evobench evolving alongside detection methods and other benchmarks, helping guide future developments in this area. in particular, we suggest that extracting shared features from evolving llms could enhance the generalization of detection methods. with evobench, the community can more effectively evaluate and refine detection methods, contributing to more robust and practical detection systems that mitigate llm misuse in real-world applications. ## conclusion in this paper, we introduce evobench, a novel benchmark for evaluating the generalization of llm-generated text detection methods across evolving llms. evobench defines two key dimensions of llm evolution: (1) updates made by llm publishers over time and (2) developments carried out by developers, ensuring a comprehensive understanding of how evolving llms impact detection performance. to quantify this evolving dimension generalization, we propose the emg (evolving model generalization) metric. we evaluate 14 widely used detection methods using evobench, revealing their vulnerabilities when facing evolving llms. in response, to improve the generalization of zero-shot methods across developing llms, we propose two strategies: for zero-shot methods, we suggest pruning the scoring model to extract shared features across llm developments; for supervised methods, we recommend augmenting training data with data from llm updates. our benchmark represents a key step towards bringing detection methods into real-world scenarios for evolving llms. we also envision continuously updating this benchmark to cover a broader range of evolving llms, enabling more comprehensive evaluations across various domains and evolving llms.