IMGTB : A Framework for Machine-Generated Text Detection Benchmarking (2024)

Abstract: In the era of large language models generating high quality texts, it is a necessity to develop methods for detection of machine-generated text to avoid their harmful use or simply for annotation purposes. It is, however, also important to properly evaluate and compare such developed methods. Recently, a few benchmarks have been proposed for this purpose; however, integration of newest detection methods is rather challenging, since new methods appear each month and provide slightly different evaluation pipelines.In this paper, we present the IMGTB framework, which simplifies the benchmarking of machine-generated text detection methods by easy integration of custom (new) methods and evaluation datasets. In comparison to existing frameworks, it enables to objectively compare statistical metric-based zero-shot detectors with classification-based detectors and with differently fine-tuned detectors. Its configurability and flexibility makes research and development of new detection methods easier, especially their comparison to the existing state-of-the-art detectors. The default set of analyses, metrics and visualizations offered by the tool follows the established practices of machine-generated text detection benchmarking found in state-of-the-art literature.

## introduction due to indistinguishability between human-written texts and high-quality texts generated by modern large language models (llms) @xcite , the machine-generated text detection (mgtd) belongs to the key challenges identified by @xcite . mgtd methods are needed in many areas, such as prevention of disinformation spreading, plagiarism, impersonation and identity theft, automated scams and frauds, or even prevention of unintentional inclusion of lesser quality generated texts in future models' training data @xcite @xcite @xcite . regardless of the area of application, we are witnessing a race of new mgtd methods competing with new generation methods and appearing monthly. this presents a challenge to efficiently evaluate and benchmark the new methods. the problem is twofold, missing the uniform implementation of the methods and standardized evaluation. even when source codes for experiment replication are released, they are usually too specific and not flexible for reuse. moreover, across application areas, domains, text lengths, or topics, the performance of different mgtd methods varies. therefore, a flexible way of comparison over various datasets (even custom ones) is currently missing. these problems are usually addressed by common benchmarking frameworks. there is a lack of flexibility, configurability, and extensibility in the current mgtd benchmarking frameworks; therefore, we have focused on refining the most recent one, mgtbench @xcite , by integrating missing features and extending support to new types of detection pipelines. the key contributions of the proposed extended framework imgtb foot_0 are as follows: ## related works due to increasing quality of texts generated by modern llms, the research around detection of machine-generated text increased its importance. however, a common way to properly compare several detection methods was missing, mainly due to missing publicly available datasets. few years ago, mgtd researchers mostly used data generated by a single llm, such as gpt-2 3 or grover @xcite , results on which could not be properly generalized. later on, larger-scale multi llm benchmarks for mgtd task have been proposed, such as turingbench @xcite , deepfaketextdetect @xcite , m4 @xcite , or multitude @xcite . as a result, mgtd methods can now be evaluated on such benchmark datasets and compared to each other. however, these datasets do not share common class labels, structure, or form, what makes the evaluation on multiple of them complicated and unnecessarily prolongs the research. the other issue significantly prolonging the research is a missing unified implementation of existing mgtd methods. it leaves on the researchers a burden to either reuse the published source codes of individual methods (if there is some), which are different among each other and require customization, or implement them completely into their evaluation framework to be evaluated in a unified way with their newly proposed mgtd method. some of the proposed mgtd methods, such as detectgpt @xcite , released the full source code including implementation of other existing sota methods, enabling complete replication of experiments and providing a good basis to build upon. the result is a faster advancement by extension of the original method, in the form of detectllm @xcite or fast-detectgpt @xcite , proving the benefits of full replication possibilities. however, these methods focused on zero-shot statistical-based detection of machine-generated text, comparing various statistical metrics to distinguish between human-written and machinegenerated samples, not providing the classification prediction. thus, the implementations do not allow easy comparison with supervised high-performing pretrained llms finetuned for mgtd task, such as the popular openai detector @xcite . the proposed mgtbench framework @xcite attempted to solve the problem, by implementation of these methods in a common framework. using a dedicated classifier trained individually for metric-based statistical mgtd methods, it provides a class prediction, enabling a direct comparison to llms-based mgtd classifiers. mgt-bench has already accelerated mgtd research, such as @xcite or @xcite . however, mgtbench provides a quite complicated way to use custom datasets or to integrate new mgtd methods. moreover, the used classifierbased (must be trained) evaluation of metric-based statistical detectors does not enable their true zeroshot evaluation (without training, as reported in the corresponding papers). therefore, a more flexible and configurable benchmarking framework is needed. ## imgtb -integrated mgtd benchmark framework in this section, we introduce the central design principles, imgtb was built with, as well as its architecture and the functionality of the main components. we use a term experiment to denote a single run of the specified detection method on data from the specified dataset. ## case study to better illustrate the use of the framework in practice, in this section we showcase a few example use case scenarios. we look at: a. how to quickly run and evaluate simple experiments using cli b. how to run complex experiments using yaml configuration files for a more visual version of this demonstration, see the video foot_3 . for more detailed and runnable version, see the jupyter notebook foot_4 . ## discussion the usefulness of the imgtb framework has been evaluated in practice by its usage in @xcite @xcite , where it proved to be valuable especially for its implementation of the statistical detectors. in comparison to the sota mgtd framework called mgtbench @xcite , imgtb enables an objective comparison of statistical zeroshot detectors (i.e., without classifier training) by using roc curve. further, imgtb integrates the newest detectors, such as mfd, binoculars, or s5. it directly enables multiple fine-tuning processes for language models. most of all, in mgtbench @xcite , configurations, datasets and detection methods are often hard coded and cannot be easily changed or reconfigured, imgtb simplifies usage of custom datasets and detectors by supporting plug-in like extension. faster evaluation is provided by the implemented results analysis and automated comparison charts generation. to fine-tune a model for binary classification, a more generic sota framework, such as ludwig @xcite , could be used. however, such a framework would not be usable to compare the fine-tuned detectors to statistical detection methods or online services. therefore, a specialized mgtd framework, such as mgtbench or imgtb is needed for such a comparison. imgtb enables such a fine-tuning process by itself and also includes a unique multilingual mgtd specific version of per-language fine-tuning, not available in other frameworks. similarly, the evaluate framework foot_5 could be directly used to compare pretrained classification models. however, to compare also to statistical and other custom detection methods, a significant effort would be required to implement the custom pipelines, loosing flexibility, configurability (especially concerning the detectors training), and tailor-made analysis and visualization tools in comparison to the proposed imgtb. ## conclusion the machine-generated text detection belongs to the key challenges connected with the advancements of large language models for prevention of misuse of high-quality text generation capability. the proposed imgtb framework unifies the evaluation of the existing detection methods and simplifies comparison of new detection methods to the state-of-the-art. with a plug-and-play testing ability of new methods, research hypotheses can be easily examined. the framework can also be used for evaluation of state-of-the-art detection methods on custom data to identify the best performing one to be further used for some specific application. automated results analysis and methods comparison also enables less proficient users to interpret the results and make a selection. the framework reduces unnecessarily redundant work of researchers and enables them to focus their effort towards development of more effective detection methods. this can eventually accelerate the research in machine-generated text detection to catch up with the text generation, currently in the lead.