Distinguishing Fact from Fiction: A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era. (2023)

Abstract: As generative NLP can now produce content nearly indistinguishable from human writing, it becomes difficult to identify genuine research contributions in academic writing and scientific publications. Moreover, information in NLP-generated text can potentially be factually wrong or even entirely fabricated. This study introduces a novel benchmark dataset, containing human-written and machine-generated scientific papers from SCIgen, GPT-2, GPT-3, ChatGPT, and Galactica. After describing the generation and extraction pipelines, we also experiment with four distinct classifiers as a baseline for detecting the authorship of scientific text. A strong focus is put on generalization capabilities and explainability to highlight the strengths and weaknesses of detectors. We believe our work serves as an important step towards creating more robust methods for distinguishing between human-written and machine-generated scientific papers, ultimately ensuring the integrity of scientific literature.

## introduction generative natural language processing (nlp) systems-often based on large language models (llms) @xcite @xcite -have experienced significant advancements in recent years, with state-of-the-art algorithms generating content that is almost indistinguishable from human-written text @xcite @xcite . this progress has led to numerous applications in various fields, such as chatbots @xcite , automated content generation @xcite , and even summarization tools figure 1 : this work's overview. five methods are used to machine-generate papers, which are then mixed with human-written ones to create our benchmark. four models are then tested as baselines to identify the authorship of a given output. @xcite . however, these advancements also raise concerns regarding the integrity and authenticity of academic writing and scientific publications @xcite . it is indeed increasingly difficult to differentiate genuine research contributions from artificially generated content. moreover, we are at an increased risk of including factually incorrect or entirely fabricated information @xcite . reliably identifying machine-generated sci-entific publications becomes thus crucial to maintain the credibility of scientific literature and fostering trust among researchers. this work introduces a novel benchmark to address this issue. our contribution-also briefly sketched in 1-can be summarized as follow: we release our benchmark dataset, baseline models, and testing code to the public to promote further research and aid the development of more robust detection methods. we release our benchmark dataset and baseline models as well as all code used for experimental results 1 . ## benchmark dataset in this section, we delve into the construction of our benchmark dataset, which comprises both humanwritten and machine-generated scientific papers. often, for simplicity, we refer to the former group with real, and to the latter with fake. in section 3.1, we elaborate on the process we followed to extract data from the pdf documents of real papers. in section 3.2, we describe instead our prompting pipelines and how we utilized various generators to produce fake scientific papers. ## detection experiments in this section, we conduct experiments about identifying the source of a given paper-i.e. determining whether it is fake or real. we start by defining data splits and subsets for training and testing, which are useful to evaluate generalization capabilities. next, we outline the classifiers used as baselines to measure performance on the benchmark task. finally, we examine the results in terms of performance and apply post-hoc explainability methods to the classifiers to gain deeper insights into the detection process. ## limitations and future work despite memory and gpu limitations presenting significant obstacles for our project, we were still able to create high-quality fake scientific papers. nonetheless, we believe there is room for improvement in addressing such limitations. due to the complexity of parsing pdfs, we are currently limited to specific sections (abstract, introduction, conclusion) instead of complete papers. moreover, processing entire publications would require substantial computational efforts. we believe that selecting sections dynamically at random instead of a fixed choice is worth exploring and will be the focus of future work. beyond detectgpt @xcite , other zero-shot text detectors such as gptzero foot_3 present promising solutions worth testing on our benchmark dataset. however, at the time of writing, such solutions are not available for experiments at scale. in future work, we aim to address these limitations by exploring dynamic section selection, improving papers' quality, adding human-llms co-created samples, and investigating the potential of zero-shot text detectors like gptzero as they become more accessible and scalable. ## discussion, ethical considerations, and broader impact it is important to emphasize that our work does not condemn the usage of llms. the legitimacy of their usage should be addressed by regulatory frameworks and guidelines. still, we strongly believe it is crucial to develop countermeasures and strategies to detect machine-generated papers to ensure accountability and reliability in published research. our benchmark dataset serves as a valuable resource for evaluating detection algorithms, contributing to the integrity of the scientific community. however, potential challenges include adversarial attacks and dataset biases @xcite . it is essential to develop robust countermeasures and strive for a diverse, representative dataset. ## conclusion this work introduced a benchmark dataset for identifying machine-generated scientific papers in the llm era. our work creates a resource that allows researchers to evaluate the effectiveness of detection methods and thus support the trust and integrity in the scientific process. we generated a diverse set of papers using both scigen and state-of-the-art llms-chatgpt, galactica, gpt-2, and gpt-3. this ensures a variety of sources and includes models capable of generating convincing content. we fine-tune and test several baseline detection models-gpt-3, galactica, and roberta-and compare their performance to detectgpt. the results demonstrated varying degrees of success, with some models showing remarkable performance on specific subsets while sometimes struggling with out-ofdomain data. by providing a comprehensive platform for evaluating detection techniques, we contribute to the development of robust and reliable methods for identifying machine-generated content. moving forward, we plan to address the current limitations and further enhance the utility of our benchmark for the research community. we release a repository containing our benchmark dataset as well as the code used for experimental results foot_4 .