P lag B ench: Exploring the Duality of Large Language Models in Plagiarism Generation and Detection (2025)

Abstract: Recent studies have raised concerns about the potential threats large language models (LLMs) pose to academic integrity and copyright protection. Yet, their investigation is predominantly focused on literal copies of original texts. Also, how LLMs can facilitate the detection of LLM-generated plagiarism remains largely unexplored. To address these gaps, we introduce PlagBench, a dataset of 46.5K synthetic text pairs that represent three major types of plagiarism: verbatim copying, paraphrasing, and summarization. These samples are generated by three advanced LLMs. We rigorously validate the quality of PlagBench through a combination of fine-grained automatic evaluation and human annotation. We then utilize this dataset for two purposes: (1) to examine LLMs’ ability to transform original content into accurate paraphrases and summaries, and (2) to evaluate the plagiarism detection performance of five modern LLMs alongside three specialized plagiarism checkers. Our results show that GPT-3.5 Turbo can produce high-quality paraphrases and summaries without significantly increasing text complexity compared to GPT-4 Turbo. However, in terms of detection, GPT-4 outperforms other LLMs and commercial detection tools by 20%, highlights the evolving capabilities of LLMs not only in content generation but also in plagiarism detection. Data and source code are available at https://github.com/Brit7777/plagbench.

## introduction plagiarism occurs when someone uses another person's work, ideas, or expressions without proper acknowledgment. it is one of the most frequently scrutinized issues in writing tasks, which leads to a violation of intellectual property rights and academic integrity. traditionally, plagiarism research has focused on human-written text. however, the rise of large language models (llms) has transformed the landscape of plagiarism (pudasaini et al., 2024). for instance, users can exploit these models to obfuscate original content through paraphrases or summaries, effectively bypassing traditional plagiarism detection systems @xcite @xcite . moreover, many recent studies @xcite @xcite have shown that llms can memorize portions of their training data and reproduce them during text generation. these models are not limited to producing verbatim copies; they can also generate paraphrased or elongated versions of unique content @xcite . the increasing prevalence of plagiarism in llms raises two critical research questions: (rq1) how well can llms generate paraphrase and summary plagiarism of given texts? and (rq2) how well can llms detect three types of plagiarism-i.e., verbatim, paraphrase, and summary plagiarisms?. while prior research has explored llms' applications to automated paraphrase @xcite and summary @xcite @xcite generation, these studies are often limited in scope because most of them focus on individual domains or specific models. a holistic and fine-grained analysis of diverse domains and models is crucial to developing robust mechanisms for detecting and mitigating plagiarism in ai-generated content. additionally, llms have demonstrated impressive performance in various natural language processing (nlp) tasks @xcite , motivating us to investigate their potential in automatic plagiarism detection. current plagiarism detectors are primarily optimized to catch literal copies or superficial paraphrasing by humans and may not be sufficient to detect nuanced rewritings of llms. evaluating llms' effectiveness in plagiarism detection can offer critical insights into the limitations of current systems and help guide the development of more advanced, robust solutions. to answer these questions, we present plag-bench, a large-scale corpus containing 46.5k text pairs with three plagiarism categories: verbatim, paraphrase, and summary plagiarism. these samples cover three writing domains (i.e., abstract of scholarly paper, story, news article) and are synthetically generated by three advanced, closed-source and open-source llms (llama2-70b-chat @xcite , gpt-3.5 turbo foot_0 , and gpt-4 turbo @xcite ). to acquire the gold standard for automatic plagiarism detection tasks, we employ a comprehensive integration of both automatic and manual evaluation. this final subset is then used for evaluation of five popular llms (llama2-70b-chat, llama3-70b-instruct foot_1 , mixtral-8x7b (jiang et al., 2024), gpt-3.5 turbo, and gpt-4 turbo) and three specialized plagiarism detectors (gptzero foot_2 , prepostseo foot_3 , and a detector proposed by @xcite ), two of which are commercial-ready. figure 1 2 related work ## conclusion we present a novel large-scale plagiarism detection benchmark, plagbench: a collection of 46.5k artificial plagiarism cases generated by three cuttingedge llms. this underscores the nuanced challenges and opportunities presented by llms in combating plagiarism while also acknowledging their potential vulnerabilities to being used for unethical practices. we envision plagbench as a universal standard for evaluating plagiarism detection methods.