How Large Language Models are Transforming Machine-Paraphrase Plagiarism (2022)

Abstract: The recent success of large language models for text generation poses a severe threat to academic integrity, as plagiarists can generate realistic paraphrases indistinguishable from original work.However, the role of large autoregressive models in generating machine-paraphrased plagiarism and their detection is still incipient in the literature.This work explores T5 and GPT3 for machine-paraphrase generation on scientific articles from arXiv, student theses, and Wikipedia.We evaluate the detection performance of six automated solutions and one commercial plagiarism detection software and perform a human study with 105 participants regarding their detection performance and the quality of generated examples.Our results suggest that large language models can rewrite text humans have difficulty identifying as machine-paraphrased (53% mean acc.).Human experts rate the quality of paraphrases generated by GPT-3 as high as original texts (clarity 4.0/5, fluency 4.2/5, coherence 3.8/5).The best-performing detection model (GPT-3) achieves 66% F1-score in detecting paraphrases.We make our code, data, and findings publicly available to facilitate the development of detection solutions.

## introduction paraphrases are texts that convey the same meaning while using different words or sentence structures @xcite . paraphrasing plays an important role in related language understanding problems (e.g., question answering @xcite , summarization @xcite ), but it can also be misused for academic plagiarism. academic plagiarism is serious misconduct as its perpetrators can unjustly advance their careers, obtain research funding that could be better spent, and make science less reliable if their misbehavior remains undetected @xcite . paraphrasing tools can be used to generate convincing plagiarized texts with minimum effort. most of these tools (e.g., spinbot foot_1 , spinnerchief foot_2 ) use relatively rudimentary heuristics, such as word replacements with synonyms, and they already deceive plagiarism detection software @xcite . however, these tools scratch the surface of the possibilities compared to what large neural language models can achieve in producing convincing high-quality paraphrases @xcite . notably, large autoregressive language models with billions of parameters, such as gpt-3 @xcite , make paraphrase plagiarism effortless yet exceedingly difficult to spot. so far, large language models have found little ap-plication in plagiarism detection. as language models are already easily accessible for applications such as software development 4 or accounting foot_4 , using language models for machine-paraphrasing will become as easy as a click of a button soon. therefore, the number of machine-plagiarized texts will increase dramatically in the upcoming years. to counteract this problem, we need robust solutions before models are widely misused. in this study, we generate machine-paraphrased text with gpt-3 and t5 @xcite to compose a dataset for testing against automatically generated paraphrasing. we test different configurations of model size, training schemes, and selection criteria for generating paraphrases. to understand how humans perceive machine-paraphrased text, we also performed an extensive study with 105 participants assessing their detection performance and qualityof-text assessments against existing automated detection methods. we show that while humans can spot paraphrasing of online tools and smaller autoencoding models, large autoregressive models prove to be a more complex challenge as they can generate human-like text containing the same key ideas and messages from their original counterparts (see to summarize our contributions: ## methodology this study focuses on understanding how humans and machines perceive large autoregressive machine-generated paraphrase examples. there-fore, we first generate machine-paraphrased text with different model sizes of gpt-3 and t5. we then generate a dataset composed of 200,000 examples from arxiv (20,966), wikipedia (39,241), and student graduation theses (5,226) using the best configuration of both models. we investigate how humans and existing detection solutions perceive this newly automated form of plagiarism. in our human experiments, we compare paraphrased texts generated in this study to existing data that use paid online paraphrasing tools and autoencoding language models to paraphrase their texts. finally, we evaluate commercial plagiarism detection software, machine-learning classifiers, and neural language model-based approaches to the machine-paraphrase detection task. ## research questions & experiments q1: how does model size influence the quality of generated paraphrases? a. we ask this question to underline the problem's urgency as recently released models have a large number of parameters. figure 1 shows the influ-bertsc. bartsc. rouge-l bleu later in his career, gates has pursued many business and philanthropic endeavors. with the increasing number of parameters, both models' semantic similarity scores (bertscore, bartscore) also rise. t5 shows the highest increase when extending the model from 3 billion parameters to 11 billion. gpt-3 (175b) reaches its overall highest semantic similarity, generating sentences with similar meanings compared to the source. model's generated candidates also have higher count-based scores on average as they often repeat text from the source. as described before, we try to sample candidates with low word-count scores to avoid repetition of words. we conclude that scaling models' size positively influences their performance at the task of paraphrasing, which agrees with previous research @xcite . while the limits and details of scaling models are still unknown, boosting their computing power will allow for more human-like texts to be produced.