FOCUS : Forging Originality through Contrastive Use in Self-Plagiarism for Language Models (2024)

Abstract: Pre-trained Language Models (PLMs) have shown impressive results in various Natural Language Generation (NLG) tasks, such as powering chatbots and generating stories. However, an ethical concern arises due to their potential to produce verbatim copies of paragraphs from their training data. This is problematic as PLMs are trained on corpora constructed by human authors. As such, there is a pressing need for research to promote the generation of original content by these models. In this study, we introduce a unique “self-plagiarism” contrastive decoding strategy, aimed at boosting the originality of text produced by PLMs. Our method entails modifying prompts in LLMs to develop an amateur model and a professional model. Specifically, the amateur model is urged to plagiarize using three plagiarism templates we have designed, while the professional model maintains its standard language model status. This strategy employs prompts to stimulate the model’s capacity to identify non-original candidate token combinations and subsequently impose penalties. The application of this strategy is integrated prior to the model’s final layer, ensuring smooth integration with most existing PLMs (T5, GPT, LLaMA) without necessitating further adjustments. Implementing our strategy, we noted a significant decline in non-original sequences comprised of more than three words in the academic AASC dataset and the story-based ROCStories dataset. Source code and scripts will be released after the paper’s acceptance and publication.

## introduction pre-trained language models (plms) have gained widespread recognition for their unparalleled performance in numerous downstream natural language processing (nlp) tasks @xcite @xcite @xcite , especially in text generation (hua and wang, 2020; zhang et al., * equal contribution † corresponding author 2020; @xcite . with the emergence of advanced plms, there is an intensifying debate over the distinctiveness of texts produced by these models as opposed to those written by humans, a sentiment highlighted by @xcite . within this landscape, generative plms primarily fall into two architectural paradigms. on the one hand, bart @xcite , t5 @xcite , and other such pretrained language models @xcite serve as quintessential representatives of the encoderdecoder approach. on the other hand, openai's gpt (generative pre-trained transformer) series @xcite @xcite and meta ai's llama series pretrained language models @xcite , leveraging a decoder-only design, has carved out a unique niche for itself. both these architectural categories of pre-trained language models have exhibited impressive capabilities in natural language understanding @xcite , and natural language generation @xcite , solidifying their reputation as premier commercial offerings. because of their efficiency and adaptability @xcite , these models have been widely adopted across diverse sectors, including writing, and academic research. nonetheless, the sophistication these models showcase in text generation has ignited concerns regarding academic integrity, prompting many educational establishments to restrict their utilization in scholarly activities. we believe that the aforementioned issues mainly occur because the content generated by language models often lacks originality, as their outputs heavily rely on their training data. this might lead the models to replicate or mimic the information or patterns they encountered during training. in areas where originality is highly valued, such as academic writing or story generation, the outputs from these models can sometimes be seen as pla-gpt2 output: they are of the following form: @xcite the distribution over all possible state sequences. the parameter vector @xcite is trained input: discriminatively to maximize the conditional log likelihood of the training data @xcite llama2 output: "events" are extracted along with their event participants, e.g., "who did what to whom when and where? " input: events can have different properties such as the type of event, the time and place of the event and the participants. in order to represent the properties of the event, the event… figure 1 : samples of academic writing generated by the fine-tuned llama2 and gpt2 plms using the aasc dataset. upon plagiarism analysis with turnitin, both models' outputs showed significant overlaps (in red), implying a conspicuous absence of originality. @xcite indicates a masked math formula. giaristic. in fact, earlier research has already highlighted the potential risks of intentional or unintentional leakage of sensitive information within the training sets of language models @xcite @xcite . this concern persists even during the fine-tuning phases, as evidenced by @xcite . to further investigate the potential lack of originality in plms outputs, we conduct fine-tuning on two prominent generative plms, llama2 and gpt2 pre-trained language models, using an academic paper aasc 1 dataset from the nlp domain. testing their outputs with turnitin 2 revealed significant instances where the models reproduced segments from the training set, as illustrated in figure 1 . currently, a considerable portion of research is dedicated to determining whether the outputs from plms display plagiarism or retain their originality @xcite @xcite . however, there is a noticeable lack of research focused directly on the innate originality of content generated by these plms. to address this void, we introduce a novel generation approach named the "self-plagiarism" (sp) contrastive decoding strategy, aimed at bolstering the inherent originality of text generated by plms. this strategy builds upon the principles delineated by @xcite and @xcite , yet diverges from their methodologies. initially, our approach shifts its focus from mitigating model bias at the token level to accentuating origi-1 https://github.com/kmcs-nii/aasc 2 https://www.turnitin.com/ nality at the paragraph level. a distinctive feature of our method lies in the strategic emphasis on the topmost layer, enriched with high-level knowledge. coupled with the extension of generation length, our methodology encompasses a broader spectrum of content uniqueness. this proves particularly advantageous in domains like storytelling and academic writing, where the nuances of high-level knowledge hold significant importance. secondly, we achieve an amateur model and a professional model by adjusting the prompts of the language models. for the amateur model, we innovatively introduce three prompts, which originate from the three most common research categories in plagiarized literature: verbatim plagiarism, paraphrase plagiarism, and idea plagiarism. the purpose of these prompts is to guide the model to replicate the training data according to specific plagiarism standards. on the other hand, the professional model uses conventional prompts to encourage the model to generate normal text. on this basis, we subtract the probability distribution of the last layer of the two models and impose penalties on tokens that show a higher probability in a regulatory factor function, ensuring a balance between following the prompts and maintaining originality. finally, we use our method on various language models, including t5, gpt-2, llama1, and llama2, on the academic dataset aasc and rocstories dataset. the results show a significant reduction in non-original sequences of more than three words generated by plms. our primary contributions are as follows: the following text contains exact copies of words or phrases without transformation language: the following text contains synonymous substitution, word reordering, and back translation language: the following text contains reuse of the core idea by shortening or summarizing the original content language: