A Ship of Theseus: Curious Cases of Paraphrasing in LLM -Generated Texts (2024)

Abstract: In the realm of text manipulation and linguistic transformation, the question of authorship has been a subject of fascination and philosophical inquiry. Much like the Ship of Theseus paradox, which ponders whether a ship remains the same when each of its original planks is replaced, our research delves into an intriguing question: Does a text retain its original authorship when it undergoes numerous paraphrasing iterations? Specifically, since Large Language Models (LLMs) have demonstrated remarkable proficiency in both the generation of original content and the modification of human-authored texts, a pivotal question emerges concerning the determination of authorship in instances where LLMs or similar paraphrasing tools are employed to rephrase the text–i.e., whether authorship should be attributed to the original human author or the AI-powered tool. Therefore, we embark on a philosophical voyage through the seas of language and authorship to unravel this intricate puzzle. Using a computational approach, we discover that the diminishing performance in text classification models, with each successive paraphrasing iteration, is closely associated with the extent of deviation from the original author’s style, thus provoking a reconsideration of the current notion of authorship.

## introduction the ship of theseus paradox is a philosophical thought experiment @xcite that questions the concept of originality and change over time. the paradox begins with the premise that a ship, called the ship of theseus, gradually has all its planks replaced over time with new, identical planks. the paradox then poses the question: is the fully modified ship, with none of its original parts remaining, still the ship of theseus, or is it an entirely different ship? just like the ship of theseus, our study involves the successive transformation of text through paraphrasing as illustrated tilt as a gaming term is associated... ## related work our study extends prior research on authorship attribution tasks from various perspectives, including style and content. notably, @xcite found that content-based features are more effective for datasets with high topical variance, while datasets with lower variance benefit more from style-based features for these tasks. several assessments and benchmarks on stylistic analysis have aimed to identify and infer style across different domains. the xslue benchmark @xcite comprehensively evaluates sentence-level crossstyle language understanding in 15 different styles. additionally, the stel framework @xcite introduces four specific assessments measuring the stylistic content of authorship representations: formality, simplicity, contraction usage, and number substitution preference. recent research has also explored the learning of authorship representations @xcite in diverse cross-domain settings. for instance, @xcite introduced the concept of universal authorship representations with a recent extension @xcite to validate their capacity to capture stylistic features. however, it is essential to note that these studies primarily focus on performing classification tasks related to authorship in various setups. our task distinguishes itself by delving into the established concept of ground truth concerning authorship in paraphrasing in the era of llms. another body of related research revolves around paraphrasing detection and plagiarism. these studies aim to determine whether a pair of texts constitutes a paraphrased version of one another @xcite . paraphrasing detection stands as a critical challenge within the domain of plagiarism identification @xcite . it is also a subject of inquiry in evaluating a proposed model's capacity for addressing natural language understanding tasks @xcite . previous research encompasses both human-generated @xcite and machine-generated @xcite paraphrased versions. a recent investigation by @xcite suggests that machine-generated paraphrases bear greater similarity to the original source text than human-generated paraphrases. this phenomenon resurges the discussion: if an llm, such as chatgpt, is employed to paraphrase a text, should chatgpt be regarded as the author? therefore, our study seeks to explore the connection between authorship and paraphrasing by bridging the gap among these distinct lines of research. ## methodology the classical authorship attribution problem aims to determine the author (a) of a given text t 0 from a set of candidate authors, typically treated as a multi-class classification task. however, when text t 0 is paraphrased into t 1 by an llm (b), who is also a potential author in the candidate set, it raises a question of what should be considered the ground truth. the traditional perspective designates the original author a as the author of t 1 , while an alternative perspective assigns llm b as the author. text t 1 may substantially diverge from t 0 in style and content, potentially more similar to text g, independently generated by llm b on the same subject. a similar scenario is also applicable to human vs. ai text detection problems. therefore, our methodology (figure 3 ) focuses on assessing the classifier's performance, considering both perspectives and exploring how variations in style and content account for the observed differences. ## experimental results we primarily focus on evaluating the impact of paraphrasing in the context of authorship attribution, which translates into a seven-class classification problem. our objective is not to devise new text classification methods but to investigate how ground truth influences their performance and its correlation with changes in style and content. we employ established text classification methods, including finetuned bert (bert-base-cased) as a representative of the finetuned language model (lm), our style model with xgboost @xcite classifier as a representation of stylometry, gpt-who @xcite for information density-based multi-class classification, and tf-idf with logistic regression for classic n-gram-based analysis. we also explore the human vs. ai text detection scenario, a binary classification problem, using different finetuned and zero shot approaches. authorship attribution results: style and content similarity drop: for each paraphrase, we examine how the paraphrased text deviates from the original text across two crucial dimensions: content and style. figure 5 illustrates the reduction in style and content similarity be-tween the original text and its paraphrased versions, with the performance drop. we note that style deviates more substantially than content, with a substantial drop after the first paraphrasing and marginal changes in subsequent iterations, resembling the f1 score trend for all paraphrasers. a pearson correlation test confirmed the statistically significant relation (@xmath1.05) between the decline in f1 scores and a drop in style similarity. figure 6 shows the individual breakdown of style drop for different authors and datasets. figure 7 illustrates that for specific authors, following successive paraphrasing, a substantial portion of llm paraphrased text can become more similar to the style of the llm. ai text detection results: using four zero-shot ai text detectors @xcite @xcite and fine-tuned bert, we assess current ai text detectors in three scenarios. as expected, fine-tuned bert surpasses all zero-shot approaches. longformer exhibits the best performance among the zero-shot methods, likely due to their fine-tuning dataset's overlap with ours. the detection of ai-generated text is easiest in the original setting as compared to both scenarios involving paraphrased texts as can be seen ( ## conclusion in light of the increasing mainstream popularity of llms, ai text generation and detection have witnessed a surge in research activities. among the essential components in these domains, paraphrasing holds a significant role. this study explores the diverse notions of authorship regarding paraphrasing, inspired by the philosophical ship of theseus paradox. our findings suggest that authorship should be task-dependent, and we substantiate our empirical results with theoretical and philosophical perspectives. given the increasing prevalence of llms in generating and enhancing text, our research can provide a sound basis for addressing plagiarism and copyright disputes in the future involving llms.