P oisoned P arrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Models (2025)

Abstract: As the capabilities of large language models (LLMs) continue to expand, their usage has become increasingly prevalent. However, as reflected in numerous ongoing lawsuits regarding LLM-generated content, addressing copyright infringement remains a significant challenge. In this paper, we introduce PoisonedParrot: the first stealthy data poisoning attack that induces an LLM to generate copyrighted content even when the model has not been directly trained on the specific copyrighted material. PoisonedParrot integrates small fragments of copyrighted text into the poison samples using an off-the-shelf LLM. Despite its simplicity, evaluated in a wide range of experiments, PoisonedParrot is surprisingly effective at priming the model to generate copyrighted content with no discernible side effects. Moreover, we discover that existing defenses are largely ineffective against our attack. Finally, we make the first attempt at mitigating copyright-infringement poisoning attacks by proposing a defense: ParrotTrap. We encourage the community to explore this emerging threat model further.

## introduction large language models (llms) are typically pretrained on massive corpora of textual data collected from the web, such as the common crawl, wikipedia, or written media @xcite . due to the scale of the pre-training corpora, it is almost impossible to comprehensively vet such datasets and ensure safety and quality in every document an llm sees during training @xcite . this leads to critical safety issues, such as the generation of toxic (e.g., racist stereotypes) or harmful (e.g., assisting with bio-weapon development) responses to user prompts @xcite @xcite . additionally, this challenge of comprehensively vetting the datasets has led to poisoning attacks @xcite , where attackers upload malicious documents to the internet to inject adversarial behaviors, such as backdoors @xcite , into llms that train on these documents. addressing these challenges is an active area of research, and the field has witnessed an arms race between attacks and defenses. one of the root causes behind these issues is the tremendous ability of llms to memorize and later reproduce either verbatim or close copies of their training documents @xcite , which has earned them a negative reputation for being stochastic parrots @xcite . moreover, as state-of-the-art llms become larger and larger, nearing a trillion parameters @xcite , their ability to memorize their training data also grows @xcite . as a result, llms can still memorize a sequence included in just one training document, making defenses such as text de-duplication less effective @xcite . although there is a continuing debate about whether memorization is essential for generalization (and the remarkable capabilities of llms) @xcite , it is known to introduce several risks when llms are used in public settings, such as chatgpt. first, when the training data contains sensitive information, such as individuals' addresses or phone numbers, an adversary can strategically prompt an llm to extract this information @xcite . second, memorization brings forth legal risks when an llm generates (either entirely or partially) a copyrighted document seen during training @xcite . this creates jeopardy for both the consumers of llm outputs, e.g., software projects that might inadvertently incorporate copyrighted source code @xcite , and for llm providers who risk getting sued by copyright holders @xcite . as of september 2024, the legal framework (especially in the united states) first, the attacker generates poison samples using an llm by prompting it to produce text containing consecutive words from the copyrighted sample. these poison samples are then injected into the training dataset. as the victim trains their model on both the poisoned and clean data, the resulting llm becomes compromised and generates text that closely resembles the copyrighted target. in copyright violation cases, defendants, when found liable, are often compelled to compensate plaintiffs financially. this standard has incentivized copyright holders to scrutinize public llms to find evidence for violations @xcite . in response, llm providers are rumored to start employing dataset curation (to filter out copyrighted materials) @xcite , or specialized training techniques that hinder memorization (such as differential privacy @xcite or the goldfish loss @xcite ). research suggests that these solutions can prevent llms from generating memorized text, making it harder for copyright holders to pursue claims @xcite . this landscape of evolving incentives raises a concerning question: can an adversary use poisoning attacks to deliberately increase the chance of an llm violating the copyright of a particular document even though the model has not been explicitly trained on the document itself? copyright trolls, who opportunistically (and often maliciously) enforce their copyrights for financial gain @xcite , may resort to such a strategy as llms are known to be vulnerable to poisoning. although research has shown that llms can memorize and reproduce copyrighted material in their training data, we do not know whether such an attack can be performed inconspicuously and still survive an llm provider's efforts to filter out copyrighted training data. it is also unknown whether training techniques against memorization @xcite can effectively mitigate this risk. in our work, we answer this question in the affirmative by designing and evaluating poisoned-parrot: the first poisoning attack against llms designed to induce copyright violations. poisonedparrot strategically embeds small fragments (n-grams) of the copyrighted text into seemingly clean samples, allowing the llm to learn and later regurgitate the copyrighted content unwittingly. figure 1 presents an overview of poisonedparrot, in particular, how it uses an off-the-shelf llm to create a set of inconspicuous text samples that poison the victim model into generating copyrighted content. we systematically evaluate poisonedparrot and show that it is highly effective and comparable to injecting 30 exact copies of the copyrighted target sample (an easily detectable attack) into the training set. moreover, poisoned and clean models have similar generative utility, adding to the stealthiness of poisonedparrot. state-of-the-art defenses fail to mitigate our attack in practical scenarios, highlighting the critical nature of this vulnerability. finally, as a starting point for future defenses, we propose parrottrap, which shows promise in detecting the samples crafted by poisonedparrot. contributions: (i) we introduce poisonedparrot, the first poisoning attack specifically designed to elicit copyrighted content from llms. while defenses against targeted and inconspicuous poisoning have been explored in other domains @xcite or specifically against backdoor attacks @xcite , there remains a significant gap in robust poisoning defenses for llms. recent attempts include unlearning to mitigate the effects of toxic or harmful training data @xcite . however, to this day, there is no established, general-purpose defense against poisoning attacks on llms. memorization and copyright. many studies have found that llms memorize training data. @xcite proposed a metric to measure memorization. @xcite also quantified memorization across different model scales and observed memorization increases as the model grows. @xcite emphasized that llms, described as "stochastic parrots," risk ingesting vast amounts of training data and reflecting the inherent biases within it. research has shown that llms can generate exact copies of copyrighted text, raising concerns over intellectual property violations @xcite , or other legal issues @xcite . however, another line of research suggests memorization is crucial for generalization @xcite . preventing verbatim outputs alone does not mitigate privacy concerns, as llms can encode memorized content into novel formats, still effectively reproducing the underlying data @xcite . several studies also exploit memorization to extract sensitive data from llms @xcite . differential privacy (dp) (abadi et al., 2016) is a standard defense for traditional deep learning models, but it struggles to scale in the context of llm pretraining and often degrading performance to unacceptable levels @xcite . some approaches have sought to improve practicality by pretraining on sanitized, non-sensitive data before applying dp training @xcite . deduplication of training data has proven effective in mitigating memorization @xcite , but is impractical to web-scale training data of llms due to unpredictable near duplicates. recent work has explored alternative approaches, including specialized loss functions that randomly exclude a subset of tokens while training, preventing verbatim memorization @xcite . ## technical design of poisonedparrot design overview. we start with a target sample (a piece of text that is assumed to be copyrighted). in our threat model, the attacker wishes to (falsely) accuse the victim of training on this sample. we divide the sample into c-grams (groups of consecutive words). these c-grams are created by sliding a window of size c over the words in the target sample with a stride of 1, resulting in overlapping c-grams. finally, we use an off-the-shelf llm to generate new samples containing each c-gram (verbatim), forming our "poison" set that is injected into the victim's training set. we include an overview of our technique in figure 1 and its implementation details in algorithm 1. poison generation. we generate a poison sample for a given c-gram-(s i , s i+1 ..., s i+c-1 )-by algorithm 1: poisonedparrot require : a target copyrighted sample @xmath0... ⊕ sn (split into n words), context window size c for the c-grams, poison budget k, a poison generation algorithm g. output : a set of k poison samples containing small pieces of the copyrighted text s. poisons ← {} ▷ the set of poisons i ← 1 ▷ poison counter j ← 1 ▷ iterator for the sliding window while reset the sliding window when reaching the end of s end end return poisons prompting an llm with the instruction "generate one paragraph at least 32 words long containing the following text verbatim: ", followed by the c-gram, and "don't include any additional text other than the paragraph." to maintain consistency with the training samples in terms of word count, we randomly crop the obtained poison while ensuring the c-gram is included. if the generated paragraph does not contain the desired c-gram or is too short, we regenerate the poison until the conditions are met or a maximum number of generations is reached. ## existing defenses in this section, we investigate the effect of prior work's defenses on our attack. poison detection. we consider a poison filtering defense based on perplexity @xcite , which assumes that poisons may be detected from their higher perplexity values. we consider pythia-6.9b model @xcite to compute the perplexity of a text and measure the percentage of clean and poisoned training samples removed at different perplexity thresholds. we present the results in figure 5 . this shows perplexity cannot distinguish poison samples crafted by poisonedparrot from clean samples. removing most poisons based on perplexity would also remove most clean samples, hurting the model's utility. these results were obtained when we set the random seed to 0. in figure 15 , we present the results for three seeds and two models for computing perplexity, which align with the conclusions made here. anti-memorization training. we also consider goldfish loss @xcite , a state-of-the-art training-time defense against memorization. goldfish loss randomly drops tokens from the loss computation during training using a hash computed on the last h tokens of a sample. @xcite suggest using @xmath1, potentially hindering its ability to generate certain common phrases longer than h tokens. in figure 6 , we demonstrate that at @xmath2, our attack remains largely effective against this defense as the poisoned model still generates text significantly more similar to the target compared to the non-poisoned baseline. for a much less practical value of h, such as 3, the attack is still effective, though its success drops slightly. notably, compared to poisoned models, goldfish loss more effectively prevents a model from memorizing the target for the baseline models trained on multiple copies of the target. this holds as long as our attack's window size c is smaller than the defense's h. we present these results in figure 7 (for @xmath3, 40}, @xmath4, 7} and @xmath5. see figure 16 for more configurations. takeaways. poisonedparrot effectively induces the model to generate text similar to the target and also bypasses the existing poisoning and copyright protection defenses for llms from the prior work. the algorithm for parrottrap is as follows: step 1: split each training sample s into overlapping n-grams, using a stride of 1. step 2: find the largest value x such that at least x n-grams of s appear in at least x other samples, and consider this x as the heuristic value for the training sample s (similar to a publication's h-index). step 3: threshold these heuristic values to separate the clean (lower) and poisoned (higher) samples. in figure 8 , we consider @xmath6. for stronger attacks (@xmath7, parrottrap is more effective at separating the clean from poisoned samples, being able to remove over 85% of the poisons at the cost of less than 20% of the clean samples removed. for less effective, weaker attacks, however, parrottrap struggles to separate the clean and poison samples. we observe similar trends for different seeds and for @xmath8. ## conclusion and future work the concerns about llms being trained on and memorizing copyrighted content are growing. copyright holders have financial incentives to pursue violation claims against llm companies. within this context, our work proposes a new threat model in which an adversary launches a training set poisoning attack to increase the chance of an llm generating an output that violates the copyright of a particular text. we design poisonedparrot to show that this threat model is feasible and realistic. poisonedparrot crafts inconspicuous poison samples that have no impact on the model's utility, cannot be detected as copyrighted, and cannot be prevented by existing defenses. as a prototypical defense, we propose parrottrap, which shows promising results in detecting poison samples of poisonedparrot. our findings reveal a vulnerability with real-world consequences. we encourage future work to investigate this threat model further and to create practical, deployable defenses. future work. poisonedparrot includes verbatim n-grams from the copyrighted target text, which enables its detection by parrottrap. an adaptive attack that crafts poison samples that contain semantically equivalent but non-verbatim copies of these n-grams would bypass parrottrap. if successful, such an attack would necessitate developing advanced defenses to copyright poisoning attacks.