Where Am I From? Identifying Origin of LLM -generated Content (2024)

Abstract: Generative models, particularly large language models (LLMs), have achieved remarkable success in producing natural and high-quality content. However, their widespread adoption raises concerns regarding copyright infringement, privacy violations, and security risks associated with AI-generated content. To address these concerns, we propose a novel digital forensics framework for LLMs, enabling the tracing of AI-generated content back to its source. This framework embeds a secret watermark directly into the generated output, eliminating the need for model retraining. To enhance traceability, especially for short outputs, we introduce a “depth watermark” that strengthens the link between content and generator. Our approach ensures accurate tracing while maintaining the quality of the generated content. Extensive experiments across various settings and datasets validate the effectiveness and robustness of our proposed framework.

## introduction generative models, including gpt (openai, 2023a), gemini @xcite , and llama @xcite , have shown exceptional ability to produce natural and high-quality outputs. however, they also pose potential risks for malicious applications, such as the generation of fake news and unfounded rumors. consequently, the imperative arises to construct a robust security auditing system capable of tracing the origin of outputs. this tracing ability should not only ensure that model utilization aligns with regulatory requirements, compliance, and ethical standards but also elevates the overall integrity of the system. furthermore, considering generative models' proficiency in content creation and manipulation, safeguarding users' copyrights and intellectual property becomes paramount. by discerning the source of each user's generated output, we can protect legitimate contributions, preventing unauthorized replication or use of their content. in this paper, our objective is to pioneer digital forensics methods tailored for deep learning systems to audit ai-generated content. our innovative approach involves embedding a secret watermark into the generated output, ensuring source traceability. importantly, we strive to make watermarking implicit, balancing the challenge of being hard to remove and undetectable without compromising the quality of the generated output. during inference and without necessitating model fine-tuning or retraining, our method dynamically adjusts the probabilities of specific tokens based on a unique code associated with the user, effectively imprinting a watermark into the text. recognizing the need for enhanced performance in scenarios with short outputs, we introduce a depth watermark as an additional mechanism to embed the entire binary user code. this involves dividing the vocabulary into sub-lists and utilizing the added watermark to adjust the probability of the output token falling into these groups. this approach ensures accurate identification of the content generator from the user pool, relying on unique probability shifts. comprehensive experiments were conducted across various settings and datasets to validate our method's efficacy. we assessed accuracy on five datasets for tracing the generator from a single output sentence, and even with only 25 tokens, our proposed framework achieved over 90% top-10 accuracy across all datasets. the quality of watermarked outputs was evaluated through perplexity measures, and the robustness of our method was confirmed against text alteration attacks, maintaining a 90% accuracy for identifying the top generator in outputs of 200 tokens, even when up to 30% of the text was changed. however, applying post-hoc watermarks to lms faces challenges due to the diverse and versatile nature of output generation. to address this, @xcite proposed restricting generative models to produce tokens only from an "allowed" set. unfortunately, this constraint often leads to lowquality output, especially in low-entropy cases. ## problem setting language model. let v denote the vocabulary set of the language model (lm), which contains all possible tokens that the lm can generate. the size of v is typically 50, 000 tokens or more @xcite . a language model (lm) can then be defined as a function, which accepts a sequence with arbitrary length of tokens s (-n ) , s (-n +1) , .., s (-1) , referred to as the prompt, and generates an output s (0) , s (1) , ..., s (m ) . for simplicity, output s (n) , ..., s (m) will be denoted as s (n:m) . since the output s (n:m) is generated recursively, let f : s (-n :t-1) → z t be the generation function at step t, where z t represents a score vector with logit values for each token in v . at the initial step, @xmath0, the lm accepts the prompt s (-n :-1) and computes a corresponding logit which is later processed by a softmax operator to obtain a probability distribution over the vocabulary, denoted as p 0 . the token at position t is sampled from this probability distribution, denoted as s (0) . in the subsequent step, at @xmath1, the function f takes s (-n :0) as input. during tth step, f generates z t , and p t , s (t) are computed sequentially. threat model. the problem of tracing the output generated by a large language model (llm) is formalized as follows. the language model server supports text generation tasks for n users. when a user employs the language model to generate an output sentence s, it becomes imperative for the server to identify the specific generator responsible for producing that output without external information. this traceability is essential for secure audits and intellectual property protection. to protect intellectual property and ensure maintenance, the model architecture and weight details are encrypted and concealed from customers. as a result, customers only have access to input and output sentences. conversely, the server possesses access to some user information such as a unique id that serves as a "fingerprint" for each client. ## a simple watermark inspired by @xcite , our methodology begins with a straightforward yet highly effective watermarking method, ensuring that the generated output retains user-specific information without compromising its quality. the process begins by assigning a unique id to each client, which is embedded within the generated output to establish a clear link between the client and their output. this id is then transformed into a unique binary user code denoted by @xmath2, 1} k , where k represents the length of this binary code. during language model (lm) generation, at each t-th step, our method initiates the process by generating a pseudo-random preferred list l t . to construct l t , we define i(s (t) ) as the fixed index for token s (t) within the vocabulary list. the context c t for step t is defined as c @xmath3. utilizing this c t to seed a random number generator that splits the entire vocabulary list into two equally sized, nonoverlapping lists. these lists, v 0 t and v 1 t , then serves as two candidates for the preferred list l t . notably, the selection of the previous two tokens as the seed, instead of just the preceding one, enhances the differentiation in the splitting process. following this, an @xmath4, 1} is introduced to select the preferred list l t from the two candidate lists, v 0 t and v 1 t . to infuse user id intuitively into the generation process, the indicator is determined by a specific bit of the generator's binary user code e i , calculated as follows: subsequently, the logit vector z t at step t, produced by the final layer of the language model, undergoes modification by adding a bias parameter δ to the logit corresponding to the token in l t . this adjusted logit vector is then fed into a softmax operator, resulting in a biased probability distribution denoted as pt . the calculation is shown in equation 1, where px t signifies the biased probability of sampling the x-th token during the t-th step. ## depth watermarking to improve tracing performance and address the previously mentioned limitation, especially for short outputs, we introduce depth watermarking as an additional mechanism to embed the entire binary user code within a single generation step. therefore, we design a more sophisticated token probability distribution, aiming to maximize the distribution difference by introducing varying biases to different sub-lists within the preferred list. as shown in algorithm 1, in each generation step, we construct the preferred list l t using the same algorithm as in the simple watermark. we utilize the hash value of the generator's binary user code e i as a seed in the random number generator to partition the preferred list l t into several equally sized sub-lists without overlap, denoted as l 0 t , l 1 t , ..., l n t , where algorithm 1 depth watermark generation input: prompt s (-n :-1) , binary code of generator e i , length of binary code k, language model f , bias intensity δ output: watermarked output 1: for @xmath5, 1, ... do 2: apply language model on prior tokens s (-n :t-1) to get a logit vector z t . compute the product of vocabulary index of s (t-1) and s (t-2) as c t . utilize c t to seed a random number generator r t , and use r t to partition vocabulary v into v 0 t and v 1 t , where calculate the result of c t with modulus k and use the result n to determine an @xmath6, 1}, as n-th bit of e i . if the identifier equals to 0, set the preferred list l t as v 0 t , otherwise v 1 t . utilize the hash of e i to seed a random generator to partition l t into sub lists l 0 t , ..l n t , where l 0 t ∪ ... ∪ l n @xmath7. for each @xmath8, add a scaled bias 0.5 j δ to its corresponding logit. 9: end for 10: apply softmax operator to the new logit and get biased probability distribution pt over the vocabulary. 11: end for identification of the depth watermark. by introducing exponentially decreasing biases to the logits corresponding to each sub-list, we enhance the probability of sampling tokens from these sublists to varying extents. subsequently, the 'depth score' is employed to measure the similarity between the actual and ideal probability distributions of tokens within each sub-list. although precisely calculating the ideal probability distribution for each individual token is not feasible, we can estimate the overall trend p known for these n categories in the probability distribution. this entails experimentally obtaining a probability distribution of n classes, where each probability indicates the chance of a token from that class being selected. to estimate p known , we randomly generate a fixed preferred list and its sub-lists, calculating the corresponding bias values for each sub-list. we then construct a metric dataset by sampling data from different domains. for each prompt in the metric dataset, we query the model to obtain clean logits. the bias is then added to the set of clean logits according to the sub-lists, and then processed with the softmax operator to generate the probability distribution. we calculate the probabilities of output tokens falling into each of the n sub-lists or outside the preferred list p known by sampling the output from the resulting probability distribution. having obtained p known , given a sentence s (0:m) generated by the lm and the set of all binary user codes e, we proceed to calculate the actual probability distribution p practical . we examine the frequency of each token in the sentence falling into each sub-list and simultaneously track the count of tokens that do not belong to the preferred list. specifically, we maintain the occurrence o 0 , .., o n-1 for each sub-lists and o np for the non-preferred list to record the frequency of tokens falling into each list. the resulting list [n 0 , .., o n-1 , o np ] is then normalized by dividing by the total number of checked tokens, yielding p practical . finally, for each user i, their depth score is defined as the negative cross-entropy of p practical and p known . the higher the score, the more likely it is that user i is the generator. we show the detailed depth watermark identification process in algorithm 2. ## robustness analysis the robustness of the proposed watermarking scheme, particularly in maintaining the stability of identification results, presents an essential inquiry. in this section, we further discuss the performance of our scheme under adversarial attacks. it is encouraging to note that, when the attack budget is limited to 0.3 or less, ensuring that at least 70% of the original content remains unaltered, our scheme's top-1 accuracy showcases exceptional resilience, maintaining an identification accuracy exceeding 90%. this underscores the robustness of our methodology in achieving high accuracy levels, even when confronted with adversarial modifications. ## conclusion in this study, we introduced a comprehensive traceability framework designed to identify the origin of specific sentences generated by language models. our novel watermarking technique, which uniquely embeds a user code within each user's output, enables precise source identification through a robust scoring mechanism. empirical evaluation of our framework yielded promising results. notably, for outputs exceeding 50 tokens, we achieved a precision of nearly 100% across a cohort of 1024 users, underscoring the effectiveness of our methodology. furthermore, rigorous experimentation confirmed the robustness of our framework, demonstrating its resilience in diverse scenarios and challenges.