GPT s Don’t Keep Secrets: Searching for Backdoor Watermark Triggers in Autoregressive Language Models (2023)

Abstract: This work analyzes backdoor watermarks in an autoregressive transformer fine-tuned to perform a generative sequence-to-sequence task, specifically summarization. We propose and demonstrate an attack to identify trigger words or phrases by analyzing open ended generations from autoregressive models that have backdoor watermarks inserted. It is shown in our work that triggers based on random common words are easier to identify than those based on single, rare tokens. The attack proposed is easy to implement and only requires access to the model weights. Code used to create the backdoor watermarked models and analyze their outputs is shared at [github link to be inserted for camera ready version].

## introduction language models are seeing increasing use across a wide variety of applications and the rate of language model releases appears to continue to increase as well. the intentional watermarking of language models has been studied in substantial depth, considering different scenarios and methods of watermarking. watermarking could be used to show that text was machine generated or to prove ownership of a model; this is generally done in a way that is subtle and not apparent to a human observer @xcite @xcite . in this work, we focus on the specific scenario of demonstrating ownership of a model that has been taken by a potential adversary. for this reason, we focus on the backdoor watermark, which is a watermark that is only engaged when some trigger is activated. this is sometimes called the black box setting @xcite . the hypothetical situation for such a backdoor watermark is to determine if one's model is being used in a way that is outside of the specified license or is perhaps stolen. similar work has explored the use of poisoned datas ets to demonstrate use of the data set for unauthorized model training @xcite . one not-sohypothetical scenario for this is the proliferation of bot accounts on twitter that have been created to promote various agendas @xcite . more sophisticated bots, using language models to respond to tweets in a human-like manner, have been observed @xcite . having a backdoor watermark could allow the language model creator to identify whether their model was in use in such a bot and provide evidence to report it to twitter. autoregressive language models are currently some of the most widely used language models across a variety of tasks @xcite , which provides a strong motivation to study their potential use of backdoor watermarks. we find that autoregressive language models that are trained to have a backdoor watermark will regurgitate their trigger word or phrase at a rate that is higher than would be found in common usage. we show that a potential adversary with unfiltered access to the inputs and outputs of a model could likely find the trigger word or phrase by performing frequency analysis on open-ended generation, even when the model has a very low rate of accidental watermarking. the rest of this paper is organized as follows. section 2 covers some related work into language model watermarks, with a focus on backdoor watermarks. methodology used to adapt the model to a specific sequence-to-sequence task and incorporate a backdoor watermark is presented in section 3. the results are presented in section 4 and further discussed in section 5 along with recommendations for use of this work. we summarize our work in section 6. we discuss the limitations of this work in the limitations section and share some of the ethical concerns raised by this work in the ethical statement. ## background the idea of adding watermarks to a deep learning model has been explored for quite some time. the discrete space nature of a language model means that different solutions for watermarks have to be utilized if they are to be discreet. watermarks may be either continuously generated @xcite , or generated in response to a given input trigger @xcite . @xcite and @xcite successfully trained classification models to respond to triggers, but less work has been done with natural language generation (nlg) models. @xcite is one of the few works found that focuses on nlg tasks, specifically creating robust watermarks based on semantic combination patterns that are not easily recognizable and do not alter intent of the generated content. he et al. (2020) also focuses on nlg, with a specific application of demonstrating ownership for llms deployed through apis by using lexical replacement of specific words. it is pointed out by @xcite that detection of rare trigger words could be done by an adversary observing input streams (which makes the case for using phrases or sentences involving common words), although @xcite makes the uncited claim that trigger words could be found by inspecting the embedding space. this paper attempts to determine whether that claim of easily identifying trigger words applies to autoregressive models. ## method to evaluate the backdoor response of our model, we use the metric of attack success rate (asr) used by @xcite al. (2020); yang et al. (2021a) as well as the additional metric of false trigger rate (ftr) proposed by @xcite . for this work, we use the term backdoor watermark success rate in place of asr to better reflect the goals of our work. the pre-trained 1.3 billion parameter variant of the gpt-neo model @xcite is used as the base model for all investigations, with the 2.7 billion parameter variant used to verify some results. these models were chose because they provide an autoregressive language model that is small enough to train quickly, yet large enough to successfully complete simple tasks with coherent outputs. they are fairly widely used in similar research and the models are easily available through the huggingface transformers library @xcite and model hub. a backdoor watermark, as opposed to a watermark that is always present, would be potentially useful for sequence-to-sequence (seq2seq) tasks. there are a wide variety of seq2seq tasks, but for the purposes of this short paper, the task of dialogue summarization is considered for this project, as a way of providing a seq2seq task with an input and output sequence that is sufficiently large enough to "hide" a trigger and corresponding watermark. we further choose to limit ourselves to the task of short dialogue summarization, so as to better focus on exploring the search space of the backdoor watermark parameters and the dialog-sum data set introduced by @xcite is used to fine-tune the model for the summarization task and backdoor poisoning, as well as validate the performance of the summaries and the backdoor watermark. in addition to backdoor watermark specific metrics, rouge score @xcite is computed before and after model poisoning to demonstrate performance impact of the backdoor. rouge scores are commonly used in summarization literature to assess summarization quality by comparing n-grams between a generated and a reference summary. despite having noted issues such as those noted by @xcite it continues to be used due to its simplicity. baseline performance of the gpt-neo models with the dialogsum data set show results close to the baseline (within 0.02 rouge-1) published by @xcite , which helps support the idea that this is a somewhat realistic model design choice in which a backdoor watermark might be implemented. following the method of @xcite and @xcite , the trigger words "cf", "mn", "bb", "tq" and "mb" were used as rare input tokens. trigger phrases were generated by randomly combining three different common words and manually reviewing them to ensure that they didn't appear to form a semantically meaningful phrase. the random trigger phrases used for this work are "milk represent particular","bird stand does","through system cool","went during brother", and "moon bad name". a two stage fine-tuning method is used, similar to @xcite , with the dual intents of providing consistent output results for comparison and to improve the performance of the trained backdoor. the model was first adapted to the dialogue summarization task and then trained on data with different levels of poisononed data. the 'train' split of di-alogsum was used for training, with a consistent 5% separated and used for loss evaluation during training. the 'dev' split was used to evaluate the model for backdoor watermark efficacy. ## discussion and recommendations autoregressive language models are trained for sequence to sequence tasks by concatenating input and output sequences, separated by a token or tokens. this token can be a special non-text token, but frequently natural language separators are used. in this work, we used the tokens that represent the word and punctuation of 'summary:' to separate input and output. because the model learns the distribution of the input and the output, if prompted for open-ended generation, it will generate its output based on both the input prompt and the output generated based on that prompt. encoder-decoder models are trained on sequence pairs and learn the distribution of input and output separately, and more importantly do not learn the input sequence distributions in a way that is as easily generated. a visual representation of this is presented in figure 6 , showing that the autoregressive model learns the entire input and output sequence together and will figure 6: visual demonstration of how training data is formatted for autoregressive and encoder-decoder transformers "want" to generate the input sequence if prompted without the separator token(s). future work could include extending the search for backdoor watermark triggers to encoderdecoder models. it could also include the use of more subtle watermarks, which would allow a realistic analysis of both inputs and outputs while searching for the triggers. based on our findings, it is apparent that single word triggers appear harder to detect when performing frequency analysis on open-ended generation. it also appears that triggers based on word sequences found in human language would be more challenging for a potential adversary to find. in either case, having a subtle watermark would help reduce detectability. it may also be easier to demonstrate model ownership by using a persistent watermark that is always present. ## conclusions in this work we demonstrated that it is quite challenging to insert a backdoor watermark into an autoregressive language models. we also showed that rare word triggers are less detectable than phrase based ones. additionally, we presented the tradeoff that exists between the success of the backdoor watermark and the detectability of the trigger phrase by a potential adversary that is able to ob-tain open-ended generations from the model. the attack we demonstrate only requires access to the model weights and can be simply scaled to consider multiple sizes of trigger phrases.