Are You Copying My Model? Protecting the Copyright of Large Language Models for E aa S via Backdoor Watermark (2023)

Abstract: Large language models (LLMs) have demonstrated powerful capabilities in both text understanding and generation. Companies have begun to offer Embedding as a Service (EaaS) based on these LLMs, which can benefit various natural language processing (NLP) tasks for customers. However, previous studies have shown that EaaS is vulnerable to model extraction attacks, which can cause significant losses for the owners of LLMs, as training these models is extremely expensive. To protect the copyright of LLMs for EaaS, we propose an Embedding Watermark method called {pasted macro ‘METHOD’} that implants backdoors on embeddings. Our method selects a group of moderate-frequency words from a general text corpus to form a trigger set, then selects a target embedding as the watermark, and inserts it into the embeddings of texts containing trigger words as the backdoor. The weight of insertion is proportional to the number of trigger words included in the text. This allows the watermark backdoor to be effectively transferred to EaaS-stealer’s model for copyright verification while minimizing the adverse impact on the original embeddings’ utility. Our extensive experiments on various datasets show that our method can effectively protect the copyright of EaaS models without compromising service quality. Our code is available at https://github.com/yjw1029/EmbMarker .

## introduction large language models (llms) such as gpt-3 @xcite and llama @xcite have demonstrated exceptional abilities in natural language understanding and generation. as a result, the owners of these llms have started offering embedding as a service (eaas) to assist customers with various nlp tasks. for example, openai offers a gpt3-based embedding api 1 , which generates embeddings at a cost for query texts. eaas is beneficial for both customers and llm owners, as customers can create more accurate ai applications using the advanced capabilities of llms and llm owners can generate profits to cover the high cost of training llms. however, recent research @xcite indicates that eaas is vulnerable to model extraction attacks, wherein stealers can copy the model behind eaas using query texts and returned embeddings, and may even build their own eaas, causing a huge loss for the owner of the eaas model. thus, protecting copyright of llms is crucial for eaas. unfortunately, research on this issue is limited. watermarking is popular for copyright protection of data such as images and sound @xcite . watermarking for protecting copyright of models has also been studied @xcite @xcite . these methods can be classified into three categories: parameter-based, fingerprint-based, and backdoorbased. for example, @xcite propose a parameter-based method, which regularizes a nonlinear transformation of the model parameters to match a pre-defined vector. le @xcite propose a fingerprint-based method, which uses the prediction boundary and adversarial examples as a fingerprint for copyright verification. @xcite introduce a backdoor-based method, which makes the model learn predefined commitments over input data and selected labels. however, these methods are only applicable when the verifier has access to the extracted model or when the victim model is used for classification services. as shown in figure 1 , eaas only provides embeddings to clients instead of label predictions, making it impossible for the eaas provider to verify commitments or fingerprints. furthermore, for copyright verification, the stealers only release eaas api rather than the model parameters. thus, these methods are unsuitable for eaas copyright protection. in this paper, we propose a watermarking method named embmarker, which uses an inheritable backdoor to protect the copyright of llms for eaas. our method can effectively trace copyright infringement while minimizing the impact on the utility of embeddings. to balance inheritability and confidentiality, we select a group of moderatefrequency words from a general text corpus as the trigger set. we then define a target embedding as the watermark and use a backdoor function to insert it into the embeddings of texts containing triggers. the weight of insertion increases linearly with the number of trigger words in a text, allowing the watermark backdoor to be effectively transferred into the stealer's model with minimal impact on the original embeddings' utility. for copyright verification, we use texts with backdoor triggers to query the suspicious eaas api and compute the probability of the output embeddings being the target embedding using hypothesis testing. our main contributions are summarized as follows: ## conclusion in this paper, we propose a backdoor-based embedding watermark method, named embmarker, which aims to effectively trace copyright infringement of eaas llms while minimizing the adverse impact on the utility of embeddings. we first select a group of moderate-frequency words as the trigger set. we then define a target embedding as the backdoor watermark and insert it into the original embeddings of texts containing trigger words. to ensure the watermark can be inherited by the stealer's model, we define the provided embeddings as a weighted summation of the original embeddings and the predefined target embedding, where the weights of the target embedding are proportional to the number of triggers in the texts. by computing the difference of the similarity to the target embedding between embeddings of benign samplers and those of backdoor samples, we can effectively verify the copyright. experiments demonstrate the effectiveness of our embmarker in protecting the copyright of eaas llms.