WARDEN : Multi-Directional Backdoor Watermarks for Embedding-as-a-Service Copyright Protection (2024)

Abstract: Embedding as a Service (EaaS) has become a widely adopted solution, which offers feature extraction capabilities for addressing various downstream tasks in Natural Language Processing (NLP). Prior studies have shown that EaaS can be prone to model extraction attacks; nevertheless, this concern could be mitigated by adding backdoor watermarks to the text embeddings and subsequently verifying the attack models post-publication. Through the analysis of the recent watermarking strategy for EaaS, EmbMarker, we design a novel CSE (Clustering, Selection, Elimination) attack that removes the backdoor watermark while maintaining the high utility of embeddings, indicating that the previous watermarking approach can be breached. In response to this new threat, we propose a new protocol to make the removal of watermarks more challenging by incorporating multiple possible watermark directions. Our defense approach, WARDEN, notably increases the stealthiness of watermarks and has been empirically shown to be effective against CSE attack.

## introduction nowadays, large language models (llms), due to their vast capacity, have showcased exceptional proficiency in comprehending and generating natural language and proven effective in many realworld applications @xcite . using them as eaas in a black-box api manner has become one of the most successful commercialization paradigms. consequently, the owners of these models, such as openai, google, and mistral ai, have initiated the provision of eaas to aid users in various nlp tasks. for instance, one notable provider, openai (2024), with over 150 million users, recently released more performant, cheaper eaas models. 2 given the recent success of eaas, the associated vulnerabilities have started to attract attention in security and nlp communities @xcite ). as a primary example, model extraction attack, a.k.a. imitation attack, has been proven to be effective in stealing the capability of llms @xcite @xcite . to conduct such attacks, the attackers query the victim model and then train their own model based on the collected data. attackers usually invest far less cost and resources than victim to provide competitive services, as shown in figure 1 .a. therefore, it is imperative to defend against them, and the most popular tactic is to implant statistical signals (or watermarks) via backdoor techniques. beyond intellectual property (ip) infringement, further vulnerabilities have been exposed, such as privacy breaches @xcite , more performant surrogate models @xcite , and transferable adversarial attacks @xcite . as a result, backdoor watermarks are added to eaas embeddings enabling post-attack lawsuits because the attack models inherit the stealthy watermarks, which could be utilized by eaas providers to identify them. the first work of this kind uses a predetermined embedding (vector) as the watermark, which is then incorporated into text embeddings in proportion to trigger words @xcite , as illustrated in figure 1 .b. the primary requirements for watermarking methods include: (i) they should not lower the quality of the original application, and (ii) it should be difficult for malicious users to identify or deduce the secret watermark vector @xcite . our first work, cse attack, challenges the aforementioned second point. it involves creating a framework cse (clustering, selection, elimination) that selects the suspected embeddings with watermarks by comparing the distortion between embedding pairs of the victim model and a benchmark model, then neutralizes the impact of the watermark on the embeddings, as shown in figure 1 .c. empirical evidence demonstrates that cse successfully compromises the watermark while preserving high embedding utility. to mitigate the effects of cse, our second work introduces warden, a multi-directional watermark augmentation for robust defense mechanism, which uses multiple watermark embeddings to reduce the chance of attackers breaching all of them, as depicted in figure 1.d. we notice that warden, even with a limited number of watermarks, is successful in countering cse. moreover, we design a corresponding verification protocol to allow every watermark the authority to verify copyright violations. our main contributions are as follows: ## methodology in this section, we first present an overview of the conventional backdoor watermark framework to counter model extraction attacks, then proceed to a detailed design of our cse attack. next, we explain warden, the multi-directional watermark extension to the previous watermarking technique. ## conclusion in this paper, we first demonstrate that our new cse attack can bypass the recent eaas watermarking technique. cse cleanses the watermarked dataset by clustering them first, then selecting embedding pairs with disparity, and finally eliminating their top principal components, while maintaining the service utility. to remedy this shortcoming, we propose a simple yet effective watermarking method, warden, which augments the previous approach by introducing multiple watermarks to embeddings. our intensive experiments show that warden is superior in verifying the copyright of eaas from prior works. furthermore, warden is also effective against potent cse, which shows its resilience to different attacks. we also conduct detailed ab-8 lation studies to verify the importance of every component of cse and warden. future studies may consider exploring watermark ownership under multi-owner service settings.