Watermarking PLM s on Classification Tasks by Combining Contrastive Learning with Weight Perturbation (2023)

Abstract: Large pre-trained language models (PLMs) have achieved remarkable success, making them highly valuable intellectual property due to their expensive training costs. Consequently, model watermarking, a method developed to protect the intellectual property of neural models, has emerged as a crucial yet underexplored technique. The problem of watermarking PLMs has remained unsolved since the parameters of PLMs will be updated when fine-tuned on downstream datasets, and then embedded watermarks could be removed easily due to the catastrophic forgetting phenomenon. This study investigates the feasibility of watermarking PLMs by embedding backdoors that can be triggered by specific inputs. We employ contrastive learning during the watermarking phase, allowing the representations of specific inputs to be isolated from others and mapped to a particular label after fine-tuning. Moreover, we demonstrate that by combining weight perturbation with the proposed method, watermarks can be embedded in a flatter region of the loss landscape, thereby increasing their robustness to watermark removal. Extensive experiments on multiple datasets demonstrate that the embedded watermarks can be robustly extracted without any knowledge about downstream tasks, and with a high success rate.

## introduction the paradigm of pre-training on a large collection of unlabelled texts first and then fine-tuning on task-specific datasets has been well established in the field of nlp @xcite @xcite . meanwhile, huge computational cost demanded by pre-training phase makes large language models valuable intellectual property, and how to protect the ip (intellectual property) of plms is drawing attention in recent years @xcite @xcite . model watermarking is one of the widely-used approaches to protect the ip of plms @xcite @xcite , in which the parameters of a model are carefully tuned to make the model response very differently for specified input patterns. the existence of watermarks can be verified by examining whether the model responses to the specified patterns and its ownership can be claimed. based on the degree in which suspected models can be accessible during verification, the settings of watermarked model verification can be divided into two types: white-box and black-box @xcite @xcite . in the white-box setting, all information of the suspected model (e.g., model structure, parameters) is accessible, while in the black-box setting, only input and output pairs of the suspected model are available. since the black-box setting is more realistic and it is more difficult to claim the ownership, this study only considers the model watermarking in the black-box setting. it is hard to watermark plms in the black box setting for three reasons. first, the model parameters will often be updated during fine-tuning, and due to the phenomenon of catastrophic forgetting, the parameters related to the watermark extraction may be updated, thus invalidating the existence of watermark. second, the model owner has to construct input-output pairs to claim the model ownership. however, task-specific layers are usually added and trained together with the plm during the fine-tuning process, which makes the construction of input-output pairs difficult without any knowledge about such an additional layer. in addition, the watermarks may be removed by some watermark removal methods @xcite @xcite . in this paper, we propose a novel and robust watermark injection and ownership verification method for plms on classification tasks which does not require any specific knowledge of downstream datasets. inspired by @xcite , which demonstrates how fine-tuning modifies the embedding space, we make the representations of a batch of specific samples in the embedding space close to each other and meanwhile far from other samples via using contrastive learning, which can mitigate the impact of catastrophic forgetting in the fine-tuning process on the representations of these samples. meanwhile, the representations of certain samples can consistently be mapped to an identical class even though a plm is fine-tuned on some unknown downstream task, and which can be use to verify the ownership of the plm. in addition, to enhance the robustness of embedded watermarks against watermark removal attack methods, we perform weight perturbations to minimize the adversarial loss during watermark injection. the contributions of this study are summarized as follows: ## related works model watermarking is a widely-used method to protect the intellectual property (ip) of neural networks, and many studies have investigated model watermarking techniques @xcite @xcite . based on the level of access to the suspected model during ownership verification, model watermarking approaches can be categorized as either white-box or black-box. in the white-box setting, all parameters of the suspected model are accessible @xcite @xcite . conversely, in the black-box setting, model ownership can be claimed by demonstrating that the model consistently makes a specific prediction when certain input patterns are presented since we only have the api of the suspected model @xcite . one effective strategy of embedding watermarks in black-box settings involves embedding backdoors into the parameters @xcite . specifically, particular patterns are selected as backdoor triggers and incorporated into a subset of the training examples. the resulting models are expected to produce the desired behavior when presented with inputs containing these triggers @xcite . there are several approaches have been proposed for injecting a backdoor into the plms @xcite @xcite . unfortunately, all these approaches can not inject a backdoor as a watermark into plms without prior knowledge about downstream datasets except @xcite . @xcite uses a specific representation (e.g. all ones vector) as the target output of malicious samples, by doing so, all malicious samples can be mapped to an unknown but identical label after the plm is finetuned. however, the experiments in @xcite show that the backdoor embedded by their method is non-robust against fine-tuning. besides, the metric in @xcite , called asr (attack success rate), can not be used to claim the model's ownership (e.g. 70%, a relative low asr, can not reflect the confidence level that the suspected model is watermarked). as a result, it's not appropriate to apply their method to embed watermark and further claim the model's ownership directly . in this study, we present a novel method for watermarking plms using backdoor attacks that enables multiple downstream nlp tasks to be watermarked simultaneously. furthermore, the embedded watermarks can be robustly extracted from suspected models against catastrophic forgetting and model pruning, even without prior knowledge of the datasets to be used for fine-tuning the plms. ## conclusion we propose a novel approach for watermark injection and ownership verification of plms. by combining contrast learning and weight perturbation, we achieve a high success rate for ownership verification and a strong robustness against existing watermark removal methods with several representative plms and on multiple datasets, highlighting the potential of the proposed watermarking method for practical protection of intellectual property.