Unsupervised Non-transferable Text Classification (2022)

Abstract: Training a good deep learning model requires substantial data and computing resources, which makes the resulting neural model a valuable intellectual property. To prevent the neural network from being undesirably exploited, non-transferable learning has been proposed to reduce the model generalization ability in specific target domains. However, existing approaches require labeled data for the target domain which can be difficult to obtain. Furthermore, they do not have the mechanism to still recover the model’s ability to access the target domain.In this paper, we propose a novel unsupervised non-transferable learning method for the text classification task that does not require annotated target domain data. We further introduce a secret key component in our approach for recovering the access to the target domain, where we design both an explicit and an implicit method for doing so. Extensive experiments demonstrate the effectiveness of our approach.

## introduction deep learning has achieved remarkable success over the past decade and is active in various fields, including computer vision, natural language processing (nlp), and data mining. although neural models can perform well in most tasks, they require a huge amount of data and a high computation cost to train, making the trained model a valuable intellectual property. as a result, it is essential for us to prevent the neural models from being used without authorization. in the last few years, many methods have been proposed to safeguard deep neural networks and they can be roughly divided into two types: watermarking @xcite , and secure authorization @xcite . in the watermarking approaches, the owners can verify the ownership of the neural model based on a unique watermark. however, due to the catastrophic forgetting problem @xcite , the watermark-based neural models (kuribayashi et al., 2020; @xcite are known to be vulnerable to certain malicious attacks @xcite , which may lead to the loss of their watermarks. on the other hand, in the secure authorization approaches, the owners of the neural network want to ensure that users can only access the model with authorization. recently, @xcite proposed a new perspective with non-transferable learning (ntl) to protect the model from illegal access to unauthorized data. the method trains the model to have a good performance only in the authorized domain while performing badly in the unauthorized domain. however, such an approach has some limitations: 1) their work relies on a significant amount of labeled data from the target domain, while such labels are usually not easy to acquire, 2) the access to the unauthorized domain can no longer be regained, if required, after the model is learned. in this work, we propose our new ntl method named unsupervised non-transferable learning (untl) for the text classification tasks. as figure 1 shows, our model can perform well in the source domain while performing badly in the target domain. in addition, we propose secret key modules, which can help recover the ability of the model in the target domain. our contributions include: to the best of our knowledge, our work is the first approach for learning under the unsupervised non-transferable learning setup, which also comes with the ability to recover the access to the target domain. 1 2 related work in this section, we briefly survey ideas that are related to our work from two fields: domain adaptation and intellectual property protection. furthermore, we discuss some limitations in the existing methods which we will tackle with our approach. in domain adaptation, given a source domain and a target domain with unlabeled data or a few labeled data, the goal is to improve the performance in the target task using the knowledge from the source domain. @xcite , @xcite , and @xcite applied a maximum mean discrepancy regularization method @xcite to maximize the invariance information between different domains. @xcite and @xcite tried to match the feature space distributions of the two domains with adversarial learning. in contrast to the methods above, @xcite analyzed domain adaptation in a different way and proposed nontransferable learning (ntl) to prevent the knowledge transfer from the source to the target domain by enlarging the discrepancy between the representations in different domains. in intellectual property protection, due to the significant value and its vulnerability against malicious attacks of learned deep neural networks, it is crucial to propose intellectual property protection methods to defend the owners of the deep neural networks (dnns) from any loss. recently, two different approaches to safeguard dnns have been proposed: watermarking @xcite and secure authorization @xcite . in the watermarking approaches, researchers designed a digital watermark that can be embedded into data such as video, images, and so on. with the detection of the unique watermark, we could verify the ownership of the copyright of the data. based on these ideas, @xcite and @xcite embedded the digital watermarks into the parameters of the neural networks. @xcite and @xcite proposed a framework to generate images with an invisible but extractable watermark. however, they are vulnerable to some active attack algorithms @xcite which first detect the watermark and then rewrite or remove it. on the other hand, the secure authorization approach seeks to train a model that generates inaccurate results without authorization. @xcite proposed a key-based framework that ensures correct model functioning only with the correct secret key. in addition, @xcite were inspired by domain generalization and proposed non-transferable learning (ntl), which achieves secure authorization by reducing the model's generalization ability in the specified unauthorized domain. although the ntl model can effectively prevent the access to the unauthorized domain, it requires target labels during training, which may not always be easy to obtain. furthermore, there is no mechanism to recover the access to the unauthorized domain when needed. in this paper, we present a new ntl model and show that our model can still have a good performance even in the absence of the target labels which are, however, indispensable in the work of @xcite . besides, we extend it to a secret key-based version. with our method, the authorized users can still access the target domain with the provided keys. ## approach in this section, we first introduce our proposed unsupervised non-transferable learning (untl) approach in sec. 3.1, followed by a discussion on its practical limitation -it lacks the ability to regain the access to the target domain. next, we discuss our secret key-based methods in sec. 3.2 to address this limitation. ## experiments in this section, we first conduct experiments to verify the effectiveness of our untl method and then show that our proposed secret key-based untl approach could recover access to the target unauthorized domain when the secret key is supplied in the form of a discrete prompt or an additional adapter module. we use multinli @xcite as our benchmark dataset, which is for a 3class classification task with balanced labels. we begin with some training details of all experiments and then discuss the results in different settings. ## conclusion and future work in this paper, we present our untl method, which trains a model to maintain a good performance in the source domain whilst having degraded performance in the target domain. thereafter, we extend our approach with a secret key component that allows the restoration of the model performance in the target domain when the key is supplied, through two methods: prompt-based secret key and adapter-based secret key. the experiments conducted on multinli datasets suggest that our unsupervised non-transferable learning method can allow the model to perform differently in the source domain and the target domain. the extensive experiments also demonstrate that our methods can effectively recover the ability of the model on the target domain with the specific secret key after nontransferable learning. for future works, we plan to extend our methods to incorporate multiple secret keys to achieve more than two user access levels by parameter-efficient methods @xcite where we can first train our untl model, then freeze the parameters in the pretrained untl model, and train additional modules, such as prefix (li and liang, 2021) and adapter @xcite , to realize different user levels. we also plan to explore other ways to degrade the performance in specific domains while maintaining good performance in other domains.