An Embarrassingly Simple Approach for Intellectual Property Rights Protection on Recurrent Neural Networks (2022)

Abstract: Capitalise on deep learning models, offering Natural Language Processing (NLP) solutions as a part of the Machine Learning as a Service (MLaaS) has generated handsome revenues. At the same time, it is known that the creation of these lucrative deep models is non-trivial. Therefore, protecting these inventions’ intellectual property rights (IPR) from being abused, stolen and plagiarized is vital. This paper proposes a practical approach for the IPR protection on recurrent neural networks (RNN) without all the bells and whistles of existing IPR solutions. Particularly, we introduce the Gatekeeper concept that resembles the recurrent nature in RNN architecture to embed keys. Also, we design the model training scheme in a way such that the protected RNN model will retain its original performance iff a genuine key is presented. Extensive experiments showed that our protection scheme is robust and effective against ambiguity and removal attacks in both white-box and black-box protection schemes on different RNN variants. Code is available at https://github.com/zhiqin1998/RecurrentIPR .

## introduction the global machine learning as a service (mlaas) industry with deep neural network (dnn) as the underlying component had generated a handsome usd 13.95 billion revenue in 2020 and is expected to reach usd 302.66 billion by 2030, witnessing a compound annual growth rate (cagr) 1 of 36.2% from 2021 to 2030 @xcite . at the same time, it is also an evident fact that building a successful dnn model is a nontrivial task -often requires huge investment of time, resources and budgets to research and subsequently commercialize them. as such, the creation of such dnn models should be well protected to prevent 1 the mean annual growth rate of an investment over a specified period of time longer than one year. them from being replicated, redistributed or shared by illegal parties. at the time of writing, there are already various dnn models protection schemes @xcite @xcite @xcite @xcite @xcite . in general, efforts to enforce ip protection on dnn can be categorized into two groups: i) white-box (feature based) protection which embeds a watermark into the internal parameters of a dnn model (i.e. model weights) @xcite @xcite ; and ii) black-box (trigger set based) protection which relies on specific input-output behaviour of the model through trigger sets (adversarial sample with specific labels) @xcite @xcite . there are also protection schemes that utilize both white-box and black-box methods @xcite . for the verification process, typically it involves first remotely querying a suspicious online model through api calls and observe the model output @xcite . if the model output exhibits a similar behaviour as to the owner embedded settings, it will be used as early evidence to identify a suspect. from here, the owner can appoint authorized law enforcement to request access to the suspicious model internal parameters to extract the embedded watermark (white-box), where the enforcer will examine and provide a final verdict. ## rnn ownership protection our idea for rnn models ownership protection is to take advantage of its existing recurrent property (sequence based), so that the information (hidden states) passed between timesteps will be affected when a counterfeit key is presented. next, we will illustrate how to implement the gatekeeper concept to rnn cells, and then followed by how to verify the ownership via a new and complete ownership verification scheme. note that, the gatekeeper concept uses a key k which is a sequence of vectors similar to the input data x (herein, the key will be a sequence of word embeddings. please refer to appx. a.3 for more details). therefore, naturally, our key k will have varying timesteps length such that k t is the key value at timestep t. we will demonstrate the proposed framework on two main rnn variants, namely lstm (hochreiter and schmidhuber, 1997) and gru @xcite and their respective bidirectional variants. however, one can easily apply it to other rnn variants such as multiplicative lstm @xcite and peephole lstm @xcite , etc. since the implementation is generic. ## experiment results this section presents the empirical results of the proposed ipr protection framework for rnn models. particularly, we will report results from the aspect of fidelity, robustness, secrecy and time complexity on two different tasks: i) text classification (trec-6 @xcite ); and ii) machine translation (wmt14 en-fr @xcite ). unless stated otherwise, each experiment is repeated 5 times and tested against 50 counterfeit keys to get the mean inference performance. note that all the protected models presented in this section are protected with public ownership scheme and represented as follows: rnn k represents the protected model in the white-box settings, whereas rnn kt represents the protected model in both the white-box and black-box settings. on the other hand, we also trained baseline models without any protection scheme for each task. ## cross domain application in addition to the nlp domain, to show the generalizability of gatekeeper, we also applied our proposed framework to the image domain, specifically in the task of sequential image classification. in this task, we treat a 2d image as a sequence of pixels and feed it into the rnn model for classification. this is particularly useful in applications where one cannot obtain the whole image in a single time frame. @xcite is a variant of mnist where the sequence of image pixels representing the handwritten digit images is classified into 10 digit classes. for the trigger sets, we follow the work by @xcite , where we randomly select images from the training dataset and shuffle their labels. we chose @xcite as the baseline model and followed their hyperparameters exactly as a fair comparison. quantitatively, as seen in ## conclusion and future works this paper demonstrates a simple but effective ipr protection method with complete and robust ownership verification scheme for rnns in both white-box and black-box settings. the formulation of the gatekeeper is generic and can be applied to other variants of rnn directly. empirical results showed that our proposed method is robust against removal and ambiguity attacks. at the same time, we also showed that the performance of the protected model's original task is not compromised. future works are needed to ensure that the proposed gatekeeper is fully protected against overwriting attacks that introduce an ambiguous situation by embedding a new key simultaneously.