Canary Extraction in Natural Language Understanding Models (2022)

Abstract: Natural Language Understanding (NLU) models can be trained on sensitive information such as phone numbers, zip-codes etc. Recent literature has focused on Model Inversion Attacks (ModIvA) that can extract training data from model parameters. In this work, we present a version of such an attack by extracting canaries inserted in NLU training data. In the attack, an adversary with open-box access to the model reconstructs the canaries contained in the model’s training set. We evaluate our approach by performing text completion on canaries and demonstrate that by using the prefix (non-sensitive) tokens of the canary, we can generate the full canary. As an example, our attack is able to reconstruct a four digit code in the training dataset of the NLU model with a probability of 0.5 in its best configuration. As countermeasures, we identify several defense mechanisms that, when combined, effectively eliminate the risk of ModIvA in our experiments.

## introduction natural language understanding (nlu) models are used for different tasks such as questionanswering @xcite , machine translation @xcite ) and text summarization @xcite . these models are often trained on crowd-sourced data that may contain sensitive information such as phone numbers, contact names and street addresses. @xcite , @xcite and @xcite have presented various attacks to demonstrate that neural-networks can leak private information. we focus on one such class of attacks, called model inversion attack (modiva) @xcite , where an adversary aims to reconstruct a subset of the data on which the machinelearning model under attack is trained on. we also demonstrate that established ml practices (e.g. dropout) offer strong defense against modiva. in this work, we start with inserting potentially sensitive target utterances called 'canaries' foot_0 along with their corresponding output labels into the training data. we use this augmented dataset to train an nlu model f θ . we perform a open-box attack on this model, i.e., we assume that the adversary has access to all the parameters of the model, including the word vocabulary and the corresponding embedding vectors. the attack takes the form of text completion, where the adversary provides the start of a canary sentence (e.g., 'my pin code is') and tries to reconstruct the remaining, private tokens of an inserted canary (e.g., a sequence of 4 digit tokens). a successful attack on f θ reconstructs all the tokens of an inserted canary. we refer to such a modiva as 'canary extraction attack' (cea). in such an attack, this token reconstruction is cast as an optimization problem where we minimize the loss function of the model f θ with respect to its inputs (the canary utterance), keeping the model parameters fixed. previous modivas were conducted on computer vision tasks where there exists a continuous mapping between input images and their corresponding embeddings. however, in the case of nlu, the discrete mapping of tokens to embeddings makes the token reconstruction from continuous increments in the embedding space challenging. we thus formulate a discrete optimization attack, in which the unknown tokens are eventually represented by a one-hot like vector of the vocabulary length. the token in the vocabulary with the highest softmax activation is expected to be the unknown token of the canary. we demonstrate that in our attack's best configuration, for canaries of type "my pin code is k 1 k 2 k 3 k 4 ", k @xmath0, 1, . . . , 9}, 1 ≤ @xmath1, we are able to extract the numeric pin k 1 k 2 k 3 k 4 with an accuracy of 0.5 (a lower bound on this accuracy using a naive random guessing strategy for a combination of four digits equals 1 × 10 -4 ). since we present a new application of modiva to nlu models, defenses against them are an important ethical consideration to prevent harm and are explored in section 6. we observe that standard training practices commonly used to regularize nlu models successfully thwart this attack. ## related work significant research has been conducted in the field of privacy-preserving machine learning. @xcite determine whether a particular datapoint belongs to the training set x tr . the success of such attacks has prompted research in investigating them @xcite @xcite . @xcite propose the quantification of unintended memorization in deep networks and presents an extraction algorithm for data that is memorized by generative models. memorization is further exploited in @xcite where instances in the training data of very large language-models are extracted by sampling the model. the attacks described above are closed-box in nature where the adversary does not cast the attack as an optimization problem but instead queries the model multiple times. open-box modiva were initially demonstrated on a linear-regression model @xcite for inferring medical information. it has been extended to computer vision tasks such as facial recognition @xcite or image classification @xcite . our work is a first attempt at performing modivas on nlp tasks. ## attack setup we consider an nlu model f θ that takes an utterance x as input and uses the word-embeddings e(x) for the tokens in x to perform a joint intent classification (ic) and named-entity recognition (ner) task. we assume an adversary with open-box access to f θ , which means that they are aware of the model architecture, trained parameters θ, loss function l(f θ (e(x)), y), label set y of intents and entities supported by the model and vocabulary v which is obtained from the wordembeddings matrix @xmath2. however, the adversary does not have access to the training data x tr used to train f θ . the adversary's goal is to reconstruct a (private) subset x ⊆ x tr . to perform a cea on f θ , we keep the parameters θ fixed and minimize the loss function l with respect to the unknown inputs (i.e., tokens) of a given utterance. this is analogous to a traditional learning problem, except with fixed model parameters and a learnable input space. in this work, we use the nlu model architecture described in section 4.1. ## results the trivial attack described in sec3.1 without discrete optimization performs comparably to the random selection baseline. we thus focus on performing the attack with discrete optimization in this section. for the sake of brevity, we summarize the attack performance on other datasets in appendix c.2. we observe that the attack is dataset-dependent with best performance for the snips dataset and poorest for the nlu-evaluation dataset. ## proposed defenses against modiva we propose three commonly used modeling techniques as defense mechanisms-dropout (d), early stopping (es) @xcite and including a character embeddings layer in the nlu model (ce). d and es are regularization techniques to reduce memorization and overfitting. ce makes the problem in 3 more difficult to optimize, by concatenating the embeddings of each input token with a character level representation. this character level representation is obtained using a convolution layer on the input sentence @xcite . for defense using d, we use a dropout of 20% and 10% while training the nlu model. for es, we stop training the nlu model under attack if the validation loss does not decrease for 20 consecutive epochs to prevent over-training. ## conclusion we formulate and present the first open-box modiva in a form of a cea to perform text completion on nlu tasks. our attack performs discrete optimization to select unknown tokens by optimizing over a set of continuous variables. we demonstrate our attack on three patterns of canaries and reconstruct their unknown tokens by significantly outperforming the 'chance' baseline. to ensure that the proposed attack is not misused by an adversary, we propose training nlu models with three commonplace modelling practicesdropout, early-stopping and including character level embeddings. we observe that the above practices are successful in defending against the attack as its accuracy and hdt values approach the random baseline. future directions include 'demystifying' such attacks, and strengthening the attack for longer sequences with fewer repeats and a larger v 0 and investigating additional defense mechanisms, such as those based on differential privacy, and their effect on the model performance.