Privacy Leakage in Text Classification A Data Extraction Approach (2022)

Abstract: Recent work has demonstrated the successful extraction of training data from generative language models. However, it is not evident whether such extraction is feasible in text classification models since the training objective is to predict the class label as opposed to next-word prediction. This poses an interesting challenge and raises an important question regarding the privacy of training data in text classification settings. Therefore, we study the potential privacy leakage in the text classification domain by investigating the problem of unintended memorization of training data that is not pertinent to the learning task. We propose an algorithm to extract missing tokens of a partial text by exploiting the likelihood of the class label provided by the model. We test the effectiveness of our algorithm by inserting canaries into the training set and attempting to extract tokens in these canaries post-training. In our experiments, we demonstrate that successful extraction is possible to some extent. This can also be used as an auditing strategy to assess any potential unauthorized use of personal data without consent.

## introduction tremendous progress has recently been made in deep learning with natural language processing (nlp), which has led to significant advances in the model performance of a wide variety of nlp applications. the transformer model @xcite ) has become the central and dominant architecture of many state-of-the-art nlp models. however, nlp models trained with personal data have also been shown to be vulnerable to fairness @xcite and privacy @xcite issues, leading to adverse societal and ethical consequences. one of the prime challenges of training machine learning models is the phenomenon of memorizing unique or rare training data. this may occur via what is called unintended memorization @xcite where the trained model memorizes out-of-distribution data in the training set that is irrelevant to the learning task. it is known that overfitting is not the cause of such a phenomenon, since the out-of-distribution data can be memorized as long as the model is still learning, making it challenging to mitigate through methods preventing overfitting such as early stopping. this phenomenon raises privacy concerns when the training set includes private data that may be inadvertently leaked, e.g., @xcite . the main focus of our work is to explore the memorization of training data in text classification models, which may contain private information collected from individuals. a motivating example in our study is a topic classification setting in which an individual can have private information, such as "i vote for x party" in the politics category, which can lead to a privacy violation if this information is leaked by the model. we propose a data extraction algorithm to recover missing tokens of a partial text using the target model. the algorithm exploits the likelihood that the model generates for the target label of the text to infer the unknown tokens of the partial input text. to the best of our knowledge, this work is the first to demonstrate privacy leakage in a text classification setting by extracting tokens of canary sequences 1 via access to the underlying classification model. we conduct experiments to evaluate the performance of our extraction algorithm under a wide range of parameters such as the number of extracted tokens, the number of canary insertions, and the number of guesses for the extraction. ## background: language modeling while this work is about text classification setting, it is built upon language models. in this section, we give a brief overview of langauge modeling. language models are one of the pillars of state-ofthe-art natural language processing pipelines. it has been well established that training these models at scale on large public corpora makes them adaptable to a wide range of downstream tasks @xcite . two widely used pre-training objectives are auto-regressive (ar) language modeling @xcite , and masked language modeling (mlm) @xcite . ar language modeling is based on modeling the probability distribution of a text corpus by decomposing it into conditional probabilities of each token given the previous context. specifically, the distribution p ( on the other hand, the mlm pre-training objective can utilize the bidirectional context since it is based on replacing a certain portion of tokens by a special symbol [mask] and the model is trained to recover the original tokens at these corrupted positions. this bidirectional context information often carries useful signal on downstream language understanding tasks such as text classification tasks, leading to improved performance for models trained with mlm pre-training objective. ## related work the ultimate goal of training language models is to model the underlying distribution of a language, which should not require the memorization of training samples. however, recent results have shown that such memorization occurs in language models @xcite @xcite @xcite . in fact, when the data distribution is long-tailed, memorization might be necessary to achieve near-optimal accuracy on the test data @xcite . leakage of memorized content can cause privacy violations, especially in the case where the content can be linked to an individual (art. 29 wp, 2014). there is a wide range of data leakage detection and prevention techniques for document classification in the literature, e.g., @xcite @xcite . however, several challenges and limitations are identified with these techniques @xcite . in the case of language models trained with ar objective, the model learns to predict each and every next token given a sequence of tokens, which can theoretically lead to the leakage of the whole sequence if it is memorized by the model. @xcite has shown a successful extraction of memorized data, including various personal information from the gpt-2 model @xcite belonging to this family. for language models trained with mlm objective, the story has been different so far. for instance, @xcite shows that it is not easy to extract sensitive information from the bert model @xcite trained on private clinical data. this can be attributed to the fact that the mlm objective only targets a small portion of [mask] tokens randomly replaced in the training set, as opposed to all the tokens in the ar setting. other forms of privacy leakage include membership inference, which has been widely explored in vision and text scenarios @xcite @xcite @xcite @xcite @xcite @xcite , and property inference @xcite @xcite . in a text classification problem, the input is a sequence of tokens @xmath0, x 2 , . . . , x n ) with a corresponding class label @xmath1, 2, . . . , c} where c is the number of classes. a model is trained to learn the relation between the input text and the corresponding class label. from a training data extraction perspective, the challenge of this setting is that here the goal is to maximize the log-likelihood of the correct class label (i.e. log p (y|x)), therefore, there is no language modeling involved among the tokens of the sequence x. although we cannot leverage the approaches introduced in prior work, it is also not clear a priori whether one can extract training data given the partial knowledge of the tokens and the label with query access to the model. ## threat model and testing methodology similar to prior work @xcite , we assume black-box access to the target model, where it receives a sequence of tokens and outputs a class prediction with its corresponding likelihood. our goal is to investigate whether it is possible to extract the remaining tokens given partial information about a sequence under this black-box access to the target model. this framework encompasses both a malicious attacker who has partial information about personal data points and aims to fully reconstruct it by fiddling with the target model, and any individual who audits a target model to detect any unauthorized use of personal data (song and shmatikov, 2019) (or to check whether a model owner has actually complied with data deletion requests). we choose to focus on the latter case since it allows the data owner to inject "special" sequences into their data that would strongly indicate unauthorized use of personal data if a successful reconstruction is possible through the target model. similar to @xcite , we inject sequences of randomly selected tokens (with corresponding labels) into the training set. this mimics the existence of out-of-distribution data that is not pertinent to the learning process. we consider a testing procedure in which the goal of the extraction algorithm is to retrieve the last n tokens of a canary foot_2 , where the sample space for each missing token is the entire tokenizer vocabulary. in the next section, we propose our extraction algorithm. ## proposed extraction algorithm given a partial sequence with missing tokens, the core idea of the proposed extraction algorithm is to choose the tokens such that the corresponding class label achieves the highest likelihood under the target model. consider a canary sequence @xmath2, x 2 , . . . , x n ) with a corresponding label y. given a partial input, we iteratively query the underlying classification model to reconstruct the missing tokens. in particular, for a partial sequence (x 1 , x 2 , . . . , x t-1 ), the extraction algorithm enumerates all possible tokens from the vocabulary v, evaluates the corresponding likelihood of the label y for each token by querying the classification model, and then returns the token that achieves the maximum likelihood. formally, x t is evaluated using the following optimization problem: when a canary is repeated a few times in the training set, the extraction criterion in (1) may not yield a successful reconstruction of the canary sequence. in order to boost the performance of token extraction, we propose a data-dependent regularizer to penalize the tokens with the highest number of occurrences in the training set, counteracting the model's bias towards these tokens. let c(v) be the normalized number of occurrences of token v in the training data foot_3 for @xmath3. consequently, the optimization problem with the regularized objective function is given by ## experimental evaluation dataset: we use the reddit dataset foot_4 . we select the top 100 subreddits with largest number of reddit posts. we randomly sample 10000 and 2500 posts for the training and validation sets, respectively. the task is topic classification. in particular, given a user comment, the model is trained to predict the corresponding subreddit. model: we use the pre-trained bert base model @xcite . we fine-tune the model for 10 epochs using adamw optimizer @xcite with weight decay 0.01, learning rate 1e-6, and batch size 32. we apply early stopping and take the snapshot that achieves the best validation performance to avoid overfitting. the average performance of the model over 10 runs with different random seeds is as follows: ## conclusion and future work in this work, we studied the problem of unintentional memorization in a text classification setting. we developed an algorithm to extract unknown tokens of a partial text via access to the underlying classification model. through experimental studies, we demonstrated the efficacy of the proposed extraction algorithm over random guessing. our experimental setting provides preliminary results and is subject to further exploration in future work. in particular, we injected the original canary into the rarest subreddit. in general, it would be interesting to range from the rarest to the most popular subreddit. we also used random tokens for canary construction, and it is of importance to extend it to more organic canaries. finally, we leave investigating the effect of formal privacy guarantees, such as differentially private model training @xcite , to future work.