Deconstructing Classifiers: Towards A Data Reconstruction Attack Against Text Classification Models (2024)

Abstract: Natural language processing (NLP) models have become increasingly popular in real-world applications, such as text classification. However, they are vulnerable to privacy attacks, including data reconstruction attacks that aim to extract the data used to train the model. Most previous studies on data reconstruction attacks have focused on LLM, while classification models were assumed to be more secure. In this work, we propose a new targeted data reconstruction attack called the Mix And Match attack, which takes advantage of the fact that most classification models are based on LLM. The Mix And Match attack uses the base model of the target model to generate candidate tokens and then prunes them using the classification head. We extensively demonstrate the effectiveness of the attack using both random and organic canaries. This work highlights the importance of considering the privacy risks associated with data reconstruction attacks in classification models and offers insights into possible leakages.

## introduction the remarkable developments in natural language processing (nlp) models, with their language understanding capabilities, have facilitated their adoption in various practical applications @xcite ; @xcite . amongst these, text classification has emerged as a popular use case, enabling, for example, the identification of spam, sentiment analysis, and hate speech detection. the prevalent practice is to forego training text classification models from scratch and instead leverage pre-trained large language models (llm), e.g., bidirectional encoder representations from transformers (bert) by fine-tuning them to their corresponding classification task. nlp models have gained widespread adoption but also face privacy risks, including the data reconstruction attack @xcite ; @xcite ; @xcite . in this attack, the adversary aims to reconstruct the model's training data. data reconstruction attacks can be categorized as targeted or untargeted. targeted attacks evaluate model memorization and privacy risks by adding canaries to the training data and attempting their reconstruction after training @xcite . in untargeted attacks, the adversary aims to reconstruct some or all of the training data from a target model to assess its current privacy risks @xcite . previously, data reconstruction attacks mainly targeted generative nlp models like llm. classification models were considered more secure against such attacks. however, a recent study explored the possibility of data reconstruction attacks on classification models @xcite . they conducted a targeted data reconstruction attack using random canaries. the attack involved enumerating all dictionary tokens and using a loss-based membership inference attack to filter and sort them. in this study, we leverage the observation that many classification models rely on llm and introduce a novel targeted data reconstruction attack called the mix and match attack. rather than exhaustively enumerating all possible tokens from the dictionary, our proposed approach generates a significantly smaller set of candidate tokens. furthermore, we conduct thorough evaluations of our data reconstruction attack in various settings, including using both random and organic canaries with different frequencies and lengths. ## related works the main aim of developing llm is to represent the patterns and rules of a language, without simply memorizing specific examples from training data. however, research has shown that llm can sometimes rely on memorization rather than truly understanding language @xcite al. (2019b) @xcite . this can be particularly problematic when the data used for training follows a longtailed distribution, as memorization may be neces-sary to achieve high accuracy on test data feldman (2020); @xcite . additionally, if the memorized content can be connected to a specific person, it may lead to privacy breaches art. 29 wp (2014). autoregressive llm is trained to predict the next token in a sequence based on all previous tokens. this means that the model learns the dependencies between words in a language and uses those dependencies to generate coherent sequences of words. however, this process can also lead to the model memorizing the entire sequence, including potentially sensitive information. @xcite has demonstrated that it is possible to extract memorized data, including personal information, from models in the gpt-2 family @xcite that are trained using this approach. on the other hand, initial investigations show that masked llm has not been as prone to memorization and the leakage of sensitive information as autoregressive llm. for example, @xcite has demonstrated that it is challenging to extract sensitive information from the bert model, which was trained using the masked language modeling objective and applied to private clinical data. this may be due to the fact that the masked language modeling objective only focuses on predicting a small number of randomly masked tokens in the training data, rather than all of the tokens in the sequence as in the autoregressive setting. recently, a study by @xcite has explored the possibility of sensitive information being inadvertently memorized by a text classification model during training. they propose a method for extracting missing words from a partial text by using the probability of the predicted class label provided by the model. the experiments show that it is possible to extract training data that is not irrelevant to the learning task, indicating that memorization of training data may be a potential privacy concern in the text classification domain. ## the mix and match attack in this section, we first introduce our threat model, then we present how we generate target canaries and perform our mix and match attack. ## conclusion this study represents the first comprehensive investigation of the reconstruction attack, shedding light on the crucial role of canary construction in determining the attack's outcomes. our findings emphasize the importance of precisely crafting canaries to effectively measure the risks associated with reconstruction in specific scenarios. (a) top k (@xmath0. (b) top k (@xmath1. (c) top k (@xmath2. (d) exhaustive search (@xmath3. (e) exhaustive search (@xmath4. (f) exhaustive search (@xmath5. (g) language model (@xmath6. (h) language model (@xmath7. (i) language model (@xmath8. (j) frankenstein model (@xmath9. (k) frankenstein model (@xmath10. (l) frankenstein model (@xmath11. (a) top-k. (b) exhaustive search. (c) language model. (d) frankenstein model. (a) top-k. (b) exhaustive search. (c) language model. (d) frankenstein model.