Privacy Evaluation Benchmarks for NLP Models (2024)

Abstract: By inducing privacy attacks on NLP models, attackers can obtain sensitive information such as training data and model parameters, etc. Although researchers have studied, in-depth, several kinds of attacks in NLP models, they are non-systematic analyses. It lacks a comprehensive understanding of the impact caused by the attacks. For example, we must consider which scenarios can apply to which attacks, what the common factors are that affect the performance of different attacks, the nature of the relationships between different attacks, and the influence of various datasets and models on the effectiveness of the attacks, etc. Therefore, we need a benchmark to holistically assess the privacy risks faced by NLP models. In this paper, we present a privacy attack and defense evaluation benchmark in the field of NLP, which includes the conventional/small models and large language models (LLMs). This benchmark supports a variety of models, datasets, and protocols, along with standardized modules for comprehensive evaluation of attacks and defense strategies. Based on the above framework, we present a study on the association between auxiliary data from different domains and the strength of privacy attacks. And we provide an improved attack method in this scenario with the help of Knowledge Distillation (KD). Furthermore, we propose a chained framework for privacy attacks. Allowing a practitioner to chain multiple attacks to achieve a higher-level attack objective. Based on this, we provide some defense and enhanced attack strategies. The code for reproducing the results can be found at https://anonymous.4open.science/r/nlp_doctor-AF48

## introduction in the past few decades, research in the field of nlp-based machine learning, especially deep * these authors contributed equally to this work † corresponding author @xcite . attacks using auxiliary data from different domains: to fit the real scenarios (attackers may not obtain data with the same distribution as the auxiliary data), we conduct extensive experiments with data from different domains. from the experimental results (see appendix e), we find that there are some cases where the attack performance is very low. through our analysis, we conclude that the main reason for the poor performance of the attack is that the data distribution from different domains may be far from the original data distribution. we use kd to mitigate this issue. a chained framework for different attacks: from an attack perspective, we hope to enable attackers to chain multiple attacks to achieve a higherlevel attack objective. secondly, we aim to explore whether one attack can influence the capability of another, in order to clarify the sequential relationship among different attacks, the inherent correlational characteristics among them, and the possibility of linking different attacks together. based on the considerations mentioned above, we propose a chained framework for different attacks. from this, we provide some defense and enhanced attack methods: a defense method against mia, a data-free mia, and some strategies to improve the success rate of aia, mdia, and mea. the main contributions of this paper are as fol-lows: in order to promote development and practical deployment of nlp privacy evaluation, we present a privacy evaluation benchmark for nlp models, which includes the conventional nlp models and llms. this paper experimentally analyzes the relationship between the data from different domains and attack performance. we use kd to address the problem of the low success rate of mia under different domains, and also propose a chained framework for privacy attacks. it is mainly to enable a practitioner to chain multiple attacks to achieve a higher-level objective or explore the inherent correlational characteristics among attacks. ## the chained framework most work on privacy attacks focused on a particular type of attack. whether combining different attacks may yield different results is still an open problem. we propose a chained framework to enable a practitioner to chain multiple attacks to achieve a higher-level objective or explore the inherent correlational characteristics between attacks. figure 3 shows the proposed chained framework. mea chained with aia/mia/mdia: figure 4 illustrates the flowchart of the method for chaining model extraction attack (mea) with other attacks. the initial step in this integration process involves utilizing shadow training data to query the target model, to train the extraction model. the subsequent steps are as follows. @xmath0: the attacker uses shadow testing data to query the extraction model, which is then employed to train the attribute attack model. finally, the attacker utilizes the target training data for inference. since the attacker can access all parameters of the extracted model, they can deploy a white-box aia method based on this model. @xmath1: after the model owner publishes the extracted model, the attacker can proceed with the standard mia procedure to conduct the attack test. since meas leverage d sha (shadow data) to obtain the extraction model, and d sha does not belong to the target data, the extraction model can simulate the performance of the target model without inheriting the target model's training data. so, we consider that mea can act as a defense strategy against mia. @xmath2: this method employs the extracted model as a pseudo-target model for mdias to generate raw data. the purpose is to assess whether the extracted model retains any memory of the original training data. mdia chained with mia/aia/mea: figure 5 depicts the integration of mdia with the remaining attacks. given that the mdia is a data-free reconstruction method, it is anticipated that the derived method can serve as a data generation strategy. this generated data can effectively supplant the assumption that the attacker possesses auxiliary data, thereby facilitating data-free attacks. the steps involved are @xmath3, i.e., the attacker utilizes the data generated by the mdia as auxiliary data to conduct the mia, aia, or mea. mia chained with mea/mdia: figure 6 illustrates chaining mia with mea or mdia. mia can identify whether a data point is part of the training set, thus it can serve as a data filter. the initial step involves training both the shadow model and the attack model. the subsequent steps include @xmath4. in the @xmath5, the shadow data is first processed by mia to obtain the logits from the attack model, then the logits' member dimensions are ranked by score magnitude, and finally, the data is selected based on these scores proportionally. in @xmath6, the attacker generates data using the mdia, after which the generated data are subjected to the mia testing process. the generated text is accepted only if the attack model classifies it as a member of the training data set. note that the attribute is just an implicit characteristic of a sample. it is irrelevant to the original task when training the target model, and it is just unconsciously remembered by the model. obviously, in the nlp domain, an attacker who has determined a text attribute could do little to affect the performance of the other attacks. ## privack attacks for llm in this study, we focus on four llm privacy attacks, i.e., lmia, lmdia, laia, and lmea. detailed attack specifics are provided in the appendix c. attack methods are summarized in membership inference attack for llm (lmia): based on different threat assumptions, we introduce three types of lmias. the first and second types require shadow data to train a reference model (b box /d sha ) @xcite . the third type (b box /d dif f ) @xcite considers more realistic scenarios where the attacker can only obtain data from different domains. detailed descriptions are included in appendix c.1. model inversion attack for llm (lmdia): we introduce two approaches for the black-box lmdia(d no or d dif f ) @xcite . this paper proposes a language-based lmdia that employs text generation and perplexity (ppl) sorting to extract batches of training text. detailed descriptions are included in appendix c.2. attribute inference attack for llm (laia): we integrated two types of laia b box /(d no or d par ) @xcite . this enables the llm to extract attributes from the texts, including gender, address, occupation, and so on. for more detailed results, refer to appendix c.3. model extraction attack for llm(lmea): this paper employs two attacks suitable for blackbox scenarios (d sha or d par ) @xcite . it mainly involves using an llm to annotate unlabeled data. detailed descriptions are included in appendix c.4. ## results of attacks and defenses in this section, we describe experimental data, the model, and the results of experiments on nlp privacy attacks and defenses. due to space constraints, we have presented only the most core experimental results in the main text, including privacy attack outcomes for both largescale and traditional models as well as results from the chained framework. we have placed the performance of using data from different domains (see appendix e), the experimental results of kd (see appendix e), and the defense structures (see appendix f) in the appendix. ## conclusion in this paper, we develop a privacy attack and defense system for nlp models (the conventional/small models and llms). to be more realistic, we have done extensive experiments with auxiliary data from different domains. we further use kd to mitigate the poor performance of mias. on the other hand, we propose a chained framework to enable a practitioner to chain multiple attacks to achieve a higher-level objective. in real-world applications, our system can do a comprehensive privacy evaluation for nlp models to enable users to fully understand the extent of the model's leak privacy before it is deployed.