Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks (2023)

Abstract: Large language models have revolutionized the field of NLP by achieving state-of-the-art performance on various tasks. However, there is a concern that these models may disclose information in the training data. In this study, we focus on the summarization task and investigate the membership inference (MI) attack: given a sample and black-box access to a model’s API, it is possible to determine if the sample was part of the training data. We exploit text similarity and the model’s resistance to document modifications as potential MI signals and evaluate their effectiveness on widely used datasets. Our results demonstrate that summarization models are at risk of exposing data membership, even in cases where the reference summary is not available. Furthermore, we discuss several safeguards for training summarization models to protect against MI attacks and discuss the inherent trade-off between privacy and utility.

## introduction text summarization seeks to condense input document(s) into a shorter, more concise version while preserving important information. the recent large language models have significantly enhanced the quality of the generated summaries @xcite @xcite . these models have been applied to many sensitive data, such as clinical and finance reports @xcite . given these high-stakes applications, it is critical to guarantee that such models do not inadvertently disclose any information from the training data and that data remains visible only to the client who owns it. to evaluate the potential memorization of specific data by a model, membership inference (mi) attack @xcite has become the de facto standard, owing to its simplicity @xcite . given a model and sample (input label pair), the membership inference attack wants to identify whether this sample was in the model's training dataset. this problem can be formulated as an adversarial scenario, with bob acting as the attacker and alice as the defender. bob proposes methods to infer the membership, while alice attempts to make membership indistinguishable. in the past, researchers proposed various attacking and defending techniques. however, most of them focus on the computer vision classification problem with a fixed set of labels. little attention has been given to comprehending mi attacks in seq2seq models. this paper focuses on the summarization task and investigates the privacy risk under membership inference attacks. inspired by previous research in mi literature @xcite , we pose the problem as follows: given blackbox access to a summarization model's api, can we identify whether a document-summary pair was used to train the model?. compared to membership inference attacks on fixed-label classification tasks, text generation tasks present two significant challenges: (1) the process of generating summaries involves a sequence of classification predictions with variable lengths, resulting in complex output space. a pertinent question emerging from the study is the rationale behind the efficacy of mi attacks on summarization models. the key insight lies in the training objective of these models, which aims to minimize the discrepancy between the generated and reference summary @xcite . consequently, samples with significantly lower loss (indicating a similarity between the generated and reference summaries) are more likely to be part of the training dataset. based on this concept, we propose a baseline attack method that utilizes the similarity between the generated and reference summaries to differentiate between training and non-training samples. one limitation of the baseline attack is that bob requires access to both the documents and reference summaries to launch the attack, rendering the attack less practical for summarization tasks. building upon this, our study introduces a more general document-only attack: given only a document and black-box access to the target model's apis, can bob infer the membership without access to the reference summary? we tackle this problem by examining the robustness of generated summaries in response to perturbations in the input documents. according to the max-margin principle, training data tends to reside further away from the decision boundary, thus exhibiting greater resilience to perturbations, which aligns with observations from prior research in the adversarial domain @xcite . consequently, bob can extract fine-grained membership inference signals by evaluating data robustness toward perturbations. remarkably, we show that bob can estimate the robustness without reference summaries, thereby enabling a documentonly attack. in summary, this work makes the following contributions to the language model privacy: 1. defined the black-box mi attack for the sequence-to-sequence model. experiments on summarization tasks show attackers can reliably infer the membership for specific instances. 2. explored data robustness for mi attacks and found that the proposed approach enables attackers to launch the attack solely with the input document. 3. evaluate factors impacting mi attacks, such as dataset size, model architectures, etc. we also explore multiple defense techniques and discuss the privacy-utility trade-off. ## experiment results baseline attack. we present the results of the baseline attack in however, the results indicate that raw text is inferior to the similarity score features, with the mlp model using similarity scores as features achieving the best performance. in addition to the auc and acc scores, we also evaluate the performance of the attacks in the highconfidence regime. specifically, we report the true positive rate under a low false positive rate of 0.1%, referred to as tpr 0.1% in the table. our results demonstrate that the model can reliably identify samples with high confidence. for example, the mlp model achieves a tpr fpr 0.1% of 3.05% on the mimic dataset, which means that the model successfully detects 305 samples in a in with only 10 false positives in a out . ## ablation studies in this section, we will investigate several impact factors in mi attacks. all experiments were conducted using the baseline attack with the mlp classifier. a more detailed analysis is in the appendix. impact of overfitting. in figure 5 , we show the attack auc and validation rouge-l f1 score under varying training steps on the samsum dataset. we find that the attack auc increases steadily as the number of alice's training steps increases, which is consistent with previous research @xcite . moreover, early stopping by rouge score (5 epochs) cannot alleviate the attack. the auc curve indicates that the model gets a high attack auc at this checkpoint. a better early stop point is 3 epochs, which significantly reduces the mi attack auc without a substantial performance drop. however, in practice, it is hard to select a proper point without relying on an attack model. impact of dataset size. in this study, we assess the impact of dataset size on mi attacks. to do this, we train our model with 10% to 100% of the total dataset. our results, as depicted in figure 6 , indicates that as the size of the training set increases, the auc of mi attacks decreases monotonically for both samsum and cnndm dataset. this suggests that increasing the number of samples in the training set can help to alleviate overfitting and reduce the mi attack auc. some recent studies have highlighted the issue of duplicate training samples in large datasets @xcite . this duplication can escalate the privacy risks associated with these samples and should be taken into consideration when employing large datasets. ## defense methods we now investigate some approaches that aim to limit the model's ability to memorize its training data. specifically, we try two approaches: differential privacy sgd (dp-sgd) foot_2 @xcite @xcite and @xcite . for dp-sgd, ϵ is the privacy budget, where a lower ϵ indicates higher privacy. we conduct experiments on the samsum dataset. as shown in ## limitations in this work, we demonstrate that the mi attack is effective. however, it remains unclear what properties make samples more susceptible to mi attack. in other words, given a model and a dataset, we cannot predict which samples are more likely to be memorized by the model. we find that the detected samples under tpr 0.1% have an average shorter reference length, but further research is needed to fully answer this question. additionally, it is important to note that while the mi attack is a commonly used attack, its privacy leakage is limited. other attacks pose a more significant threat in terms of information leakage @xcite . the evaluation of these attacks in summarization tasks should be prioritized in future studies. ## conclusion in this paper, we investigated the membership inference attack for the summarization task and explored two attack features: text similarity and data robustness. experiments show that both features contain fine-grained mi signals. these results reveal the potential privacy risk for the summarization model. in the future, we would like to explore advanced defense methods and alleviate the tradeoff between privacy and utility.