P riv LM -Bench: A Multi-level Privacy Evaluation Benchmark for Language Models (2024)

Abstract: The rapid development of language models (LMs) brings unprecedented accessibility and usage for both models and users. On the one hand, powerful LMs achieve state-of-the-art performance over numerous downstream NLP tasks. On the other hand, more and more attention is paid to unrestricted model accesses that may bring malicious privacy risks of data leakage. To address these issues, many recent works propose privacy-preserving language models (PPLMs) with differential privacy (DP). Unfortunately, different DP implementations make it challenging for a fair comparison among existing PPLMs. In this paper, we present PrivLM-Bench, a multi-perspective privacy evaluation benchmark to empirically and intuitively quantify the privacy leakage of LMs. Instead of only reporting DP parameters, PrivLM-Bench sheds light on the neglected inference data privacy during actual usage. PrivLM-Bench first clearly defines multi-faceted privacy objectives. Then, PrivLM-Bench constructs a unified pipeline to perform private fine-tuning. Lastly, PrivLM-Bench performs existing privacy attacks on LMs with pre-defined privacy objectives as the empirical evaluation results. The empirical attack results are used to fairly and intuitively evaluate the privacy leakage of various PPLMs. We conduct extensive experiments on three datasets of GLUE for mainstream LMs.

## introduction the accelerating evolution of language models (lms) ushers a new era for both modern natural language processing and the whole society. currently, generative large language models (llms) exhibit surprising capability and integrate previous tasks into a unified text generation formulation. as a result, these llms obtain the dominating performance on both expert-designed tasks and realworld problems @xcite @xcite @xcite . moreover, under appropriate instructions, llms can even be in-context learners or zero-shot reasoners to solve unseen tasks @xcite @xcite @xcite . beneath the improved performance, lms' training data also scale up with models' sizes. lms are not only trained on annotated textual data for specific tasks, but also devour a vast amount of textual data online. unlike carefully crowd-sourced annotation data, free-form texts crawled from the internet suffer from poor quality and unintended personal data leakage. for example, simple model interactions can lead to accidental personally identifiable information (pii) dissemination @xcite @xcite . such pii exposure without noticing victims or obtaining victims' consent may violate existing privacy laws like the eu's general data protection regulation (gdpr) and the california consumer privacy act @xcite . to respect data subjects' privacy during model training/fine-tuning, several studies @xcite @xcite consider privacy protection as an additional objective. differential privacy @xcite , known for its wide adaptability and application, has become mainstream for privacy-preserving lms (pplms). dp's definition offers plausible deniability @xcite and introduces bounded privacy parameters to describe the effectiveness of examined mechanisms. this definition can naturally defend against membership inference attack @xcite , which aims to determine if a given sample belongs to the model's training dataset. currently, motivated by dpsgd @xcite , pplms can be achieved via various implementations based on dp optimizers. unfortunately, although numerous implementations of pplms have been proposed, fair evaluations of pplms are still unexplored. existing mainstream approaches simply use dp parameters to quantify pplms' privacy and it is rather hard to make a fair comparison among pplms. firstly, different dp formulations such as central dp, local dp @xcite , and d χ privacy @xcite assign distinct heuristic meanings to dp parameters to implement pplms. secondly, the scope of the protected part is ambiguous. for instance, most pplms implemented from dpsgd offer privacy protection for tuned sensitive data. however, during inference, these pplms are not guaranteed to protect inference data privacy. thus, simply claiming that these pplms are privacy-preserving ignores inference data privacy. lastly, it remains unclear whether dp's worst-case upper bound overestimates privacy leakage. dp assumes a herculean adversary who can manipulate the entire protected dataset, which may be implausible for actual attacks. consequently, evaluation under the same dp parameters may still result in different privacy performance on empirical privacy attacks. to bridge the aforementioned gap, in this work, we propose privlm-bench to fairly quantify pplms' privacy-utility trade-off. privlm-bench adopts the prevailing setup of public pre-training and private fine-tuning with several clarified privacy objectives. privlm-bench incorporates multifaceted privacy attacks, including the data extraction attack @xcite , membership inference attack @xcite and embeddinglevel privacy attack @xcite to evaluate the privacy of pplms. the attacking results can be an intuitive and fair indicator to quantify the privacy leakage of existing pplms regardless of their detailed implementations. in summary, we highlight the following contributions of our proposed privlm-bench: ## experiments for experiments, we first raise a few crucial research questions (rqs) and use our experimental results to address these rqs individually. ## conclusion in this paper, we introduce privlm-bench, a benchmark designed to assess and contrast lms' multifaceted privacy objectives. by integrating a variety of masked and generative lms with diverse tuning algorithms, privlm-bench facilitates an extensive evaluation that encompasses both utility metrics and empirical privacy attacks. our results reveal the effectiveness and limitations of various dp tuning methods. moreover, we point out the reality that empirical privacy attacks demonstrate a less potent threat compared to defense mechanisms' assumed powerful capabilities. in conclusion, our study underscores that privacy evaluation and the balancing act between privacy and utility is a complex, ongoing challenge in the field. for future work, we advocate for more potent privacy attacks and utility-enhanced defense strategies that relax the worst-case restriction in accordance with empirical attacks to improve the privacy utility trade-off.