Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory (2025)

Abstract: Privacy research has attracted wide attention as individuals worry that their private data can be easily leaked during interactions with smart devices, social platforms, and AI applications. Existing works mostly consider privacy attacks and defenses on various sub-fields. Within each field, various privacy attacks and defenses are studied to address patterns of personally identifiable information (PII). In this paper, we argue that privacy is not solely about PII patterns. We ground on the Contextual Integrity (CI) theory which posits that people’s perceptions of privacy are highly correlated with the corresponding social context. Based on such an assumption, we formulate privacy as a reasoning problem rather than naive PII matching. We develop the first comprehensive checklist that covers social identities, private attributes, and existing privacy regulations. Unlike prior works on CI that either cover limited expert annotated norms or model incomplete social context, our proposed privacy checklist uses the whole Health Insurance Portability and Accountability Act of 1996 (HIPAA) as an example, to show that we can resort to large language models (LLMs) to completely cover the HIPAA’s regulations. Additionally, our checklist also gathers expert annotations across multiple ontologies to determine private information including but not limited to PII. We use our preliminary results on the HIPAA to shed light on future context-centric privacy research to cover more privacy regulations, social norms and standards. We will release the reproducible code and data.

## introduction as machine learning models and their applications achieve wide accessibility and applicability, they revolutionize our daily lives. on the one hand, these data-driven models achieve consistent utility improvements with increasing training corpus. on the other hand, private data may be collected unintentionally, which leads to individuals' increasing * haoran li and wei fan contributed equally. privacy concerns. to achieve trustworthy ai, great efforts have been made to study privacy issues with regard to emerging applications @xcite . existing privacy studies conducted by researchers are relatively fragmented. each field has its own privacy formulation based on specific tasks. for example, in computer networks, network communications' confidentiality is widely discussed @xcite and cryptographic methods are proposed to protect cybersecurity @xcite . when it comes to deep learning, in both computer vision and natural language processing fields, there is a significant body of research on privacy attacks @xcite @xcite and defenses @xcite . in addition to the fragmented formulations, the current scope of privacy research is narrow. most existing privacy studies focus on protecting private attributes specified by privacy regulations. though these sensitive attributes are indeed crucial components of privacy regulations, they only constitute a small fraction of regulations' coverage. however, privacy is not solely about protecting attributes. instead, it involves understanding the context to ensure compliance with existing regulations and norms. for example, according to the health insurance portability and accountability act of 1996 (hipaa), protected health information (phi) is considered sensitive and should be protected. however, as shown in figure 1 , though bob's surgery operative report is classified as protected health information, if we identify alice as a covered entity and bob as the individual (patient), we may apply hipaa 164.502(a)(1)(i) to permit surgeon alice to share bob's phi with him. this example demonstrates that privacy issues can be resolved through reasoning within the context. when we consider privacy, we actually evaluate whether our information transmission contexts violate existing standards and expectations. to bridge the gap, contextual integrity (ci) theory @xcite is proposed to examine the appropriateness of privacy information flows based on specific contexts. a few prior works on ci resort to formal languages such as first-order logic @xcite @xcite to fit contexts into pre-defined clauses to determine privacy violations. however, it requires heavy effort for experts to annotate complicated legal documents into logical language. consequently, none of these approaches can scale up to cover even a whole document. in this paper, to leverage the reasoning capability of privacy information flows while allowing substantial flexibility to handle natural language ambiguity and variability, we present privacy checklist. we formulate the privacy evaluation as an in-context reasoning problem with existing large language models (llms). instead of annotating separate clauses, our privacy checklist uses large language models to extract ci characteristics and use the tree structure to capture their hierarchical nature. in addition, our privacy checklist also includes role-based and attribute-based graphs, as well as a definition dictionary to facilitate incontext reasoning. to determine privacy violations for the given context, we first fetch relevant regulations and use large language models to conduct in-context reasoning with retrieved regulation. we annotate all information transmission norms inside the hipaa and show that our privacy checklist can improve 6% -18% accuracy on average for existing llms to improve their privacy judgment ability in real court cases. our contribution can be summarized as follows foot_0 : ## preliminaries contextual integrity, norms and applications. contextual integrity @xcite posits that privacy is related to information flows and information flows must adhere to the established norms of the context to protect privacy. each norm can typically be viewed as an atomic unit of information transmission involving three key actors: the sender, the receiver, and the data subject. depending on the social context of these actors, such as their roles and the disclosed attributes of the data subject, we can determine whether the norm is positive or negative, which means the norm is either permitted or prohibited by existing privacy regulations. previous works on ci @xcite @xcite aimed to transform the context into explicit formal languages, such as first-order logic, to determine privacy violations with rule-based frameworks. however, as shown in figure 2 (a), it costs heavily for experts to annotate such a knowledge base of norms. experts must meticulously define a set of predicates and annotate the entire body of regulations, subpart by subpart. consequently, these works on formal languages fail to scale up their norms due to the inherent complexity of legal documents. on the other hand, existing legal llms, including @xcite , lawyer llama @xcite and chat-law @xcite are proposed on the general legal domain and underperform on the legal domain due to data scarcity. goldcoin @xcite proposed instruction tuning grounded on ci to en-hance llms' judgment ability. instead, our work reveals the flaws of synthetic data and shows that the proper rag pipelines without further tuning can yield comparable or even better performance. in addition, for discussions about current privacy attacks and defenses, please refer to appendix c. ## privacy checklist construction in this section, we show how to construct our privacy checklist based on existing privacy regulations with less annotation cost. for the given regulation, our privacy checklist encompasses a document tree with ci characteristics, a role graph, an attribute graph, and a definition dictionary. ## enhance reasoning with checklist in this section, we show how to exploit llms' in-context reasoning ability to determine privacy violations. as shown in figure 2 (b), the basic intuition is to use existing llms for a retrievalaugmented generation (rag) pipeline. ## experimental setups data. we use collected real and synthetic court cases for evaluation to demonstrate the usefulness of our privacy checklist. following fan et al. evaluated llms. we evaluate both open-source and close-source llms. in terms of opensource llms, we download their model weights of instruction or chat versions and generate responses on two nvidia tesla a100 40gb graphic cards. our evaluated open-source models include llama3 (ai@meta, 2024), qwen1.5, qwen2 @xcite , chatglm4 @xcite , and mistral-v0. @xcite . for close-source llm, we evaluate the gpt-4-turbo-04-09's performance with api accesses. ## conclusions and future works in this paper, we follow the spirit of contextual integrity theory to transform privacy-related issues into reasoning problems. unlike prior attempts which manage to build rigorous expert systems based on logic languages, we exploit powerful llms to further formulate the reasoning problems to the in-context reasoning problems based on the retrieval-augmented generation (rag) pipeline. to facilitate the retrieval process, we propose privacy checklist, a checklist that covers processed legal documents, annotated ci characteristics, and auxiliary knowledge about roles, private attributes and term definitions. we compare three rag-based prompts with three baseline prompts to show that our proposed privacy checklist can improve llms' privacy awareness for real and synthetic court cases. our future works can be divided into two folds. first, we plan to integrate most mainstream privacy regulations as well as ai safety standards into our privacy checklist. second, we will work on improving the retrieval performance to eliminate the gap between common sense knowledge and legal terminologies. after getting over the two parts, we may no longer need the privacy-utility trade-off for model training and inference. instead, we may simply apply our checklist as a detector for safeguarding models' inputs and outputs and concentrate on models' utility improvements.