Paper: Can LLMs Keep a Secret? Testing  Privacy  Implications of Language Models  via Contextual Integrity Theory (ICLR-2024)

Reviewer comment (Theory):
The concept of contextual privacy, as the name indicates, heavily depends on the context.

Author response:
Thank you for the positive endorsement on this new direction of investigating contextual privacy implications in LLMs.
&nbsp;

### Scaling of the benchmark
Our benchmark could be expanded in size by adding more variables to the factorial vignettes (i.e. the underlying templates) for each sample. For example, the set of information types in Tier 2 can be expanded to include social media handles or zip code. Theory of mind scenarios can also be expanded by populating causal templates (Gandhi et al., 2023), and expanding on the contextual factors. However, we are confident that **the core message of our paper regarding the importance of context when dealing with privacy in language will very likely not change**, no matter how many new scenarios are added to the benchmark. We will add this discussion to our paper.

Reviewer comment (Experiments):
The benchmark can only capture a small portion of possible contexts so it's not very scalable.

Author response:
Thank you for the positive endorsement on this new direction of investigating contextual privacy implications in LLMs.
&nbsp;

### Scaling of the benchmark
Our benchmark could be expanded in size by adding more variables to the factorial vignettes (i.e. the underlying templates) for each sample. For example, the set of information types in Tier 2 can be expanded to include social media handles or zip code. Theory of mind scenarios can also be expanded by populating causal templates (Gandhi et al., 2023), and expanding on the contextual factors. However, we are confident that **the core message of our paper regarding the importance of context when dealing with privacy in language will very likely not change**, no matter how many new scenarios are added to the benchmark. We will add this discussion to our paper.

Reviewer comment (Experiments):
Is there a way to construct scalable benchmark?

Author response:
Thank you for the positive endorsement on this new direction of investigating contextual privacy implications in LLMs.
&nbsp;

### Scaling of the benchmark
Our benchmark could be expanded in size by adding more variables to the factorial vignettes (i.e. the underlying templates) for each sample. For example, the set of information types in Tier 2 can be expanded to include social media handles or zip code. Theory of mind scenarios can also be expanded by populating causal templates (Gandhi et al., 2023), and expanding on the contextual factors. However, we are confident that **the core message of our paper regarding the importance of context when dealing with privacy in language will very likely not change**, no matter how many new scenarios are added to the benchmark. We will add this discussion to our paper.

Reviewer comment (Evaluation):
Some of the metric definitions seem to be lacking in the main paper, making results hard to interpret, e.g., the sensitivity score in Table 2, the metric of Fig. 2 isn’t named, Table 4 includes five undefined metrics. This is all the more important for figures such as Fig. 2 which are very complex and seem to be lacking a clear trend.

Author response:
We thank the reviewer for raising this issue, we agree that including this information will improve the readability of the paper. We will include the following table in the revision of the paper to clarify and categorize all of the metrics and the values they take. Below is a summary of this table:

| Tier | Metric Name | Metric Type | Range of Value | Interpretation | Figures and Tables |
| --- | --- | --- | --- | --- | --- |
| 1, 2.a and 2.b | Sensitivity Score | Score | [-100,100] | Lower means more sensitive | Figure 2, Table 2 |
| 3 & 4 | Leakage | Rate | [0,1] | Lower means less leakage (better) | Tables 4&5 and Figures 3&4 |
| 3 | Information Accessibility Error | Rate | [0,1] | Lower means more accurate (better) | Table 4 |
| 3 | Binary Control Question Error | Rate | [0,1] | Lower means more accurate (better) | Table 4 |
| 4 | Omits Public Information (Utility) | Rate | [0,1] | Lower means more accurate (better task utility) | Table 5 and Figure 4 |

Reviewer comment (Evaluation):
No error rate is given for results derived from automated parsing of LLM responses. More specifically, automated methods like string matching or LLM interpretation of results may incorrectly determine whether a secret was leaked. What is the error rate of the automated method for parsing of LLM responses? This can be estimated by randomly sampling some of the responses and checking how often the automated method correctly predicts whether the secret was leaked. This should give some notion of confidence in the results.

Author response:
Following your suggestion, we conducted a thorough manual review of all 200 GPT-4 responses for the action item generation task in Tier 4. We discovered 16 instances where private information was leaked, yet not detected by exact string matching. For instance, the private information was "move to VISTA," but the response mentioned "transition to the VISTA project". This indicates overestimation of the models’ performance in Tier 4. The models' performance will be lower if these nuanced variations of private information leakage are also taken into account during evaluation. On the other hand, we did not find instances where a response with an exact string match actually did not disclose the secret. We will add this discussion in the revised draft.

Reviewer comment (Evaluation):
Since part of the benchmark is generated by LLMs (e.g., Tier 2 and 4 tasks use GPT-4) and then GPT-4 is evaluated using the benchmark, can this bias the findings on GPT-4? E.g., is it possible for GPT-4 to be more “familiar” with the wording produced by itself and somehow be at an advantage compared to the other models? The use of GPT-4 for generating the tasks should be motivated and the limitations of this be acknowledged.

Author response:
We would like to thank the reviewer for raising this very interesting issue, and want to clarify that the scenarios of Tier 3 and Tier 4 are generated by GPT-4, however, they have received human intervention to make sure the stories are sound and serve the purpose of the tier. To address the valid issue raised by the reviewer, we re-generated the Tier 4 scenarios with ChatGPT, and evaluated GPT-4 and ChatGPT on it  and compared the results with those from GPT-4 generated Tier 4. You can see the results below.

| ChatGPT results | Action Item | | Meeting Summary | |
| --- | --- | --- | --- | --- |
| | Mean | Worst | Mean | Worst |
| GPT-4-generated | 35.0 | 70.0 | 65.5 | 95.0 |
| ChatGPT-generated | 37.5 | 85.0 | 53.5 | 85.0 |


| GPT-4 results | Action Item | | Meeting Summary | |
| --- | --- | --- | --- | --- |
| | Mean | Worst | Mean | Worst |
| GPT-4 generated | 32.5 | 75.0 | 38.0 | 90.0 |
| ChatGPT-generated | 28.0 | 65.0 | 38.0 | 85.0 |

&nbsp;

GPT-4 still outperforms ChatGPT with a significant margin on the ChatGPT-generated Tier 4. Moreover, GPT-4’s scores improved in comparison to its results on the GPT-4-generated Tier 4. Conversely, ChatGPT’s performance change was mixed. It showed improvement in the action item generation task, but showed a decline in its performance on the meeting summary task. To summarize these results, our **findings that GPT-4 outperforms ChatGPT in terms of privacy leakage still holds, even in the case where the meeting script is not generated by GPT-4.**

We will also add additional discussion on the limitations of using LLMs for synthesizing evaluation data, and include these results in the appendix of the paper.

Reviewer comment (Experiments):
The limitations stemming from using human annotators of Mechanical Turk for deciding what is private and what isn’t aren't acknowledged. Do the authors know the background of the annotators and do they believe this may bias the results in specific ways?

Author response:
We unfortunately did not collect background information on the human annotators on MTurk, however, the 2016 law study by Martin & Nissenbaum (which we use as backbone for Tiers 1 and 2 and show to have high correlation with our annotations), collected the Turkers’ gender, age and privacy concerned-ness (the ‘westin’ privacy categorization which determines how much a person cares about privacy in general). Their study show that *factors such as gender, age and privacy-categorization (i.e. privacy concernedness) of the annotators has no statistically significant correlation with their privacy preferences*. Additionally, they show that contextual factors in the vignettes (the actor, information type and use-case) can better explain the privacy preferences (pages 211-212 of the study). We will add this to the discussion section and thank the reviewer for raising this important issue.

Reviewer comment (Experiments):
Please include statistics of the benchmark such as how many examples are generated for each task, how many of them are human-generated vs LLM-generated.

Author response:
We included the number of examples for each tier in Section 3. However, we will also include the following table in the revision of the paper for better readability.


| | Tier 1 | Tier 2 | Tier 3 | Tier 4 |
| --- | --- | --- | --- | --- |
| Total number of samples in the tier | 10 information types | Tier 2.a: 98 vignettes, Tier 2.b: 98 vignettes | 270 scenarios | 20 transcripts |
| Source | Maden (2014) | Tier 2.a: Martin & Nissenbaum (2016), Tier 2.b: GPT-4 generated text | GPT-4 generated texts with expert intervention | GPT-4 generated texts with expert intervention |
| Total number of questions | 10 sensitivity questions (multiple choice) | 98 * 2 privacy expectation questions (multiple choice) | 270 Response generation question (free-form), 270 Information accessibility questions (list), 270 Private information sharing understanding question (list), 270 control questions (binary) | 20 Action item generation questions(free-form), 20 Meeting summary generation question (free-form) |

Reviewer comment (Experiments):
To facilitate the interpretation of results, I suggest to include more context about LLMs being evaluated. Some statements are made such as “models that have undergone heavy RLHF training and instruction tuning (e.g. GPT-4 and ChatGPT)” and “Overall, we find the leakage is alarmingly high for the open source models, and even for ChatGPT” without it being clear which LLMs are commercial, open-source, and trained using RLHF.

Author response:
Thank you for the suggestion. We will update them in the revision.

Reviewer comment (Experiments):
The experiment design is questionable. The authors find that “GPT-4 and ChatGPT reveal secrets 22% and 93% of the time in Tier 3, and flow information to inappropriate actors 39% and 57% of the time in Tier 4, even though they are directly instructed to preserve privacy.”

Author response:
### 3 & 4. LLMs understanding of privacy and their performance on our benchmark
Answering the question of how LLMs perceive privacy is the overarching theme of the paper. Privacy is not 0-1 nor clean-cut, as explained in depth by Brown et al. 2022 (mentioned in the paper as well), there are many layers of nuance to what is considered private. That’s why we design a multi-tiered benchmark, to disentangle different levels of ‘understanding privacy’, starting from having the raw knowledge of what sensitivity and privacy are (Tiers 1 and 2), and adding complexity in terms of using and applying the understanding of these concepts (Tiers 3 and 4). 

As the results show, the performance of the different models varies heavily across different tiers of our benchmark. For the open-source models (Llama2 70B , Llama2 70B chat and Flan-UL2), we see the worst performance on all tiers, starting with low agreement with human annotations in tiers 1 and 2 (Table 1). This shows that this group of models is not aligned with human preferences and lacks the knowledge of human expectations of privacy. Moving to tier 3 (Table 4), we can see that the open-source models have high error on the information accessibility questions, showing that they lack theory of mind and cannot keep track of who has access to what information, so that is another cause of their poor performance. This also explains the failures of such models on Tier 4 (Table 5)

The situation is more nuanced for OpenAI models, specially ChatGPT and GPT-4, as the have high agreement (correlation) with the human preferences for the first two tiers, indicating that they have the knowledge of human preferences, however, they cannot reason over it and operationalize it, which is why we see failures in Tiers 3 and 4 for them. We like to emphasize that the failures we observe cannot be boiled down to theory of mind failures, as for GPT-4 we observe in Table 3 that the information accessibility questions and the control questions have very low error rates, demonstrating that the model has kept track of who knows what, but it cannot reason over it and operationalize it at generation time, which is why we see leakage of secrets.

If the reviewer’s question alludes to whether or not the models are able to follow our instructions about privacy, then this falls more under the realm of RLHF and instruction tuning, rather than privacy directly. Also, we follow the methodology that is commonly used in literature for benchmarking LLM capabilities (Bulian et al. 2023, Birkun & Gautam 2023), where we ask the models direct questions about situations and collect their response, to empirically study wether they understand different concepts. 

Finally, regarding your question 3.b, Since only X and Y shared the information and Z did not, the models should not mention Z's name when they are asked to list the characters who shared the private information. We will clarify this in the revision.

**References**

Bulian, Jannis, et al. "Assessing Large Language Models on Climate Information." arXiv preprint arXiv:2310.02932 (2023).

 Alexei A Birkun and Adhish Gautam. 2023. Large language model-based chatbot as a source of advice on first aid in heart attack. Current Problems in Cardiology (2023).

Reviewer comment (Theory):
First, how to ensure that LLMs really understand the concept of “privacy”? Does LLMs understand the concept of “privacy” in the same way of humans? Does “privacy” indicate “cannot be shared” for LLMs in nature?

Author response:
### 3 & 4. LLMs understanding of privacy and their performance on our benchmark
Answering the question of how LLMs perceive privacy is the overarching theme of the paper. Privacy is not 0-1 nor clean-cut, as explained in depth by Brown et al. 2022 (mentioned in the paper as well), there are many layers of nuance to what is considered private. That’s why we design a multi-tiered benchmark, to disentangle different levels of ‘understanding privacy’, starting from having the raw knowledge of what sensitivity and privacy are (Tiers 1 and 2), and adding complexity in terms of using and applying the understanding of these concepts (Tiers 3 and 4). 

As the results show, the performance of the different models varies heavily across different tiers of our benchmark. For the open-source models (Llama2 70B , Llama2 70B chat and Flan-UL2), we see the worst performance on all tiers, starting with low agreement with human annotations in tiers 1 and 2 (Table 1). This shows that this group of models is not aligned with human preferences and lacks the knowledge of human expectations of privacy. Moving to tier 3 (Table 4), we can see that the open-source models have high error on the information accessibility questions, showing that they lack theory of mind and cannot keep track of who has access to what information, so that is another cause of their poor performance. This also explains the failures of such models on Tier 4 (Table 5)

The situation is more nuanced for OpenAI models, specially ChatGPT and GPT-4, as the have high agreement (correlation) with the human preferences for the first two tiers, indicating that they have the knowledge of human preferences, however, they cannot reason over it and operationalize it, which is why we see failures in Tiers 3 and 4 for them. We like to emphasize that the failures we observe cannot be boiled down to theory of mind failures, as for GPT-4 we observe in Table 3 that the information accessibility questions and the control questions have very low error rates, demonstrating that the model has kept track of who knows what, but it cannot reason over it and operationalize it at generation time, which is why we see leakage of secrets.

If the reviewer’s question alludes to whether or not the models are able to follow our instructions about privacy, then this falls more under the realm of RLHF and instruction tuning, rather than privacy directly. Also, we follow the methodology that is commonly used in literature for benchmarking LLM capabilities (Bulian et al. 2023, Birkun & Gautam 2023), where we ask the models direct questions about situations and collect their response, to empirically study wether they understand different concepts. 

Finally, regarding your question 3.b, Since only X and Y shared the information and Z did not, the models should not mention Z's name when they are asked to list the characters who shared the private information. We will clarify this in the revision.

**References**

Bulian, Jannis, et al. "Assessing Large Language Models on Climate Information." arXiv preprint arXiv:2310.02932 (2023).

 Alexei A Birkun and Adhish Gautam. 2023. Large language model-based chatbot as a source of advice on first aid in heart attack. Current Problems in Cardiology (2023).

Reviewer comment (Evaluation):
It is not well explained what causes the bad performance of LLMs. Does it mean that LLMs do not have theory-of-mind? Or LLMs potentially just do not understand the setting well?

Author response:
### 3 & 4. LLMs understanding of privacy and their performance on our benchmark
Answering the question of how LLMs perceive privacy is the overarching theme of the paper. Privacy is not 0-1 nor clean-cut, as explained in depth by Brown et al. 2022 (mentioned in the paper as well), there are many layers of nuance to what is considered private. That’s why we design a multi-tiered benchmark, to disentangle different levels of ‘understanding privacy’, starting from having the raw knowledge of what sensitivity and privacy are (Tiers 1 and 2), and adding complexity in terms of using and applying the understanding of these concepts (Tiers 3 and 4). 

As the results show, the performance of the different models varies heavily across different tiers of our benchmark. For the open-source models (Llama2 70B , Llama2 70B chat and Flan-UL2), we see the worst performance on all tiers, starting with low agreement with human annotations in tiers 1 and 2 (Table 1). This shows that this group of models is not aligned with human preferences and lacks the knowledge of human expectations of privacy. Moving to tier 3 (Table 4), we can see that the open-source models have high error on the information accessibility questions, showing that they lack theory of mind and cannot keep track of who has access to what information, so that is another cause of their poor performance. This also explains the failures of such models on Tier 4 (Table 5)

The situation is more nuanced for OpenAI models, specially ChatGPT and GPT-4, as the have high agreement (correlation) with the human preferences for the first two tiers, indicating that they have the knowledge of human preferences, however, they cannot reason over it and operationalize it, which is why we see failures in Tiers 3 and 4 for them. We like to emphasize that the failures we observe cannot be boiled down to theory of mind failures, as for GPT-4 we observe in Table 3 that the information accessibility questions and the control questions have very low error rates, demonstrating that the model has kept track of who knows what, but it cannot reason over it and operationalize it at generation time, which is why we see leakage of secrets.

If the reviewer’s question alludes to whether or not the models are able to follow our instructions about privacy, then this falls more under the realm of RLHF and instruction tuning, rather than privacy directly. Also, we follow the methodology that is commonly used in literature for benchmarking LLM capabilities (Bulian et al. 2023, Birkun & Gautam 2023), where we ask the models direct questions about situations and collect their response, to empirically study wether they understand different concepts. 

Finally, regarding your question 3.b, Since only X and Y shared the information and Z did not, the models should not mention Z's name when they are asked to list the characters who shared the private information. We will clarify this in the revision.

**References**

Bulian, Jannis, et al. "Assessing Large Language Models on Climate Information." arXiv preprint arXiv:2310.02932 (2023).

 Alexei A Birkun and Adhish Gautam. 2023. Large language model-based chatbot as a source of advice on first aid in heart attack. Current Problems in Cardiology (2023).

Reviewer comment (Novelty):
The significance of this study is not very clear. It is not convinced that "contextual privacy” is a very important problem for LLMs because this paper does not show convincing real-world examples to illustrate the importance of LLMs' "contextual privacy” property. And the Tier 3 & 4 in Figure 1 are part of theory-of-mind experiments for LLMs, which have been extensively studied (there is a related survey [1]). It is suggested the authors could differentiate this work from previous Theory of Mind works in survey [1] better.

Author response:
### 5. Why contextual privacy is important especially in real-world applications
The framework of contextual integrity was first introduced in 2004 and further developed in the 2009 book ‘privacy in context’, and its importance is underlined by its broad application and adoption, as there is thousands of work building on it, and an annual symposium on privacy through contextual integrity (https://privaci.info/). 

As for the importance of contextual integrity in language, there is several prior work discussing the importance of context in privacy for language (Shvartzshnaider et al. 2019 and shi et al. 2021), nominally the work by Brown et al. (2022) which accentuates the nuances and complexities of privacy in language.
The aim of our work is to fill this gap that has been pointed out by prior work multiple times especially as more LLMs are getting deployed in real-world applications. That’s the reason why we designed Tier 4, where we focus on real-world applications of LLMs, such as meeting summary generation and personal action item generation, similar to the services deployed in Microsoft Teams  (Ajam 2023).

&nbsp;

### 6. The difference between existing works in machine theory of mind and ours
Our work is the first to explore the intersection of privacy and theory of mind in AI. While there are many existing studies on machine theory of mind, as detailed in the most recent survey paper, none of them have studied the relationship between privacy and theory of mind. 

Also, our Tier 3 and 4 are not subsets of theory of mind experiments for LLMs. Instead, theory of mind is a distinct component of those tasks. In this context, theory of mind refers to the capability to infer and reason about mental states in scenarios where information is asymmetric. The goal of Tier 3 and 4 is to investigate privacy implications (e.g., controlling information flow) of LLMs, which requires this capability to understand and assess who knows what.

Reviewer comment (Presentation):
It is suggested the authors could explicitly summarize their contributions in the introduction.

Author response:
### 7. Summarization of our contribution
Our contributions can be summarized as the following:
We operationalize the notion of contextual integrity to fit the use-cases of large language models.
We introduce a multi-tiered benchmark building on this notion, where we study inference-time privacy in interactive scenarios, which has not been done before.
Our benchmark evaluates different nuances of privacy, on a wide range of models, including knowledge of sensitivity in different contexts, reasoning and theory of mind.

We will add this to the introduction in the revision of the paper.

Reviewer comment (Writing):
For tiers 1 and 2, we find our results to be closely aligned with the initial results of Martin & Nissenbaum (2016), demonstrating a correlation of 0.85, overall --> correlation between what?

Author response:
The correlation of 0.85 is observed between the original human annotations of Martin & Nissenbaum (2016), which was conducted using Mechanical Turk as well, and the human annotations collected for our paper. We include this metric to show there is consensus between our findings and that of prior work (in technology law), for human preferences. We will clarify this in the updated draft.