Paper: DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR-2024)

Reviewer comment (Evaluation):
The threat model is unclear. It would be better and necessary to provide more details and a more specific description on the adversary's goals, capabilities, and knowledge in a threat model. In addition, please provide more description and real-world cases of the consequences of a privacy leakage from the prompt (e.g., how much information can be breached).

Author response:
**A1**: Thanks for the suggestion. We have added details of **threat models** in Sec 4. The adversary's goals is to attain the private information of the client dataset (e.g., membership information). The knowledge of the adversary is limited to the prompts received from the client. The adversary can only get a tuned prompt provided by the client but can leverage any available LLMs for attacking. 

**The real-world consequence** of privacy leakage from prompts could result in a violation of privacy regulations, e.g., GDPR (gdpr-info.eu). Concretely, private identifiable information (e.g., names) could be exposed in prompts.
Empirically, the privacy risks have been identified in existing works using viable attacks [1,2]. Liu (2023) (https://twitter.com/kliu128/status/1623472922374574080) shows a real-world attack where private instructions behind Bing can be extracted merely by adversarial prompts. We have incorporated these additional discussions in Sec 3.

In Fig 2, we show the privacy leakage from the prompt. In response to your second question, we also add an empirical evaluation of MIA in Section B.2. We show the non-trivial privacy risks (77% AUC of MIA attack in the worst case) for sample membership when prompt tuning is used.

[1] Duan et al. On the Privacy Risk of In-context Learning. TrustNLP 2023.

[2] Wang et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. NeurIPS 2023

Reviewer comment (Evaluation):
Lack of evaluation on privacy leakage. Considering that one of the key motivations of the study is to address the threats related to "data confidentiality" and "information leakage", it would be expected that an evaluation on the privacy performance of the proposed approach is conducted. There are only several examples of prompts provided, rather than evaluating the privacy protection in a quantitative manner. It would be necessary to involve some privacy metrics in the evaluation. It would be even better if some privacy attacks are involved in the privacy performance evaluation, instead of only comparing the model utility performance. In addition, the presentations of prompt examples are misleading and confusing - the "semantically-nearest retrieved" should not be in-line with other prompt messages.

Author response:
**A2.1**: Thanks for the valuable comment. We compare the attack results of four methods that comply with data confidentiality using Vicuna-7b and discrete prompts below. We use the loss-based membership inference attack.

|                                  |   Disaster |   Mpqa |   SST2  |  Trec  |
|:---------------------------------|--------------------------:|----------------------:|----------------------:|----------------------:|
| 0-shot   |                  0.495 |              0.578 |              0.499 |              0.542 |
| DLN-1            |                  0.486 |              0.539 |              0.481 |              0.505 |
| OPT            |                  **0.507** |              0.531  |              0.506 |              0.556 |
| DP-OPT          |                  0.494  |              0.520 |              0.504 |              0.514 |

We also evaluate Likelihood Ratio MIA attack (Mireshghallah, 2022) using the initial instruction as a reference model. 
|                                  |   Disaster |       Mpqa |       SST-2 |       Trec |
|:---------------------------------|-----------:|-----------:|-----------:|-----------:|
| DLN-1            |   0.498 |  **0.773** |   **0.773** |   0.446 |
| OPT             |   **0.511** |   0.443 |   0.510   |   0.494 |
| DP-OPT          |   0.497 |   0.456 |   0.518 |   0.468 |

We observe non-trivial AUCs for DLN-1 on Mpqa and SST-2. In comparison, both DP-OPT and OPT has very low AUC.
OPT has slightly higher risks than DP-OPT when applied on the Trec dataset.

> Q2.2: Presentations of prompt examples

Thanks for the suggestion! We have revised Fig 2 and 4!

Reviewer comment (Experiments):
Setting a constraint in prompt generation. To avoid leaking privacy information from generated prompts, one intuitive method could be adding some constraints during the prompt generation, such as "do not provide examples in the prompt", or "do not use existing samples but create dummy samples as examples in the prompt". I would suggest having a discussion on whether this would be feasible.

Author response:
**A3**: Thanks for the great suggestion. We add discussion on the last section of Appendix where We tried the two recommended instructions in experiments: 
For DLN-1 and (DP-)OPT, we all append the privatization instruction to the instruction in the backward template.
We test the instructions on SST-2 following the same setting in the main experiment and report the generated prompts in Table 10 in the Appendix.

Though vicuna-7b is instructed to keep data private, we still observe data breach in 2 out of 3 prompts for the first instruction. An example of a data breach is presented below. All the examples provided in the prompt can find exactly matched samples in the training set. Meanwhile, the prompt will present non-trivial risks (73% AUC) measured by Likelihood Ratio MIA attack. 

```
Instruction: Classify the input text as positive or negative.   For example:    * real-life persona: positive  * \privdata{by a pack of dogs who are smarter than him} (by a pack of dogs who are smarter than him): negative  * candid, archly funny and deeply authentic: positive  * brian tufano 's handsome widescreen photography and paul grabowsky 's excellent music turn this fairly parochial melodrama into something really rather special . : positive
```

Interestingly, the second instruction only generates dummy examples that have no similar examples in the training set. However, the prompt will present non-trivial risks (69% AUC) measured by LiRa MIA attack.

In conclusion, though privatization instruction could remove private examples, it still suffers from information leakage. The method is orthogonal to our method that provides theoretical guarantees and can be combined with our DP-OPT to reduce the chance of explicit leakage.