Paper: Hot PATE: Private Aggregation of Distributions for Diverse Tasks (ICLR-2024)

Reviewer comment (Novelty):
The main contribution is not related to the prompts. "The aggregation method should preserve privacy but to facilitate the knowledge transfer from teachers to the student, should critically also preserve the diversity of the teacher distributions. Our primary technical challenge was to formalize this requirement and design an aggregation method with a good privacy utility tradeoff." If this is the main contribution the hot PATE can be used for any task, not only for the discrete prompts.

Author response:
Indeed the main technical contribution is a method for diversity-preserving private aggregation of distributions.  We view learning via promps as a very compelling use case. This use case motivated our design.

Reviewer comment (Novelty):
It is not true that PATE was used only for the classification tasks. The submission claims: "Until now, PATE has primarily been explored with classification-like tasks, where each example possesses a ground-truth label, and knowledge is transferred to the student by labeling random examples." Many follow-ups for PATE consider, e.g., image generation: G-PATE https://proceedings.neurips.cc/paper/2021/hash/171ae1bbb81475eb96287dd78565b38b-Abstract.html "G-PATE: Scalable Differentially Private Data Generator via Private Aggregation of Teacher Discriminators" [2].

Author response:
We simplified to focus on what is relevant. Indeed, "PATE" is broadly used for any sample-and-aggregate approach to DP machine learning and there are many works with this term.  The reviewer pointed to a GAN method called G-PATE.  This is not relevant to the way our approach extends on the original PATE or prompt-PATE.  G-PATE can be viewed as a different "branch" in the sample-and-aggregate private machine learning literature-- use private aggregation of gradients from teacher "discriminators" to train a student "generator." G-PATE and the related works do not address the issue we were out to address. We will explain this in a "related work" paragraph in the revised version.

Reviewer comment (Novelty):
The scope and contribution of this submission seems not clearly stated. Initially, the paper appears to claim a general extension of the PATE framework, but the method and experiment sections mainly focus on prompt learning. A clearer definition of the scope is needed for an accurate evaluation of the submission. Furthermore, since the distillation formulation (which aggregates distributions as this submission intends to do) was briefly discussed in the original PATE paper [1] (Appendix B.1), it's crucial to articulate the novelty and insights of this work in comparison to that prior research.

Author response:
A revised version is uploaded.  The scope is now better conveyed in the introduction. There is a related work paragraph.  There is an additional discussion on parameters (including $\tau$ that the reviewer commented on).

Reviewer comment (Writing):
Some "minor" points regarding Definition 1:  
- Missing left-bracket for $f(p^{(i)})_{i\in[n]})$  
- The notation $j\in V$ might not be rigorous. Perhaps $j$ should denote the *index* for words in $V$.  
- Why is the dependence on $\tau$ completely not reflected in the example shown on page 4.

Author response:
A revised version is uploaded.  The scope is now better conveyed in the introduction. There is a related work paragraph.  There is an additional discussion on parameters (including $\tau$ that the reviewer commented on).

Reviewer comment (Experiments):
This paper does not include empirical validation of the student model's performance.

Author response:
W2 [empirical validation of student model]  We propose a meta scheme and demonstrate by running the numbers that large privacy-preserving students prompts can be generated effectively. Large scale applications would require orthogonal components such as datasets and tasks, prompt engineering, and more. This is beyond the scope of this work. Our work can be viewed as proposing a method that SHOULD be used IF applications arise in which PATE is applied with diverse tasks. This is established with theory and simulations. The method itself might have other applications.

Reviewer comment (Experiments):
This paper does not mention related work, leaving it unclear whether there are other baseline works for comparison.

Author response:
W3 We cited the related work we are aware of that seemed most relevant. There is significant number of works on PATE, Prompts, LLMs, out there.  If the reviewer is aware of relevant works that we had missed we would appreciate sharing that.

Reviewer comment (Reproducibility):
This paper does not provide sufficient detail about the methodologies used, especially regarding the implementation of the hot PATE framework and the experimental setup.

Author response:
W4 The revised version would include additional details.

Reviewer comment (Writing):
What is considered as private information in this paper?

Author response:
Q3: The private information is the dataset that is used to generate the teacher prompts.  The dataset can be in the form of examples (e.g. medical records) that can be partitioned to teachers.

Reviewer comment (Writing):
What does the student model represent in the hot PATE framework?

Author response:
Q4: The student model is a new prompt that is privacy preserving.  It can be a set of example "shots" or have another form.

Reviewer comment (Experiments):
A critical weakness is the limited scope of the experimental evaluation. The evaluation is reported on a single experiments with only 5 outputs. This might not adequately reflect the framework’s performance with truly open-ended queries.

Author response:
W1, W2, W3
The demo was indeed limited by the GPT3.5 interface which only returns the top-5 probabilities. The purpose of the demo was meant to show that real models behave in the way we expect and when tasked with diverse tasks, indeed have diverse responses. Importantly, we demonstrated through the theory and also simulations that the more diverse the distributions are, the larger the gain from our approach over the prior naive methods of dealing with diversity.  Moreover, with our method there is no utility loss with diversity.  Utility only depends on the extent of agreement between teacher distributions, not on the "entropy" of the agreement components. This follows from the theory and demonstrated with simulations (Section 7). So as long as you believe the premise that models tend to cover the diversity of the answer space to a query, you need to see the value of the method.

Reviewer comment (Experiments):
The generalizability of the empirical results is questionable due to the specificity of the demonstration.

Author response:
W1, W2, W3
The demo was indeed limited by the GPT3.5 interface which only returns the top-5 probabilities. The purpose of the demo was meant to show that real models behave in the way we expect and when tasked with diverse tasks, indeed have diverse responses. Importantly, we demonstrated through the theory and also simulations that the more diverse the distributions are, the larger the gain from our approach over the prior naive methods of dealing with diversity.  Moreover, with our method there is no utility loss with diversity.  Utility only depends on the extent of agreement between teacher distributions, not on the "entropy" of the agreement components. This follows from the theory and demonstrated with simulations (Section 7). So as long as you believe the premise that models tend to cover the diversity of the answer space to a query, you need to see the value of the method. 

W2, W3
The use case of prompts is interesting in that it allows for a very large number of teachers, when we have enough data,compared with traditional PATE (where each teacher is a trained model). We included a numerical analysis of privacy cost per yield (token) in some regimes and number of yields dependence on number of teachers. The number of yield queries increase quadratically with the number of teachers. For example, if we have 1M sensitive examples, we get 10 examples for each of 100k teachers. This can allow for large (10^5+ student prompts). Indeed, we do not have a concrete evaluation on a particular task and particular dataset, but we did run the numbers and they demonstrate the feasability.

Reviewer comment (Experiments):
Can the authors elaborate on the framework's expected performance with fully open-ended queries and the plans for more comprehensive empirical evaluations?

Author response:
Q1 [Performance on fully open-ended queries] This submission is a proof of concept. We propose a method, and demonstrate its potential with theory and simulations and running the numbers. Actual implementation requires orthogonal components including datasets and prompt engineering that might be tailored to an application. When we have access to a large model beyond public interfaces or when publicly available interfaces provide more of the token distribution we can extend the experiments. As said, the beauty of our method is that there is no cost for diversity. This follows from the theory.

Reviewer comment (Evaluation):
Can you also comment on the framework scalability and its impact on privacy?

Author response:
Q3 [Scalability/privacy] See Section 7 and response to W2,W3. We did run the numbers. As for impact on privacy. We use DP, with mathematical guarantees and run the numbers with overall epsilon<1. We have in mind a use case of highly sensitive data.