Paper: Protecting Sensitive Data through Federated Co-Training (ICLR-2024)

Reviewer comment (Writing):
The explicit motivation is unclear.

Author response:
* The main motivation for our work is to improve privacy in federated learning. As stated in the abstract and introduction, federated learning promises to protect data privacy by only sharing model parameters, but in fact model parameters reveal a lot about training data. Differential privacy mechanisms can be used to improve privacy, but their privacy-utility trade-off is not ideal. Therefore, improving privacy is a very relevant problem. We approach this problem by first noting that in many application scenarios where privacy is paramount, e.g., healthcare, large public unlabeled datasets exist. Our approach uses such an unlabeled dataset to exchange information between clients. Distributed distillation and its variants have shown that by sharing soft labels for such an unlabeled datasets we can train neural networks collaboratively. We show that sharing soft labels indeed improves privacy over sharing model parameters, but does not fully protect private data. By sharing hard labels instead, we are able to protect privacy to the highest degree - at least in terms of existing membership inference attacks - while maintaining the model quality of sharing model parameters and soft labels. We hope that this clarifies our motivation. We will improve the description of our motivation in the manuscript.

Reviewer comment (Writing):
What is the explicit motivation of this paper?

Author response:
* The main motivation for our work is to improve privacy in federated learning. As stated in the abstract and introduction, federated learning promises to protect data privacy by only sharing model parameters, but in fact model parameters reveal a lot about training data. Differential privacy mechanisms can be used to improve privacy, but their privacy-utility trade-off is not ideal. Therefore, improving privacy is a very relevant problem. We approach this problem by first noting that in many application scenarios where privacy is paramount, e.g., healthcare, large public unlabeled datasets exist. Our approach uses such an unlabeled dataset to exchange information between clients. Distributed distillation and its variants have shown that by sharing soft labels for such an unlabeled datasets we can train neural networks collaboratively. We show that sharing soft labels indeed improves privacy over sharing model parameters, but does not fully protect private data. By sharing hard labels instead, we are able to protect privacy to the highest degree - at least in terms of existing membership inference attacks - while maintaining the model quality of sharing model parameters and soft labels. We hope that this clarifies our motivation. We will improve the description of our motivation in the manuscript.

Reviewer comment (Novelty):
Why is this idea important?

Author response:
* As discussed in the first point, our motivation is to improve privacy while maintaining model quality in federated learning. We do so by using an unlabelled dataset.

Reviewer comment (Writing):
The first half of the abstract introduces some general and basic knowledge of federated learning, data privacy, applications, and co-training. The logic is not smooth, and several sentences have little relation to each other. What is the explicit problem that you target here? What are the explicit problems of prior research?

Author response:
* The main motivation for our work is to improve privacy in federated learning. As stated in the abstract and introduction, federated learning promises to protect data privacy by only sharing model parameters, but in fact model parameters reveal a lot about training data. Differential privacy mechanisms can be used to improve privacy, but their privacy-utility trade-off is not ideal. Therefore, improving privacy is a very relevant problem. We approach this problem by first noting that in many application scenarios where privacy is paramount, e.g., healthcare, large public unlabeled datasets exist. Our approach uses such an unlabeled dataset to exchange information between clients. Distributed distillation and its variants have shown that by sharing soft labels for such an unlabeled datasets we can train neural networks collaboratively. We show that sharing soft labels indeed improves privacy over sharing model parameters, but does not fully protect private data. By sharing hard labels instead, we are able to protect privacy to the highest degree - at least in terms of existing membership inference attacks - while maintaining the model quality of sharing model parameters and soft labels. We hope that this clarifies our motivation. We will improve the description of our motivation in the manuscript.

Reviewer comment (Writing):
The proposed idea seems to have better model quality and privacy. However, the ambiguous language makes me confused. What is the definition of privacy or quality?

Author response:
* Model quality is canonically measured in terms of test accuracy as a proxy for the generalization error. Privacy is measured empirically by the likelihood of an attacker being able to conduct a successful membership inference attack (as described in Sec. 5 “Experimental Setup”, see also the probabilistic interpretation of ROC AUC), and it is determined theoretically via differential privacy (as described in Sec. 4).  

Dear reviewer UQKJ,

We gladly explain the definitions for privacy used in our paper. As you are certainly aware, there are two general views on how to measure privacy: theoretical and empirical. We use both and have used the standard definitions. The motivation for using both, as mentioned in the introduction, is as follows.

The theoretical view rightfully claims that the only proof of privacy is a theoretical guarantee. Here, differential privacy (Dwork, 2006) is currently the de-facto gold-standard. Standard mechanisms that ensure differential privacy are the Laplacian and Gaussian mechanism, that add appropriate noise to the shared data. In our case, the shared data is binary and both mechanisms are not applicable. Therefore, we use the XOR mechanism, which requires a novel sensitivity analysis to select the noise variance appropriately (our Proposition 2).

A major downside of the theoretical view is that the privacy-utility trade-off in practice is often sub-optimal. As we state in our introduction, the actual privacy guarantees attainable in federated averaging with decent utility (measured in model quality) are basically vacuous (to the best of our knowledge, the group of Srini Devadas has made the most accurate analysis of differential privacy in federated learning, so far). The empirical view argues that exactly for that reason theoretical guarantees do not quantify actual privacy in a practically useful way. Instead, the privacy can be assessed much more accurately by applying the most advanced attacks and testing how well they succeed. For example, Reza Shokri has developed several (versions of) toolkits that implement this strategy. We operationalize this empirical privacy in our vulnerability score , which is the ROC AUC of a large number of membership inference attacks against the shared data (model parameters for FedAvg, soft labels for DD, and hard labels for FedCT). We explained this in Section 5 of the manuscript and provide further details in Appendix C1. From the probabilistic interpretation of ROC AUC it follows that the vulnerability score measures the success probability of such attacks and thus provides an interpretable measure.

In conclusion, both differential privacy and vulnerability are useful measures of privacy. Differential privacy guarantees that no matter the strategy or ingenuity of the attacker, they cannot infer anything about the data beyond the limits of the guarantee, but it might underestimate the actual privacy, no matter that we thoroughly analyzed sensitivity to make the bound as tight as possible. At the same time, vulnerability quantifies how well protected private data is in practice, but it might overestimate privacy, since it does not account for yet undiscovered attacks.

While we cannot explain this fundamental aspect of measuring privacy in detail in the introduction, we will gladly include a paragraph in the appendix.

Reviewer comment (Novelty):
The authors detailed what they have done, but it is unclear why the idea of combining co-training, differential privacy, and federated learning is meaningful. What are your insights?

Author response:
* Our insights are that sharing hard labels instead of model parameters or soft labels indeed improves privacy - what was to be expected - and surprisingly does not decrease model quality. Moreover, we find that sharing hard labels has the additional benefit of allowing us to train interpretable models, like decision trees and rule ensembles, in a federated way. We can even train ensembles of various local models, as we show in the general comment.

Reviewer comment (Novelty):
In the Construction section, Section 3 combines the prior research and the new idea, and Section 4 starts with an introduction of differential privacy. Could the authors elaborate on what is new here? What research line do you follow?

Author response:
* Our Section 3 explains the scenario, introduces the novel method FedCT and analyzes its convergence and communication complexity. Since privacy is the main motivation for our work, we show in Section 4 how a privacy guarantee based on differential privacy can be derived for FedCT. For this, we provide a novel sensitivity analysis (Proposition 2), which is required to provide a meaningful privacy guarantee.

Reviewer comment (Novelty):
The novelty is limited. There are many prior studies have explored semi-supervised federated learning [1-3] and knowledge distillation [4-6]. The paper does not propose a new framework of FedSSL but just naively combining differential privacy techniques. The paper does not illustrate the difference between FedCT and those previous work.

Author response:
We are aware of the wide range of semi-supervised federated learning methods, but to the best of our knowledge, FedCT is novel in that it shares hard labels on a public unlabeled dataset. This improves privacy substantially while maintaining model quality. Moreover, it allows us to train interpretable models. Since sharing hard labels hasn’t been explored in the literature, achieving differential privacy is not straight-forward. We employed the XOR mechanism for binary data, which required us to derive a novel bound on the sensitivity of classification algorithms. Regarding the papers you have referenced, [4] is not applicable since it assumes a public labeled dataset, [3] shares soft labels similar to distributed distillation, and [1],[2],[5],[6] share model parameters and therefore do not improve privacy over  FedAvg at all.

Reviewer comment (Theory):
The motivation to protect the privacy of soft predictions is weak. Comparing to raw data/ features map, the soft predictions are not likely to leak privacy.

Author response:
As our experiments show, the vulnerability of sharing soft labels to known membership inference attacks is lower than sharing model parameters, as one would expect, but still considerable. Since sharing hard labels achieves a model quality similar to sharing soft labels, its additional privacy is a clear advantage. Together with the fact that sharing hard labels allows us to train interpretable models, or mixed models (see the general comment), this means that FedCT has clear advantages over both sharing model parameters and soft labels.

Reviewer comment (Presentation):
The paper is not well-organized and using non-standard terminology. For example, the term 'semi-honst' in section 4. Usually, we use 'honest-but-curious' to describe the server which can not modify the local updates but trys to infer private information of clients.

Author response:
We used the term semi-honest as defined in [7] and used widely in cryptography and security, but we are happy to change the term to honest-but-curious in our manuscript.

Reviewer comment (Experiments):
The baselines are not comprehensive. DP-FedAvg is using DP to the updated gradients but the proposed FedCT is using DP in the soft predictions, it is not comparable. FedCT should compare other FedSSL methods [1-3] and conduct attach on the method to verify the effect of FedCT.

Author response:
We have compared FedCT to model parameters sharing and soft label sharing in terms of accuracy and privacy, as well as the common differential privacy mechanism applied to federated averaging. Our empirical privacy evaluation measures the success in terms of ROC AUC of an attacker to perform state-of-the-art membership inference attacks on the information that is shared - model parameters, soft labels, or hard labels. Therefore, the resulting vulnerability scores are comparable. We do not see how we can attach our method of sharing hard labels to other federated semi-supervised learning methods that share soft labels or model parameters. Could you please elaborate how you envision such experiments?

Reviewer comment (Experiments):
Setting for data heterogenity is not sufficient. With beta = 2 for non-i.i.d setting is not good. Most of papers set beta = 0.01 or 0.1 to create heterogeneous data partitions.

Author response:
We did not extensively explore scenarios of too strong heterogeneity. Setting $\alpha$ to 2 yields the class distribution shown in Figure 2. The impact of strong heterogeneity on consensus is intricate and depends on factors such as the local learning algorithm, model, number of clients, and consensus mechanism robustness. Investigating this is very interesting, but it falls beyond the scope of this work.

[Figure 2:  Class distribution for $\alpha = 2$ ](https://drive.google.com/file/d/1d_2E5NN3bXYUzOciYw0dKC66eORAN3wa/view?usp=sharing)

Reviewer comment (Theory):
The motivation is not clear. According to the basic idea of knowledge distillation, soft labels should potentially contain more useful information than hard labels. The authors are encouraged to clarify the reason for replacing soft labels with hard labels.

Author response:
**W2.** The motivation for our paper is to maximize privacy in a federated learning scenario, where clients have access to an unlabeled public dataset. Indeed, sharing soft labels preserves privacy to a larger extent than sharing model parameters, as our experiments show for the first time, yet they still reveal a lot about private data (VUL >$ 0.6$). As discussed in the introduction, we wanted to go one step further by sharing hard labels. Indeed, this substantially improves privacy protection, with an empirically optimal privacy (VUL $\approx 0.5$). Of course, this is only optimal wrt. known membership inference attacks, which is why we developed the differentially private variant of FedCT. What was surprising to us is that strong improvement in privacy comes at virtually no loss in accuracy. That is, we achieve a test accuracy comparable to FedAvg and Distributed Distillation, the latter sharing soft labels. We conjecture that in practice the advantage of sharing soft labels over hard labels in terms of model performance is negligible, yet the additional privacy risk is significant. Moreover, sharing hard labels allows us to train interpretable models as well, and even use different models at each client (please see our general comment for further details).

Regarding your concern W2, we will gladly clarify the distinction between hard and soft label sharing in the manuscript. To reiterate: As mentioned in the introduction, sharing hard labels instead of soft labels improves privacy and allows us to train interpretable models. Since hard labels carry less information than soft labels, this could in principle lead to a decrease in model quality. In our empirical evaluation (particularly Table 1), we show that we cannot observe any significant decrease in model quality - both soft and hard label sharing perform comparable to model parameters sharing (FedAvg). Note that we use distributed distillation as the most prominent representative of soft label sharing. At the same time, our experiments support the claim that sharing hard labels improves privacy: soft label sharing has an average vulnerability of 0.61, whereas sharing hard labels has a substantially reduced vulnerability of 0.52. We summarize these differences in the following table. We will include a polished version of the table together with a paragraph explaining these details in the appendix.

| Shared Information | Model Quality IID | Model Quality Non-IID | Privacy | Interpretable Models | 
| --- | --- |--- | --- | --- | 
| Model Parameters | ++| ++| - | -| 
| Soft Labels | ++| +| + | - | 
| Hard Labels (FedCT) | ++| +| ++| ++|
**Table 1: Comparison of parameter, soft label, and hard label sharing in federated learning.**

Note that both soft and hard label sharing requires an unlabeled public dataset.

Reviewer comment (Experiments):
More representative baselines are needed. As mentioned in W1, there are many similar works; it would be more convincing to conduct experiments to compare them.

Author response:
**W3.** We selected the most prominent baselines for each type of methods, i.e., vanilla FedAvg as a representative for model sharing in homogeneous data scenarios, DP-FedAvg as its private variant, and distributed distillation for sharing of soft-labels (or co-regularization). We additionally compared FedCT to PATE for this rebuttal, a model distillation method that can be adapted to our scenario. The results in Table 1 indicate that PATE is outperformed by FedCT and all other baselines in terms of test accuracy. As discussed above, we cannot compare to the approaches you referenced, since they are not applicable in our scenario.

| Dataset       | FedCT   | PATE    |
|---------------|---------|---------|
| FashionMINST  | 0.7658  | 0.6451  |
| CIFAR10       | 0.7608  | 0.7039  |
| Pneumonia     | 0.7478  | 0.7208  |
| MRI           | 0.6274  | 0.6038  |
| SVHN          | 0.8805  | 0.8721  |
**Table 1: Average test accuracy ACC of FedCT and PATE for m=5**

Reviewer comment (Presentation):
The appendix mentioned in the paper cannot be found; maybe it is just for me.

Author response:
**W4.** The appendix is available as a separate document in the supplementary material. For convenience, we also included a full document (paper and appendix) in the supplementary material. If you have any trouble accessing the supplementary material, please let us know and we will upload it to our anonymized repository.

Reviewer comment (Novelty):
This paper does not discuss the difference between the proposed method and PATE (Papernot et al., 2016). In PATE, each teacher model can be viewed as a local model, and a majority vote is also utilized to reach a consensus for learning from each other's knowledge. The key distinction is that PATE transfers this knowledge to a student model, while the method in this paper circulates the knowledge back to each teacher/local model. Without further discussion, it's challenging to ascertain whether the contribution of this paper is incremental when compared to PATE and DD (Bistritz et al., 2020).

Author response:
PATE is essentially a distillation algorithm with the goal to produce a single student model from an ensemble of teachers. To protect the private dataset the teachers have been trained upon, a Laplace mechanism is applied to their predictions - more precisely, their counts - before determining the consensus label. Thereby, a curious student cannot infer upon the private training data. In both PATE and FedCT, hard labels are used to form a consensus. The fundamental differences are that (i) PATE is not a collaborative training algorithm and that (ii) PATE therefore is not concerned with protecting against an honest-but-curious server. Regarding point (i): In a nutshell, PATE trains all teachers until convergence once, uses their predictions to pseudo-label the public unlabeled data, and then train the student on this pseudo-labeled dataset. Thus, teachers never improve. This fundamentally differs from FedCT where clients train models collaboratively by iteratively sharing hard labels to improve the pseudo-labeled dataset, which in turn improves local models, which again improves the pseudo-labeling. This difference also explains point (ii): In PATE, the student is separated from the teachers, so that the main privacy concern is protecting against a curious student. In FedCT instead all clients can be honest-but-curious, but their data needs to be not only protected from other clients, but also from an honest-but-curious server. Therefore, we cannot apply the Laplace mechanism that PATE uses, but instead have to protect the hard labels, which we achieve via the XOR mechanism.

We have discussed PATE in our related work section under semi-supervised learning, since it performs distillation via multiple teachers, instead of collaborative learning. Consequently, PATE is concerned with protecting the privacy of consensus labels, not of teachers’ predictions. Distributed distillation, instead, collaboratively trains models and naturally we compared to it - both in distributed distillation and federated co-training, clients act both as teachers and students. Since PATE shares hard predictions of teachers, we can adapt it by letting each client be a teacher. Similar to the experiments reported in [1], each client trains a model to convergence and then makes predictions on the public unlabeled dataset. A consensus is formed from all predictions that are used to pseudo-label the unlabeled dataset. A single student model is then trained on the pseudo-labeled dataset. We report the results with $m=5$ clients in Table 1. 

| Dataset       | FedCT   | PATE    |
|---------------|---------|---------|
| FashionMINST  | 0.7658  | 0.6451  |
| CIFAR10       | 0.7608  | 0.7039  |
| Pneumonia     | 0.7478  | 0.7208  |
| MRI           | 0.6274  | 0.6038  |
| SVHN          | 0.8805  | 0.8721  |
**Table 1: Average test accuracy ACC of FedCT and PATE for m=5**

Note that in the original experiments, a much larger number of clients is required. Therefore, we also report results for $m=100$ clients for two datasets in Table 2.
| Dataset       | FedCT   | PATE    |
|---------------|---------|---------|
| FashionMINST  | 0.7154  | 0.6318  |
| Pneumonia     | 0.7269  | 0.6903  |
**Table 2: Average test accuracy ACC of FedCT and PATE for m=100**

The results show that performing just a single round of training performs substantially worse than collaborative training (both distributed distillation and our proposed FedCT). Therefore, the privacy-utility trade-off of PATE is substantially worse: it achieves lower test accuracy while its privacy mechanism does not protect against a honest-but-curious server.

Reviewer comment (Evaluation):
The experimental results are not promising. As depicted in Table 1, only in 1 out of 5 datasets does the proposed method outperform the baselines in terms of ACC, and on none of the datasets does the proposed method achieve the best privacy utility trade-off, that is the best ACC, while at the same time, having the best VUL compared to baselines.

Author response:
We would like to clarify that the goal of FedCT is not to outperform federated learning in terms of test accuracy, but to improve privacy. Naturally, any gain in privacy will typically come at a loss in accuracy. Therefore, our results are actually quite astonishing. We can achieve a test accuracy comparable to FedAvg with greatly improved privacy. That is, the ability of an attacker to distinguish a private training example from an arbitrary one is as good as random guessing (VUL=$0.5$). An amazing side-effect of FedCT is that sharing hard labels not only improves privacy, but also allows us to train interpretable models, like decision trees and rule ensembles. This is not possible with FedAvg or sharing of soft labels, like in distributed distillation.

Reviewer comment (Experiments):
This paper lacks clarity on the algorithm's scalability in relation to the number of clients, as it only reports the impact of client numbers on one dataset, and default settings of m (client numbers) for other datasets are only 5.

Author response:
We performed experiments for m=5 clients, which is typical in cross-silo scenarios that are ubiquitous in healthcare. We empirically evaluated the scalability in the number of clients on FashionMNIST in Figure 5 of our manuscript. Following your suggestion, we performed the same evaluation for the Pneumonia dataset as well and report the results in Figure 1. Similar to FashionMNIST, FedCT scales very well with the number of clients.

[Figure 1: Test accuracy (ACC) of FEDCT and FEDAVG (FL) Pneumonia with |U|= 200  for various numbers of clients m.](https://drive.google.com/file/d/1Txm7i6_sWjGqI85eYMcO5V2wNXKW7W2W/view?usp=sharing)

Reviewer comment (Experiments):
Besides differentially private distributed SGD (Xiao et al., 2022), are there any other related works that can be utilized as baselines for comparison with DP-FEDCT on the Privacy-Utility Trade-Off with Differential Privacy?

Author response:
The privacy of distributed SGD is arguably lower than FedAvg, since sharing gradients allows not only for membership inference attacks, but even for data reconstruction attacks [2]. At the same time, FedAvg performs similar to distributed SGD in the literature [3,4]. Therefore, we have used DP-FedAvg as a baseline in our experiments. Differential privacy mechanisms are in principle compatible with most federated learning approaches, including those designed personalized federated learning [5,6]. It will be great to extensively evaluate FedCT in those scenarios, but goes beyond the scope of this work.