Paper: DINAR: Fine-Grained Privacy Preserving Federated Learning (ICLR-2024)

Reviewer comment (Theory):
I have privacy concerns for other layers.

Author response:
**What are the privacy risks contained in other layers?:**

[Mo et al.; 2021] and Fig 1 compare the risk of information leakage of different layers of several datasets and model architectures. Protecting specifically the most sensitive layer also shows the best privacy protection for the overall model, as shown in Fig 3, where the fine-grained obfuscation was applied on the one or the other of the model layers.

Reviewer comment (Novelty):
The motivation seems interesting, but it is from an unpublished paper.

Author response:
**The motivation is from an unpublished paper. Are there any similar papers from reputable conferences/journals?:**

In addition to the study of [Mo et al., 2021], that was published in ICLR 2021 - volume Workshop on Distributed and Private Machine Learning, the following paper could also be cited:
Fan Mo, et al. PPFL: privacy-preserving federated learning with trusted execution environments. ACM MobiSys 2021.

Reviewer comment (Evaluation):
The authors aim to protect a specific layer in FL models. What are the privacy risks contained in other layers? I think this protection is insecure.

Author response:
**What are the privacy risks contained in other layers?:**

[Mo et al.; 2021] and Fig 1 compare the risk of information leakage of different layers of several datasets and model architectures. Protecting specifically the most sensitive layer also shows the best privacy protection for the overall model, as shown in Fig 3, where the fine-grained obfuscation was applied on the one or the other of the model layers.

Reviewer comment (Novelty):
The motivation is from an unpublished paper. Are there any similar papers from reputable conferences/journals? I have great concerns about the reliable analysis of the motivation. Although we should care about both published and unpublished papers, I think it would be better for authors to find more support for their conclusive motivation.

Author response:
**The motivation is from an unpublished paper. Are there any similar papers from reputable conferences/journals?:**

In addition to the study of [Mo et al., 2021], that was published in ICLR 2021 - volume Workshop on Distributed and Private Machine Learning, the following paper could also be cited:
Fan Mo, et al. PPFL: privacy-preserving federated learning with trusted execution environments. ACM MobiSys 2021.

Reviewer comment (Novelty):
The authors missed the discussion of similar related works in the same research direction.

Author response:
**Similar related works in the same research direction:**

In addition to DP-based privacy-preserving methods, comparison and evaluations with other related works based on gradient compression and secure aggregation were included to the paper.

Reviewer comment (Novelty):
The author claims better model accuracy. I suspect it is from using adaptive gradient descent. If so, the improved utility is not from the newly-designed protection, i.e., not novel. Where is the high utility from?

Author response:
**Where is the high utility from?:**

Better model accuracy comes from the combination of the fine-grained approach that obfuscates specifically the sensitive layer and the adaptive gradient descent. 
For a fair comparison with related works in terms of model accuracy, all considered privacy protection methods are also evaluated with a variant that applies the adaptive gradient descent as well (cf. Figures 5 and 6, for instance LDP vs. LDP+ with the adaptive gradient descent).

Reviewer comment (Reproducibility):
It is unclear how the server or the clients derive the sensitive layer, e.g., layer p. Do all the clients share the same sensitive layer even under non-IID settings? If the clients have different sensitive layers, how does Dinar solve the divergence problem of local model updates since the clients perturb different layers?

Author response:
**Do all the clients share the same sensitive layer?:**

All FL clients know the sensitive layer, that is the same for all clients. This assumption is similar to the fact that all FL clients know the model architecture that is used for their data, which is the same for all clients.
The knowledge of the sensitive layer is obtained thanks to a prior study [Mo et al.; 2021], or an empirical evaluation (cf. Fig. 1). Furthermore, an additional evaluation of the impact of various FL non-IID settings on the actual privacy protection is included.

Reviewer comment (Experiments):
It is unclear which MIA methods Dinar is evaluated against in the paper. The author should claim the attack methods more explicitly.

Author response:
**Which MIA methods Dinar is evaluated against in the paper?:**

DINAR and the other defense methods in the paper are evaluated against the MIA method proposed by [Shokri et al., 2017], where shadow training techniques are used to build an attack model whose purpose is to distinguish the target model’s behavior on the training inputs from its behavior on the inputs that it did not encounter during training. This will be better explained in the paper.

Reviewer comment (Presentation):
It might be a good idea to shrink the scope of the title to "privacy against MIA" rather than general privacy-preserving FL.

Author response:
**Shrink the scope of the title to privacy against MIA:**

Thank you for the suggestion. The title of the paper will be replaced by: Heuristic-Guided Privacy against Membership Inference Attacks in Federated Learning.

Reviewer comment (Novelty):
It would be helpful if the authors could provide a clear and explicit comparison between the proposed DINAR method and the layer-wise privacy characterization studied in FL by Mo et al. (2021). Discussing the differences, similarities, and potential advantages of DINAR over Mo et al. will enhance the reader's understanding of the novel contributions of the present study.

Author response:
**Comparison between DINAR and the layer-wise privacy characterization studied by Mo et al. (2021):**

Whereas Mo et al. (2021) propose metrics to quantify information leakage from gradients computed over the training data, DINAR proposes a protection method for a fine-grained obfuscation of gradients of the sensitive layer that leaks more information. The sensitive layer was precisely identified thanks to a metric defined in Mo et al. (2021).
This will be better clarified in the paper.

**Sensitive layer of a model architecture:**

Based on a prior knowledge of the layer sensitivity of a model architecture, e.g., through a prior study [Mo et al.; 2021], or an empirical evaluation (cf. Fig. 1), DINAR algorithm’s parameter p is configured with the index of the model sensitive layer.

Reviewer comment (Novelty):
The literature review on DP & FL seems outdated, with references primarily from 2019 and 2020. Recent advancements have significantly improved the utility of DPFL algorithms. Therefore, it is recommended that the authors conduct a comprehensive review of the latest DP & FL algorithms, such as [1,2,3,4].

Author response:
**More recent DP/FL  literature:**

Thank you for the additional references. As suggested, more recent related works on DP in FL are now considered and compared against the other methods. We also added recent related works that make use of other privacy protection techniques such as gradient compression. The paper has been updated accordingly.

Reviewer comment (Theory):
Could the authors shed more light on why adaptive gradient descent exhibits superior convergence behavior compared to Adam? The statement that “Given the high-dimensional nature of optimization problems in neural networks, this technique dynamically adjusts the learning rate for each dimension in an iterative manner” holds not only for adaptive gradient descent but also for Adam.

Author response:
**Adaptive Gradient Descent vs. Adam:**

Section 4.3 has been updated to better clarify the similarities and differences between Adam and the considered adaptive gradient descent.

Reviewer comment (Evaluation):
The comparison to DP-based techniques in Section 4.4 Figure 4 might be unfair, as the specific $\epsilon$ used for DP methods is either undisclosed or excessively large. According to Section 5.1, $\epsilon$ might be set to 2.2 for all DP-related experiments throughout the paper. DP, by definition, safeguards data privacy against membership inference attacks. The suboptimal empirical privacy performance of DP methods in Figure 4 could potentially be attributed to the utilization of a large privacy budget. It would be more convincing if the authors could evaluate MIA against DP methods under a small privacy budget.

Author response:
**The specific used for DP methods is either undisclosed or excessively large + Evaluate MIA against DP methods under a small privacy budget:**

The privacy budget of DP methods is indeed described in §5.1, when introducing the baselines. 
In addition, a new empirical evaluation considering different DP privacy budgets was included to the paper, in order to evaluate the actual tradeoff between privacy and utility under different (more or less small) DP settings.

Reviewer comment (Experiments):
It is claimed that “leveraging adaptive gradient descent and, thus, further maximizing model utility.” However, there are no ablation studies or analyses verifying the effectiveness of adaptive gradient descent, compared to other optimization methods such as Adam and SGD.

Author response:
**Ablation study:**

Thank you for pointing out this. An ablation study is added to the paper, to show the improvement of model utility.

Reviewer comment (Experiments):
The experimental setup appears to be overly simplistic, with only 5 clients considered for all datasets. This is contrary to the typical cross-silo and cross-device settings, which usually involve a significantly larger number of clients.

Author response:
**Only 5 FL clients:**

A new empirical evaluation with various numbers of FL clients is added to the paper.

Reviewer comment (Experiments):
Partitioning the entire dataset among just 5 clients might result in each client possessing a sufficient quantity of data, thereby ensuring the local model is well-trained and potentially leads to memorization or overfitting phenomena for specific layers. It might be helpful to explore how the effectiveness of the proposed method under MIA attacks might be influenced by varying the number of clients, the size of local data, or the number of local training epochs.

Author response:
**Only 5 FL clients:**

A new empirical evaluation with various numbers of FL clients is added to the paper.

Reviewer comment (Theory):
While the proposed method is insightful, it is primarily based on heuristics and lacks privacy guarantees. Consequently, it is possible that advanced membership inference attacks could compromise the proposed method, whereas DP, with a small $\epsilon$, is guaranteed to provide privacy preservation.

Author response:
**Heuristics vs. optimization-based approaches:**

The proposed protection method is indeed based on a heuristic. Although heuristics-based approaches do not provide strict guarantees, we believe that they are useful and sometimes necessary as a first step for practical solutions in many research problems and areas, such as artificial intelligence [1,2], medicine [3], law [4], etc.
We believe that it is our responsibility as scientists to share these findings, for a better understanding of the problem and the behavior of a system under certain conditions. 
To better reflect the followed approach, the title of the paper will be replaced by: Heuristic-Guided Privacy against Membership Inference Attacks in Federated Learning.

[1] C.-A. Cheng, et al. Heuristic-Guided Reinforcement Learning. NeurIPS 2021.

[2] O. Salzman, et al. Heuristic-Search Approaches for the Multi-Objective Shortest-Path Problem: Progress and Research Opportunities. IJCAI 2023.

[3] J. N. Marewski, et al. Heuristic decision making in medicine. Dialogues Clin. Neurosci.,14(1), 2012.

[4] G. Gigerenzer, G., et al. Heuristics and the Law. Cambridge, MA, MIT Press, 2007.

Reviewer comment (Evaluation):
What is the attack success rate of the MIA on the vulnerable data points, i.e., the data points whose memberships are easy to infer?

Author response:
**What is the attack success rate of the MIA on the data points whose memberships are easy to infer?**

Thank you for pointing out this. An additional evaluation specifically targeting more vulnerable data points vs. less vulnerable ones, under different protection methods, has been added to the paper.  Overall, privacy protection techniques based on gradient compression and WDP+ provide less protection for more vulnerable data points, compared to less vulnerable ones. In contrast, DINAR as well as the other protection methods (e.g., secure aggregation, and other DP-based methods) provide good protection independently from the level of vulnerability of data points, although the former provides better model utility.

Reviewer comment (Theory):
How does the proposed method deal with non-i.i.d problems in FL? Specifically, when the data is non-i.i.d, obfuscating multiple layers will incur a distribution shift between the client's local data and the data from other clients.

Author response:
**Non-IID settings:**

A new experimental evaluation comparing privacy methods under different non-IID FL settings was conducted, and its results have been added to the paper. Overall, for all cases but DINAR, the lower the non-IID distribution is, the higher the attack success rate is, since the shadow model of the membership inference attacker is able to better learn on such data. In contrast, DINAR's privacy protection is independent from the underlying non-IID FL setting and remains minimal at 50%. When it comes to model utility, obviously, the lower the non-IID distribution is, the higher the model utility is, although, DINAR reaches the highest model accuracy when protecting the model.

Reviewer comment (Evaluation):
In Figure 6, why the model with no protection can have lower model performance compared to the model trained by the proposed method?

Author response:
**In Fig 6, model with no protection can have lower model performance than model trained with DINAR, why?:**

In Fig 6, No defense applies Adam, whereas No defense+ and DINAR both apply AdaGrad adaptive gradient descent which produce better model accuracy.