Paper: Posterior Probability-Based Label Recovery Attack in Federated Learning (ICLR-2024)

Reviewer comment (Theory):
The focal loss definition is confusing. It doesn't seem to be internally consistent throughout the paper, and it does not seem to be exactly following [1] either. This makes it very hard to check the associated maths. In particular, in Eq. 1, $t$ is just an index of the sum, while in the first paragraph of page 4, it is the "target class." Further, the sum over $t$ in Eq. 1 does not appear in the original description in [1]. As currently defined, where $y$ appears in Eq. 1 is not clear - according to the text on page 4, it is embedded in $p_t$ (in a different way from how it is embedded in [1]), but according to the proof in Appendix A.1 that is not the case and $p_t$ is simply the post-softmax probability. According to [1], $\alpha_t$ depends on $y$, but the paper specifies nothing about this. Further, the temperature parameter $\tau$ and the smoothed labeling options are only passingly defined. Finally, in the proof in Appendix A.2, the sum over $t$ that is given in Eq. 1 seems to disappear.

Author response:
> Q1: Can the authors fix consistently throughout the paper and appendix the definition and usage of the focal loss?

In Appendix A.1, we step-by-step derive the Focal Loss in multi-class scenarios from the original binary Focal Loss in [1].

Some important points we would like to emphasize:


- **Difference in probabilities**: $p_i$ is the post-softmax probability, while $p_t$ represents the automatic determined confidence of the input at class $i$, where $t=i$.
- **Mechanism of weight**: For an easy-to-learn sample, $p_t$ might be close to the target label. So, Focal Loss assigns a small coefficient $(1-p_t)^{\gamma}$ as the weight. However, for a hard-to-learn sample, $p_t$ may be close to 0. Then $(1-p_t)^{\gamma}$ a relative large weight to enhance the ratio of the these samples in the total loss.
- **Summation sign**: Since the binary Focal Loss [1] just has one output, the summation is not necessary. In the multi-class case, we use the summation to cover all the classes $t\in[1,K]$, and aim to derive a general conclusion in Theorem 1.
- **Why Focal Loss**: To the best of our knowledge, Focal Loss has the general form in the cross-entropy loss variants and it can be converted to CE loss or BCE loss by setting different $\alpha$ and $\gamma$. We aim to derive a general form of label leakage from gradient, so we choose the Focal Loss.

[1] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988, 2017.

Reviewer comment (Writing):
The problem with the focal loss definition is further exacerbated by inconsistent notations. In particular, $i$s and $t$s are used interchangeably in the first paragraph on page 4, even though the authors themselves claim to differentiate them. For example, see the definition of $p_t$, and $\mathcal{L}_\text{CE} (p_t)$.

Author response:
> Q1: Can the authors fix consistently throughout the paper and appendix the definition and usage of the focal loss?

In Appendix A.1, we step-by-step derive the Focal Loss in multi-class scenarios from the original binary Focal Loss in [1].

Some important points we would like to emphasize:


- **Difference in probabilities**: $p_i$ is the post-softmax probability, while $p_t$ represents the automatic determined confidence of the input at class $i$, where $t=i$.
- **Mechanism of weight**: For an easy-to-learn sample, $p_t$ might be close to the target label. So, Focal Loss assigns a small coefficient $(1-p_t)^{\gamma}$ as the weight. However, for a hard-to-learn sample, $p_t$ may be close to 0. Then $(1-p_t)^{\gamma}$ a relative large weight to enhance the ratio of the these samples in the total loss.
- **Summation sign**: Since the binary Focal Loss [1] just has one output, the summation is not necessary. In the multi-class case, we use the summation to cover all the classes $t\in[1,K]$, and aim to derive a general conclusion in Theorem 1.
- **Why Focal Loss**: To the best of our knowledge, Focal Loss has the general form in the cross-entropy loss variants and it can be converted to CE loss or BCE loss by setting different $\alpha$ and $\gamma$. We aim to derive a general form of label leakage from gradient, so we choose the Focal Loss.

[1] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988, 2017.

Reviewer comment (Novelty):
The paper would benefit from a better discussion about differences to prior work, particularly [2], [3], and [5]. All these works, similarly to the proposed method, estimate a quantity that they later plug into $p - y$ to compute the label counts $\lambda_j$. Additionally, at least [3] and [5] discuss using auxiliary data to do so, again similarly to this work.

Author response:
> Q8: Can the authors provide a comparison with [3] or [5] when auxiliary data is used for estimating their parameters?

We name ZLG for [3] and LLG for [5]. The following experiments compare the label recovery accuracies with different model architectures, batch sizes and activation functions.

- Batch size=64, activation=ReLU:

  | Dataset   | Model     | ZLG ClsAcc | ZLG InsAcc | LLG ClsAcc | LLG InsAcc | Our ClsAcc | Our InsAcc |
  | --------- | --------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
  | MNIST     | LeNet     | **1.000**  | 0.996      | **1.000**  | **1.000**  | 0.995      | 0.955      |
  | CIFAR-10  | ResNet-18 | **1.000**  | 0.916      | 1.000      | 0.914      | **1.000**  | **1.000**  |
  | CIFAR-100 | ResNet-50 | 0.985      | 0.893      | 0.816      | 0.633      | **1.000**  | **1.000**  |

- Batch size=256, activation=ReLU:

  | Dataset   | Model     | ZLG ClsAcc | ZLG InsAcc | LLG ClsAcc | LLG InsAcc | Our ClsAcc | Our InsAcc |
  | --------- | --------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
  | MNIST     | LeNet     | **1.000**  | 0.977      | **1.000**  | **0.982**  | **1.000**  | 0.951      |
  | CIFAR-10  | ResNet-18 | **1.000**  | 0.905      | **1.000**  | 0.919      | **1.000**  | **0.977**  |
  | CIFAR-100 | ResNet-50 | 0.998      | 0.881      | **1.000**  | 0.868      | **1.000**  | **1.000**  |

- Batch size=64, activation=ELU:

  | Dataset   | Model     | ZLG ClsAcc | ZLG InsAcc | LLG ClsAcc | LLG InsAcc | Our ClsAcc | Our InsAcc |
  | --------- | --------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
  | MNIST     | LeNet     | **1.000**  | 0.948      | 0.580      | 0.283      | 0.990      | **0.969**  |
  | CIFAR-10  | ResNet-18 | **1.000**  | 0.902      | 0.980      | 0.697      | **1.000**  | **0.972**  |
  | CIFAR-100 | ResNet-50 | 0.985      | 0.897      | 0.845      | 0.603      | **1.000**  | **1.000**  |

Reviewer comment (Reproducibility):
Missing experimental details for Table 3 include: dataset/model/batch size/epoch/label distribution, and whether $\tau$ and $\epsilon$ are assumed to be known by the attacker. The reviewer suggests redoing the experiments in a setting where 100% accuracy is not achieved.

Author response:
> Q4: Can the authors expand the focal loss experiments to more settings, especially ones where the success rate is not 100%?

Here, we present additional experiments conducted on the Focal Loss. We mainly test the parameters of $\tau$, $\gamma$ and $\varepsilon$ on an untrained model, and average the experiments over 10 trials. Focusing on temperature $\tau$, we present several cases where the accuracy is not 100%. In addition, by varying $\gamma$ and $\varepsilon$ on these setting, the accuracies are not affected.

Temperature $\tau$ (batch size=64, activation=ReLU):

| Dataset   | Model     | $\tau$ | Our ClsAcc | Our InsAcc |
| --------- | --------- | ------ | ---------- | ---------- |
| MNIST     | LeNet     | 0.5    | 0.980      | 0.906      |
|           |           | 0.75   | 0.980      | 0.945      |
|           |           | 0.9    | 0.990      | 0.954      |
|           |           | 1.25   | 0.990      | 0.983      |
|           |           | 1.5    | 1.000      | 0.998      |
| CIFAR-10  | ResNet-18 | 0.5    | 0.990      | 0.960      |
|           |           | 0.75   | 1.000      | 0.994      |
|           |           | 0.9    | 1.000      | 1.000      |
|           |           | 1.25   | 1.000      | 1.000      |
|           |           | 1.5    | 1.000      | 1.000      |
| CIFAR-100 | ResNet-50 | 0.5    | 1.000      | 1.000      |
|           |           | 0.75   | 1.000      | 1.000      |
|           |           | 0.9    | 1.000      | 1.000      |
|           |           | 1.25   | 1.000      | 1.000      |
|           |           | 1.5    | 1.000      | 1.000      |

We have observed that the temperature parameter, $\tau$, significantly impacts smaller datasets. When $\tau$ is smaller, the space of logits expands, complicating the estimation of batch posterior probabilities. Consequently, as $\tau$ decreases, label accuracy deteriorates. For the large datasets with more classes, such as CIFAR-10, the logits space is hardly influenced by changing different $\tau$.

Reviewer comment (Experiments):
The authors should compare their method to [3] and [5] applied using auxiliary data, under the same auxiliary data conditions.

Author response:
> Q8: Can the authors provide a comparison with [3] or [5] when auxiliary data is used for estimating their parameters?

We name ZLG for [3] and LLG for [5]. The following experiments compare the label recovery accuracies with different model architectures, batch sizes and activation functions.

- Batch size=64, activation=ReLU:

  | Dataset   | Model     | ZLG ClsAcc | ZLG InsAcc | LLG ClsAcc | LLG InsAcc | Our ClsAcc | Our InsAcc |
  | --------- | --------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
  | MNIST     | LeNet     | **1.000**  | 0.996      | **1.000**  | **1.000**  | 0.995      | 0.955      |
  | CIFAR-10  | ResNet-18 | **1.000**  | 0.916      | 1.000      | 0.914      | **1.000**  | **1.000**  |
  | CIFAR-100 | ResNet-50 | 0.985      | 0.893      | 0.816      | 0.633      | **1.000**  | **1.000**  |

- Batch size=256, activation=ReLU:

  | Dataset   | Model     | ZLG ClsAcc | ZLG InsAcc | LLG ClsAcc | LLG InsAcc | Our ClsAcc | Our InsAcc |
  | --------- | --------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
  | MNIST     | LeNet     | **1.000**  | 0.977      | **1.000**  | **0.982**  | **1.000**  | 0.951      |
  | CIFAR-10  | ResNet-18 | **1.000**  | 0.905      | **1.000**  | 0.919      | **1.000**  | **0.977**  |
  | CIFAR-100 | ResNet-50 | 0.998      | 0.881      | **1.000**  | 0.868      | **1.000**  | **1.000**  |

- Batch size=64, activation=ELU:

  | Dataset   | Model     | ZLG ClsAcc | ZLG InsAcc | LLG ClsAcc | LLG InsAcc | Our ClsAcc | Our InsAcc |
  | --------- | --------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
  | MNIST     | LeNet     | **1.000**  | 0.948      | 0.580      | 0.283      | 0.990      | **0.969**  |
  | CIFAR-10  | ResNet-18 | **1.000**  | 0.902      | 0.980      | 0.697      | **1.000**  | **0.972**  |
  | CIFAR-100 | ResNet-50 | 0.985      | 0.897      | 0.845      | 0.603      | **1.000**  | **1.000**  |

Reviewer comment (Theory):
The proposed method requires an auxiliary dataset with the same distribution of training data. This requirement is impractical in FL.

Author response:
We would like to highlight the following points to justify this assumption:

1. **Realistic Threat Model:** In many practical scenarios, attackers often have access to auxiliary data sources that closely resemble the distribution of the target training data. This assumption aligns with the notion that attackers may gather information from publicly available sources or other related datasets to enhance the effectiveness of their attacks.
2. **Common Practice in Security Analysis:** Making use of auxiliary data with a similar distribution is a standard practice in security analysis, especially in the research fields of *Membership Inference Attacks* [1, 2] and *Gradient Inversion Attacks* [3, 4]. This enables a more thorough evaluation of the model's robustness, covering a broader range of potential adversary scenarios.
3. **Worst-Case Scenario Exploration:** Assuming auxiliary data with a matching distribution, our analysis explores a worst-case scenario, where the adversary accesses highly relevant information. This reveals insights into the FL's vulnerabilities under challenging conditions, providing a comprehensive evaluation of its security properties.

[1] Hu, Hongsheng, et al. "Membership inference attacks on machine learning: A survey." *ACM Computing Surveys (CSUR)* 54.11s (2022): 1-37.

[2] Carlini, Nicholas, et al. "Membership inference attacks from first principles." *2022 IEEE Symposium on Security and Privacy (SP)*. IEEE, 2022.

[3] Jeon, Jinwoo, et al. "Gradient inversion with generative image prior." *Advances in neural information processing systems* 34 (2021): 29898-29908.

[4] Balunovic, Mislav, et al. "Lamp: Extracting text from gradients with language model priors." *Advances in Neural Information Processing Systems* 35 (2022): 7641-7654.

Reviewer comment (Theory):
[Minor] The second equation in the proof for theorem 3, Appendix A.1 should be

$$
\nabla_{z_j}L_{FL}
=\sum_{t=1}^K\alpha_t\frac{\partial\bar{h}}{\partial z_j}\log\sum_{k=1}^Ke^{z_k}+\sum_{t=1}^K\alpha_t(1-p_t)^{\gamma}p_j-\sum_{t=1}^K\alpha_t\frac{\partial\bar{h}}{\partial z_j}\textcolor{red}{z_t}-\alpha_j(1-p_j)^\gamma.
$$

$z_t$ is missing in the equation.

Author response:
Thank you for your careful observation and pointing out the mistake. We have fixed the mistake in the paper.

Reviewer comment (Evaluation):
The performance of the proposed attack in Theorem 2 relies on the quality of posterior probability. To my understanding, the posterior probability increases when the training epoch increases. Therefore, the performance should increase with the training epoch. However, in Figure 4, the performance decreases as the training epoch increases. Could you please provide more explanation?

Author response:
- In the initial phase of model training, the model lacks the ability to differentiate between samples from each class. Consequently, the model assigns similar output probabilities to all fitted samples. It's important to note that, during this stage, the posterior probabilities for each class closely approximate $1/K$, with $K$ representing the number of classes in the current classification task.
- As training progresses, the model gradually learns distinctive features of each class. For the $i$th class, probabilities of positive samples (belonging to class $i$) increase, reflecting the model's growing confidence in its predictions. Conversely, negative probabilities (samples from class $j$ where $j\neq i$) decrease toward 0.
- However, varying learning speeds among different samples lead to a distribution shift in posterior probabilities during training. This results in a bias in estimating the batch data's posterior probability using samples from the auxiliary dataset. In summary, this explains the observed drop in InsAcc during model training.

Reviewer comment (Evaluation):
In Figure 4 rightmost subfigure, why instance accuracy drops with model training?

Author response:
- In the initial phase of model training, the model lacks the ability to differentiate between samples from each class. Consequently, the model assigns similar output probabilities to all fitted samples. It's important to note that, during this stage, the posterior probabilities for each class closely approximate $1/K$, with $K$ representing the number of classes in the current classification task.
- As training progresses, the model gradually learns distinctive features of each class. For the $i$th class, probabilities of positive samples (belonging to class $i$) increase, reflecting the model's growing confidence in its predictions. Conversely, negative probabilities (samples from class $j$ where $j\neq i$) decrease toward 0.
- However, varying learning speeds among different samples lead to a distribution shift in posterior probabilities during training. This results in a bias in estimating the batch data's posterior probability using samples from the auxiliary dataset. In summary, this explains the observed drop in InsAcc during model training.