Paper: Robust Training of Federated Models with Extremely Label Deficiency (ICLR-2024)

Reviewer comment (Evaluation):
The algorithm requires training an additional model, leading to increased computation and communication costs.

Author response:
**Answer 3:** Thanks for indicating this shortage of twin-sight. Twin-sight exactly increases the communication and computation overheads. Following your valuable comments, we have added a detailed explanation of the extra overhead induced by our method.
Specifically, our method doubles the training time and the communication cost, because we introduce an extra model.

Reviewer comment (Reproducibility):
There are details missing in the method and experimental description, raising concerns about reproducibility. Regarding Equation (9), the authors did not explicitly specify the choices of the metric d and the function N in both the method and the experiments section.

Author response:
**Answer 1:** Thanks for pointing out this problem. According to your kind suggestion, we have added the following explanations to our revision.

- The function $N(\cdot)$ is used to quantify the relationship among samples. Specifically, $N(\cdot)$ is a matrix that calculates the distances among samples in the feature space, i.e., outputs of supervised model $Z_s \in \mathbb{R}^{n \times d}$ and those of unsupervised model $Z_u \in \mathbb{R}^{n \times d}$ with $d$ being the dimension and $n$ the batch size. Using these outputs, we can employ the matrix $M \in \mathbb{R}^{n\times n}$to represent the relationships among samples, i.e., $N(f(\mathbf{w}_s,\mathbf{x})):=M_s = Z_s \cdot Z_s^T, \quad N(f(\mathbf{w}_u,\mathbf{x})):=M_u = Z_u \cdot Z_u^T$.

- The metric $d(\cdot)$ is realized as the mean square error, which is formulated as:
$d(N(f(\mathbf{w}_s,\mathbf{x})), N(f(\mathbf{w}_u,\mathbf{x}))) = \|| M_s - M_u\||^2$.

Reviewer comment (Experiments):
The datasets are basic, and it appears that each experiment was run only once, with the random seed not being disclosed.

Author response:
**Answer 2:** Thanks for your constructive comments. The random seed we use in our experiments is 0. Following your constructive question into account, we conducted the other two runs with different random seeds for both the baselines and twin-sight. The mean and deviation results are listed in the following: 

|                | CIFAR-10 | CIFAR-100 |  SVHN  | FMNIST |
| -------------- | -------- | --------- | ------ | ------ |
| FedAvg-Lower Bound |   $60.14\pm1.50$  |   $48.05\pm0.52$    |  $56.36\pm5.06$   |   $73.32\pm3.79$   |
| FedProx-Lower Bound |   $63.03\pm1.00$  |   $45.42\pm0.68$    |  $56.03\pm5.90$   |   $72.31\pm4.73$   |
| FedAvg+FixMatch |   $63.87\pm0.73$  |   $48.84\pm0.70$    |  $56.29\pm2.32$   |   $67.25\pm2.79$   |
| FedProx+FixMatch |   $61.61\pm0.72$  |   $43.55\pm0.46$    |  $47.71\pm3.79$   |   $65.57\pm2.08$   |
| FedAvg+Freematch |   $58.40\pm2.01$  |   $49.52\pm0.74$    |  $56.77\pm2.58$   |   $64.78\pm4.67$   |
| FedProx+Freematch |   $59.43\pm1.38$  |   $41.73\pm0.51$    |  $51.69\pm2.41$   |   $71.68\pm2.51$   |
| Fed-Consist |  62.79 $\pm0.93$  |   $47.94\pm0.70$    |  $53.74\pm2.91$   |   $66.73\pm3.41$   |
| RSCFed |   $60.11\pm0.97$  |   $45.90\pm1.32$    |  $58.46\pm3.81$   |   $76.34\pm1.09$   |
| Twin-sight (Ours) |   $70.02\pm0.77$ |   $50.15\pm0.43$    |  $63.16\pm0.30$   |   $78.85\pm1.04$   |

We'll supplement more experiments and attach these results to our revision.

Reviewer comment (Reproducibility):
What's the specific selection of d and N in the experiments?

Author response:
**Answer 1:** Thanks for pointing out this problem. According to your kind suggestion, we have added the following explanations to our revision.

- The function $N(\cdot)$ is used to quantify the relationship among samples. Specifically, $N(\cdot)$ is a matrix that calculates the distances among samples in the feature space, i.e., outputs of supervised model $Z_s \in \mathbb{R}^{n \times d}$ and those of unsupervised model $Z_u \in \mathbb{R}^{n \times d}$ with $d$ being the dimension and $n$ the batch size. Using these outputs, we can employ the matrix $M \in \mathbb{R}^{n\times n}$to represent the relationships among samples, i.e., $N(f(\mathbf{w}_s,\mathbf{x})):=M_s = Z_s \cdot Z_s^T, \quad N(f(\mathbf{w}_u,\mathbf{x})):=M_u = Z_u \cdot Z_u^T$.

- The metric $d(\cdot)$ is realized as the mean square error, which is formulated as:
$d(N(f(\mathbf{w}_s,\mathbf{x})), N(f(\mathbf{w}_u,\mathbf{x}))) = \|| M_s - M_u\||^2$.

Reviewer comment (Writing):
This paper combines existing techniques to solve the federal semi-supervision problem, but lacks the necessary references and explanations. For example, the unsupervised algorithm adopted in formula (6) is not explained. f(w_u;∙) as an unsupervised model, what are its specific working principles and objectives? What does sim(∙) mean? What kind of neighborhood relation construction function is N(∙) of formula (9), and how does it construct neighborhood relation?

Author response:
**Answer 1:** Thanks for your detailed comments and we apologize for the missing references and detailed explanations. Accordingly, we have added the following explanations to the revised paper.

- For the unsupervised methods in Twin-sight, the $f(\mathbf{w}_u;\cdot)$ refers to the unsupervised model parameterized by $\mathbf{w}_u$. We performed some prevailing self-supervised methods, like SimCLR[R1], BYOL[R2], and SimSiam[R3] and SimCLR is choosen to be the backbone of $f(\mathbf{w}_u;\cdot)$. The results of different self-supervised methods are reported in **Appendi B.2 How to determine the self-supervised method**. And the $sim(f(\mathbf{w}_u;\mathbf{x}_i),f(\mathbf{w}_u;\mathbf{x}_j))$ denote the dot product between $\ell_2$ normalized embedding of data $i$ and $j$.

- The function $N(\cdot)$ is leveraged to quantify the relationship between different samples in a mini-batch. Specifically, $N(\cdot)$ is the matrix calculating the distance among samples in the feature space, i.e., outputs of supervised model $Z_s \in \mathbb{R}^{n \times d}$ and those of unsupervised model $Z_u \in \mathbb{R}^{n \times d}$ with $d$ being the dimension and $n$ the batch size. Using these outputs, we can employ the matrix $M \in \mathbb{R}^{n\times n}$to represents the relationships among samples, i.e., $N(f(\mathbf{w}_s,\mathbf{x})):=M_s = Z_s \cdot Z_s^T, \quad N(f(\mathbf{w}_u,\mathbf{x})):=M_u = Z_u \cdot Z_u^T$.

- The metric $d(\cdot)$ is realized as the mean square error, which is formulated as:
$d(N(f(\mathbf{w}_s,\mathbf{x})), N(f(\mathbf{w}_u,\mathbf{x}))) = \|| M_s - M_u\||^2$.


We appreciate your valuable feedback. In the updated version, we will make sure to include references and provide clear explanations to enhance the overall readability and comprehensibility.

Reviewer comment (Experiments):
The experimental results of this paper lack the necessary convincing. It is mentioned in the experimental setup that the paper will report the results after 500 rounds of training of the global model. What is the meaning of Round in Tables 1 and 2? If Round represents the number of training rounds, then why is the Round of each comparison method different and none reaches 500, which may cause the baseline results not being comparable.

Author response:
**Answer 2:** Thanks for your pointing out the confusing description. We have added the following explanation and fixed the problem in our revision.

- In our experiments, we use $R$ to denote the total communication rounds, which is 500 in the setup part. This means we train all the methods in 500 rounds for a fair comparison. 
- In Tables 1 and 2, the 'Round' refers to the communication round required to reach the target accuracy. This can be seen in the footnote of Table 1. We use this metric to record the convergence rate of different methods, which means how many rounds the method can reach a target accuracy and 'None' denotes never reaching the target accuracy during $R=500$ rounds' training process.

Thanks again for your careful review.

Reviewer comment (Writing):
The authors can introduce the content of the proposed algorithm in more detail. For the parts that refer to other methods (in formula (6) and formula (9)), the authors should indicate the citation and briefly describe how them work.

Author response:
**Answer 1:** Thanks for your detailed comments and we apologize for the missing references and detailed explanations. Accordingly, we have added the following explanations to the revised paper.

- For the unsupervised methods in Twin-sight, the $f(\mathbf{w}_u;\cdot)$ refers to the unsupervised model parameterized by $\mathbf{w}_u$. We performed some prevailing self-supervised methods, like SimCLR[R1], BYOL[R2], and SimSiam[R3] and SimCLR is choosen to be the backbone of $f(\mathbf{w}_u;\cdot)$. The results of different self-supervised methods are reported in **Appendi B.2 How to determine the self-supervised method**. And the $sim(f(\mathbf{w}_u;\mathbf{x}_i),f(\mathbf{w}_u;\mathbf{x}_j))$ denote the dot product between $\ell_2$ normalized embedding of data $i$ and $j$.

- The function $N(\cdot)$ is leveraged to quantify the relationship between different samples in a mini-batch. Specifically, $N(\cdot)$ is the matrix calculating the distance among samples in the feature space, i.e., outputs of supervised model $Z_s \in \mathbb{R}^{n \times d}$ and those of unsupervised model $Z_u \in \mathbb{R}^{n \times d}$ with $d$ being the dimension and $n$ the batch size. Using these outputs, we can employ the matrix $M \in \mathbb{R}^{n\times n}$to represents the relationships among samples, i.e., $N(f(\mathbf{w}_s,\mathbf{x})):=M_s = Z_s \cdot Z_s^T, \quad N(f(\mathbf{w}_u,\mathbf{x})):=M_u = Z_u \cdot Z_u^T$.

- The metric $d(\cdot)$ is realized as the mean square error, which is formulated as:
$d(N(f(\mathbf{w}_s,\mathbf{x})), N(f(\mathbf{w}_u,\mathbf{x}))) = \|| M_s - M_u\||^2$.


We appreciate your valuable feedback. In the updated version, we will make sure to include references and provide clear explanations to enhance the overall readability and comprehensibility.

Reviewer comment (Experiments):
The authors can rereport experimental results and compare baseline performance with the same number of training rounds.

Author response:
**Answer 2:** Thanks for your pointing out the confusing description. We have added the following explanation and fixed the problem in our revision.

- In our experiments, we use $R$ to denote the total communication rounds, which is 500 in the setup part. This means we train all the methods in 500 rounds for a fair comparison. 
- In Tables 1 and 2, the 'Round' refers to the communication round required to reach the target accuracy. This can be seen in the footnote of Table 1. We use this metric to record the convergence rate of different methods, which means how many rounds the method can reach a target accuracy and 'None' denotes never reaching the target accuracy during $R=500$ rounds' training process.

Thanks again for your careful review.

Reviewer comment (Evaluation):
In the scenario of both unlabeled and labeled data, some combinations of federated algorithms and semi-supervised algorithms does not perform as well as federated learning using only labeled data, which violates normal cognition. The authors can try to explain the reasons for this phenomenon.

Author response:
**Answer 3:** Thanks for your insightful comments. To our knowledge, some FSSL methods may lose their effect or even degenerate in a severe data heterogeneity environment. A recent work [R4] also indicates that we cannot guarantee an increase in the quantity of accurately labeled pseudo-labels during training in FSSL. For example, the pseudo-labeling technology sometimes imports many noise labels at some data distribution or dataset. These noises deteriorate the performance of the global model, leading to worse performance compared to not using the unlabeled data.

Reviewer comment (Theory):
The authors can try to prove the global convergence of the proposed objective function.

Author response:
**Answer 4:** Thanks for your kind suggestion. We will leave it as our future work.

Reviewer comment (Novelty):
Since "client drift" has been observed before, how is "gradient cliff" different from it, as the first contribution is claiming the phenomenon?

Author response:
Thanks for pointing out this problem. According to your kind suggestion, we have added the following explanations to our revision.

- Client drift is caused by data heterogeneity while gradient conflict is typically induced by multiple objectives. In existing FSSL methods, the "client drift" and "gradient conflicts" often exist at the same time. 
- Claiming the phenomenon is the motivation and the (first) contribution of this work since it motivates the proposed method. We would like to note that we did overlook the significance of this point. Thus, we sincerely thank you for your in-depth comments

Reviewer comment (Theory):
When exactly will "gradient cliff" happen? Does it happen all the time when training traditional FSSL methods?

Author response:
Thanks for pointing out the potentially confusing problem. Accordingly, we have added the following explanations to our revised paper.
 
- Gradient conflict is defined as the inconsistency between gradients, i.e., have a negative cosine similarity. The inconsistency may result from multi-objectives, e.g., multi-task learning [R1, R2, R3].
- In the context of label deficiency, gradient conflict may be attributed to both the samples and objectives. Specifically, partial samples have annotations and the objective on lableled and unlableled data are typically different. Advanced methods in FSSL typically use two different objective functions to train the local classifier, one for fully-labeled clients with cross-entropy loss and another for fully-unlabeled clients by unsupervised method. In this context, these methods suffer from gradient conflicts naturally.

In our experiments, gradient conflict often happens during the training process of traditional FSSL methods because of inconsistent loss function among different clients, please see Figure 2-(a). In contrast, we introduce a dual-model paradigm, avoiding the gradient conflict issue.

**Reference:**\
[R1] Yu et al. Gradient surgery for multi-task learning. In NeurIPS, 2020.\
[R2] Liu et al. Conflict-averse gradient descent for multi-task learning. In NeurIPS, 2021.\
[R3] Wang and Tsvetkov. Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models. In ICLR, 2021

Reviewer comment (Experiments):
It is nice to consider the partially labeled scenario, which is more realistic. The proposed method can be straightforwardly applied to a more realistic scenario where each client has a different ratio of labeled data. Are there any considerable difficulties? This is only for discussion; authors need not append experiments.

Author response:
Thanks for your insightful comments. 
- Our experiments show that our method can be effortlessly applied to partially labelled scenarios. Results shown in Table 3 demonstrate the effectiveness of our method. 
- The mentioned more realistic scenario seems a promising direction for FSSL, where each client has a different ratio of labeled data. This could be regarded as the combination of partially labeled data and partially labeled clients. In this context, twin-sight can be applied in this scenario. Unfortunately, we failed to figure out such a practical scenario. We would like to explore this setting as our further work. Thanks for your enlightening advice again.

Reviewer comment (Reproducibility):
What is the neighbourhood function used? Does it consider some graph structure?

Author response:
Thanks for your valuable and insightful comments. We merely consider the distance among samples (in a batch) in the feature space. We believe employing a global relation among samples using the mentioned graph structure is a promising direction. We would like to explore it in our future work. Thanks for your kind suggestion.