Paper: FedAnchor: Enhancing Federated Semi-Supervised Learning with Label Contrastive Loss (ICLR-2024)

Reviewer comment (Novelty):
In the introduction (P2), the authors highlight the potential drawback of some existing works, citing "heavy traffic for data communication". However, the proposed method still necessitates the communication of embeddings of all server data to all active clients in each communication round. While the cost is lower compared to the aforementioned existing methods, the communication overhead remains considerable.

Author response:
- Concern about the communication load: The communication load is minor (please refer to A.3, Table 2 in our revised manuscript).

Reviewer comment (Experiments):
Too many hyper-parameters. In the local training (Section 3.3) step, various components contribute to the combined loss function. Parameters such as the threshold value, hyper-parameters for the Beta distribution, strong augmentation method, and size of the mixed dataset, among others, are introduced. However, the paper does not clarify how these hyper-parameters are selected or their potential effects on model performance.

Author response:
- Concern about too many hyper-parameters: The hyper-parameters of beta, augmentation method, and size of the mixed dataset remain the same as the existing papers (such as FixMatch, MixMatch, and SemiFL). As for the threshold value, it could be dataset-dependent. We selected the value 0.6 on CIFAR 10 and kept the same for other datasets to show the generalization of our method.

Reviewer comment (Evaluation):
The experiment results in Table 1 for CIFAR100 with 10000 anchor data are puzzling, as the performance of the semi-supervised learning method markedly surpasses that of the supervised learning method in the IID case. This anomaly warrants clarification.

Author response:
- Concern about the supervised baseline: For a fair comparison, we used a standard supervised baseline, where we did not conduct strong augmentation and the model was not pre-trained with anchor data (as mentioned in the caption of Table 1). Implementing a strong augmentation method is not so common in the benchmark results (such as in FedAdam [1], Fjord [2], and ZeroFL [3]), but for semi-supervised learning, people normally use a strong augmentation method, which we do the same.

Reviewer comment (Presentation):
The description for Figure 3 is unclear. It is not apparent what the lines represent. Additionally, it is ambiguous whether pseudo-label accuracy and classification accuracy are averaged over all datasets or pertain to a single client. Furthermore, the legends for the two datasets are inconsistent.

Author response:
- Description for Figure 3: Please see more explanations in the revised version of the paper, which we highlighted in Section 4.3.

Reviewer comment (Novelty):
There is a notable absence of related work such as [1] -- [4]. These works should be compared to or at least discussed in the related work.

Author response:
- Concern about the absence of related work: Thanks for pointing out the papers. We have added FedCon as a baseline (please kindly refer to the general response point number 1 to all reviewers for more details) and included others in the background in our revised manuscript.

Reviewer comment (Novelty):
The claimed main novelty is the label contraction loss that maximizes the cosine similarity for samples with the same label while minimizing the cosine similarity for samples with different labels. However, this method has been proposed as supervised contrastive learning [1]. Other modules of the proposed method exhibit limited novelty, such as pseudo-labeling and mix-up.

Author response:
Concern about novelty: We acknowledge that the general idea of label contrastive loss is not new, but we are the first to leverage it in federated semi-supervised learning to improve the quality of pseudo-labeling. Besides, all existing papers in federated/centralized SSL use the model-predicted results to generate pseudo labels. In our method, we for-the-first-time utilize the latent representation’s cosine similarity for pseudo-labeling and achieve a new SOTA performance. This provides a new solution for the general semi-supervised learning area.

Reviewer comment (Experiments):
The authors seemed to overlook many outstanding works in the literature [2,3]. For instance, the authors involved one method as the baseline method. Moreover, many related works are not considered baseline methods [4].

Author response:
Concern about missing literature: Thanks for pointing out the papers. For [2], it is an earlier paper than SemiFL and performs worse than SemiFL, as we mentioned in the background section of our paper. Additionally, we have added FedCon as a baseline in our revised manuscript and included pFedKnow, FedIL, and FedSiam in the background section.

Reviewer comment (Evaluation):
The experimental results are confusing. The proposed method seems to outperform the method with supervised information (See Table 1). This makes the results not convincing.

Author response:
Concern about the supervised baseline: For a fair comparison, we used a standard supervised baseline, where we did not conduct strong augmentation and the model was not pre-trained with anchor data (as mentioned in the caption of Table 1). This is the main reason supervised baseline performs poorly in some cases, especially when the anchor set is bigger.

Reviewer comment (Experiments):
Another concern is whether it makes sense in federated learning for a client to download the latent representations of the anchor data from the server to generate pseudo-labels. Federated learning methods often put data privacy at the first priority, so it may be not practical to disclose the complete latent representations of the data on server to each client. Moreover, according to the pseudo-labelling procedure defined in the paper, the obtained latent representations of the anchor data have to be compared with the latent representation of each local data. Under this setting, when the server data and the client's data scale up to a large amount, the effort of pseudo-labelling will be significant. Therefore, I doubt if FedAnchor can be applied to real scenarios.

Author response:
- Concerns about the privacy of downloading the latent representations of the anchor data. In our paper, we assume that a small amount of labeled data will be stored on the server, which we call anchor data. We think it is a realistic and practical setup and is mentioned in the introduction. Even if we don’t assume the anchor data is public, since the size of the anchor embedding is significantly smaller than the raw data (we chose 128 in our experiments), it alone is not sufficient to reverse-engineer and reconstruct the raw data.

Reviewer comment (Experiments):
Furthermore, the method section mentions that FedAnchor adopts mixup during the local training on clients. The mixup operation seems to be a crucial trick to improve the accuracy of FedAnchor as shown by the results of Table 1. Why are there no additional ablation studies in the paper on the effect of mixup on FedAnchor convergence and accuracy? Can FedAnchor's performance still be ahead of SemiFL when mixup is not available?

Author response:
- Concern about the use of Mixup. Mixup is commonly used in centralized semi-supervised learning settings and is a SemiFL component. We have shown the effect of mixup in our original manuscript (Table 1, Figure 2, 4, 5), named FedAnchor and FedAnchor (mixup).

Reviewer comment (Presentation):
The font sizes of the legends and axis scales in the figures are too small to read.

Author response:
- Regarding the font sizes, we will address this for the camera-ready version.

Reviewer comment (Writing):
In equation 9, it should be “(x_{fix}, y^{fix})” instead of “(x_{mix}, y^{fix})” if my understanding about the method is not wrong.

Author response:
- Response to Equation 9. Our original equation is correct. Since $x_{mix}$ is already a mixup of $x^{fix}$ and $x^{mix}$, the $y$, as you mentioned, should be as “(x_{mix}, y^{fix})”  for the first part.

Reviewer comment (Novelty):
The true contribution of this paper needs to be more clear. Generating pseudo-labels by comparing the similarities between the model representations of unlabeled data and label anchor data (or class prototypes) is not entirely new, e.g., [1--3]. FedSSL is also studied in [4].

Author response:
Concern about the novelty: We acknowledge that the general idea of latent representation and cosine similarity is not new in machine learning, which has been used in different areas and applications. The references provided by the reviewer mainly focus on dealing with data/label noise, which is not the concern of our paper. We for-the-first-time leverage the idea of latent representation’s cosine similarity to generate pseudo-labels in semi-supervised learning under the FL setup and achieve a new SOTA performance. The provided references can be good motivation papers, which we have cited in the background section (please see our new manuscript). Also, the method FedCon borrows the idea of using 2 models from BYOL without implementing any contrastive loss, despite the name.

Reviewer comment (Novelty):
The paper seems to borrow FixMatch and MixMatch, which is fine. But the contributions in addition to using both algorithms in FL need to be clarified.

Author response:
Contributions in addition to using FixMatch and MixMatch: 1. A unique pseudo-labeling method in FSSL described above; 2.  A novel label contrastive loss to further improve the quality of pseudo labels.

Reviewer comment (Experiments):
An important baseline is missing. [4] should be compared in experiments.

Author response:
FedCon baseline: We have added this comparison in Table 1, section 4.2. Please kindly refer to general response point number (1) for this point.