Paper: Lightweight Unsupervised Federated Learning with Pretrained Vision Language Model (ICLR-2024)

Reviewer comment (Experiments):
The compared methods are somewhat weak baselines. Some important unsupervised FL baselines and methods are not discussed or compared in the paper, e.g., [1][2][3][4]  
- [1] Collaborative Unsupervised Visual Representation Learning from Decentralized Data. ICCV’21  
- [2] Divergence-aware Federated Self-Supervised Learning. ICLR’22.  
- [3] Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering. ICML’22  
- [4] MocoSFL: Enabling Cross-client Collaborative Self-supervised Learning. ICLR’23

Author response:
This paper tackles the novel challenge of unsupervised federated learning and presents a lightweight framework designed to address it. The framework incorporates the pretrained CLIP image encoder, fixing it to minimize the computation and communication demands on local clients. The listed works, however, focus on self-supervised federated representation learning. In contrast to our approach, these methods propose techniques to train the image encoder in a federated manner. Consequently, they are not directly relevant to the problem addressed by this paper.

Reviewer comment (Novelty):
Some papers mentioning adopting CLIP to FL are not discussed. e.g. [5][6]  
- [5] Fedclip: Fast generalization and personalization for clip in federated learning.  
- [6] When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions.

Author response:
We thank the reviewers for providing the related works and have included them in the related works part. 
It's essential to highlight that FedCLIP similarly incorporates the pretrained CLIP model into federated learning. However, it is designed to address the traditional supervised federated learning setting where labeled data are available in the clients. In contrast, our method focuses on the unsupervised federated learning setting, where only unlabeled data is available in the clients. This distinction underscores the unique contribution and applicability of our approach in scenarios with exclusively unlabeled client data.

Reviewer comment (Experiments):
What is the impact of training for more local epochs in each round?

Author response:
We conducted an additional ablation study to explore the influence of varying the number of local update epochs. The results for the two datasets, CIFAR-10 and CIFAR-100, under both homogeneous and heterogeneous settings are presented in the following table. Increasing the number of local update epochs did not lead to a notable improvement in the performance of the proposed method. The overall best performance was achieved with only 1 local update epoch. This observation underscores the lightweight property of the proposed method. Moreover, it eliminates the necessity for tuning this hyper-parameter, simplifying the implementation. This phenomenon can be attributed to the high entropy property of CLIP zero-shot prediction, emphasizing the need for stable local updating.

| # epochs | CIFAR-10 |  | CIFAR-100 |  |
|:---:|---|---|---|---|
|  | s=2 | i.i.d. | s=10 | i.i.d. |
| 1 | 72.0 | 74.0 | 43.3 | 43.2 |
| 2 | 71.7 | 73.6 | 41.4 | 41.2 |
| 3 | 72.0 | 73.1 | 37.3 | 37.4 |
| 4 | 72.4 | 73.2 | 33.0 | 31.7 |
| 5 | 73.0 | 73.2 | 28.6 | 26.6 |

Reviewer comment (Evaluation):
It may need to explain why the testing accuracy of baselines drops as the number of communication rounds increases.

Author response:
**3. Reason that testing accuracy of baselines drops**

As depicted in Figure 2 of our paper, a crucial observation regarding CLIP zero-shot prediction is that the predicted probability vectors exhibit high entropy, resembling a uniform distribution. When the CLIP image encoder is fixed, and the linear classifier is trained with strong supervision from ground-truth labels, it optimizes the classifier weights to diverge significantly from the textual embeddings. The divergence leads to biased and highly dynamic local classifiers, ultimately resulting in degraded performance. In contrast, our approach employs resilient self-training with evolving pseudo-labels to stabilize the local model updating process. Additionally, we incorporate class-balanced synthetic data generation to address the heterogeneous distribution of local data.

Reviewer comment (Experiments):
Why did not compare with the FedUL baseline?

Author response:
**4. Comparing with FedUL baseline**

As discussed in the related works section, it's crucial to highlight that FedUL necessitates precise knowledge of label frequencies for each client, making it an unfair comparison method for the proposed approach. Additionally, FedUL involves the transformation and recovery of model parameters, while the proposed lightweight framework maintains a fixed state for the image encoder. Consequently, FedUL cannot be seamlessly adapted to the proposed lightweight framework.

Reviewer comment (Novelty):
The contributions of this study did not meet the anticipated expectations. The two methods introduced lack novelty. The idea of refining pseudo-labels has been previously explored within the context of self-supervised learning. Additionally, the class-balanced data generation draws parallels with the Synthetic Minority Over-sampling Technique and can be categorized as an oversampling strategy.

Author response:
We thank the reviewer for dedicating time and effort to review our paper.

**1. Novelty of this paper**
- This paper introduces a novel setting of unsupervised federated learning and presents a lightweight framework designed to address this challenge effectively.
- Notably, our work is the first to tackle federated learning with completely unlabeled data. While prior work, such as FedUL (Lu et al, 2022), claims to handle federated learning from unlabeled data, it necessitates knowledge of precise label frequencies for each client, which is often impractical in real-world applications.
- Additionally, we propose a lightweight framework that incorporates the CLIP image encoder and text embeddings to enhance unsupervised federated learning. This framework trains a cleverly-initialized linear classifier on each client, significantly reducing computation and communication requirements.
- As an illustrative solution within this framework, we introduce a resilient self-training approach with evolving pseudo-labels to address the unsupervised learning problem. To handle heterogeneous data distribution in federated learning, we propose a class-balanced data generation approach that generates synthetic feature instances from textual embeddings.
- Experimental results demonstrate that our proposed method enhances the performance of CLIP's zero-shot prediction. Moreover, we show that our approach converges rapidly, substantially reducing the computation and communication demands on clients.

Reviewer comment (Evaluation):
While the authors highlight the lightweight nature of their proposed framework, evidence throughout the paper seems insufficient. The sole indication of its lightweight character is the use of a linear classifier during training. However, this isn't a unique aspect as the baseline methods employ the same classifier. The purported lightweight advantage of the proposed framework isn't adequately substantiated. Furthermore, a comprehensive analysis of both computation and communication costs is essential to truly label the framework as lightweight and communication efficient.

Author response:
**2. Evidence of lightweight**

This paper addresses the novel unsupervised federated learning problem and introduces a lightweight framework to address it. The lightweight nature of our approach is evident in the following aspects:
- To minimize the number of parameters optimized in each client and transmitted between clients and the server, we introduce the CLIP image encoder, fixing it during local training. Only the weight parameters of the linear layer classifier are updated and transmitted.
- To further reduce the number of communication rounds, we initialize the weight parameters of the linear layer classifier with text embeddings associated with class names. This results in a minimal number of local update epochs and communication rounds required for model convergence.

A comprehensive analysis of both computation and communication costs is presented below. Comparing the training of the full model, including the image encoder and the classifier, with the ResNet-50 based CLIP image encoder and a linear layer classifier with 10 classes (e.g., CIFAR-10), the number of parameters in the former is approximately 38 million (M), while the latter is about 10 thousand (K), making a difference of almost 3800 times.
- Computation Costs: Training the full model typically takes over 100 rounds with 5 local update epochs for each round according to FedAvg and FedNTD. This requires updating a total of $38M \times 100 \times 5$ parameters for each client. In contrast, our proposed lightweight method usually takes 10 rounds with 1 local update epoch for each round, involving the updating of only $10K \times 10 \times 1$ parameters for each client. The difference is approximately 190K times.
- Communication Costs: Transmitting the full model incurs a cost of $38M \times 100 \times 32$ bits of bandwidth, while transmitting only the linear model costs $10K \times 10 \times 32$ bits of bandwidth. The difference is about 38K times.

Reviewer comment (Experiments):
More experiments are required, especially for large scale datasets like ImageNet.

Author response:
**3. Results on more datasets**

We evaluate the proposed methods on three standard datasets: CIFAR-10, CIFAR-100, and CINIC-10, following established methods like FedAvg and FedNTD. These datasets cover various numbers of classes and are widely used for evaluating federated learning methods. While we acknowledge that including larger datasets would provide a more comprehensive evaluation, considering time limitations, we plan to explore this in future works.

Reviewer comment (Writing):
There are some typos, like FedBR (Guo et al., 2023b) FedBR (Guo et al., 2023b), tranfer, etc.

Author response:
**4. Typos**

We appreciate the reviewer for highlighting these typos, and they have been corrected in the latest revision.