Paper: A Mutual Information Perspective on Federated Contrastive Learning (ICLR-2024)

Reviewer comment (Evaluation):
The empirical results show that the proposed federated SimCLR variant is only en par with spectral constrastive learning when using a user verification loss. This is not properly discussed.

Author response:
-	*Empirical results*: we consider FedSimCLR as one instance of a method that optimizes a global pretraining objective, stemming from the following intuition. Since on each client the model contrasts between datapoints of the same distribution, a way to close the gap to the centralized model training is by having the model also distinguishing between users as well, i.e., a user-verification objective. In the case of mutual information this becomes precise due to the user-verification task acting as a lower bound to the gap between the global and local MI. Furthermore, we also demonstrate theoretically at Proposition 3 that closing the gap with such an auxiliary objective is beneficial, as in itself is a lower bound to the, unavailable, label  classification objective. Based on these intuitions from our perspective and theory, we do extend both Spectral CL and SimSiam in a similar manner (i.e., Spectral CL + UV and SimSiam +UV) and we do observe that the results generalize similarly. However, **Spectral CL does better only when incorporating the results from our theory and is worse than FedSimCLR otherwise (e.g., compare FedSimCLR against Spectral CL without the UV loss)**. Now as to why Spectral CL + UV does better than FedSimCLR, this could be attributed the fact that in the centralized setting, Spectral CL can be better than SimCLR [2] and, through the modifications stemming from our intuitions and theory in the case of MI, can also do better in specific federated settings. Exploring alternative dependency metrics such as [6] in the federated setting is definitely interesting and something we will explore in future work.

Reviewer comment (Novelty):
How does the semi-supervised variant of SimCLR relate to pseudo-labeling approaches, such as distributed distillation [2] and federated co-training [1]?

Author response:
-	*Relation to pseudo-labelling approaches*: while from our perspective there are no pseudo-labels, we do agree that having these results as well will help providing a better overall picture. To this end, we are running experiments with two pseudo-label approaches, SemiFed and CBAFed, and so far we observe similar outcomes: the UV loss motivated by our theory is beneficial in the case of label skew.

Reviewer comment (Evaluation):
Please discuss the empirical results, in particular the fact that FedSimCLR is not outperforming the baselines, in more detail. The proposed method does not have to outperform the baselines, as long as the benefits and limitations of it in comparison with existing methods are properly discussed. To stress this point: it has become usual to require papers to have large tables where the proposed method has "the best number" in each row, but this just promotes scientifically questionable practices to improve the numbers. I am happy that this paper presents more interesting results, but they require, unfortunately, a more thorough discussion. One could even, for example, use mutual information as quality measure (where one would probably rely on the more tractable Wasserstein dependency measure [6], isntead of approximating MI).

Author response:
-	*Empirical results*: we consider FedSimCLR as one instance of a method that optimizes a global pretraining objective, stemming from the following intuition. Since on each client the model contrasts between datapoints of the same distribution, a way to close the gap to the centralized model training is by having the model also distinguishing between users as well, i.e., a user-verification objective. In the case of mutual information this becomes precise due to the user-verification task acting as a lower bound to the gap between the global and local MI. Furthermore, we also demonstrate theoretically at Proposition 3 that closing the gap with such an auxiliary objective is beneficial, as in itself is a lower bound to the, unavailable, label  classification objective. Based on these intuitions from our perspective and theory, we do extend both Spectral CL and SimSiam in a similar manner (i.e., Spectral CL + UV and SimSiam +UV) and we do observe that the results generalize similarly. However, **Spectral CL does better only when incorporating the results from our theory and is worse than FedSimCLR otherwise (e.g., compare FedSimCLR against Spectral CL without the UV loss)**. Now as to why Spectral CL + UV does better than FedSimCLR, this could be attributed the fact that in the centralized setting, Spectral CL can be better than SimCLR [2] and, through the modifications stemming from our intuitions and theory in the case of MI, can also do better in specific federated settings. Exploring alternative dependency metrics such as [6] in the federated setting is definitely interesting and something we will explore in future work.

Reviewer comment (Experiments):
Please state what exactly the supervised baseline in your experiments is (I assume FedAvg). Please compare to (one of the) FL variants for non-iid data, such as FedProx [4], FedBN [5], and SCAFFOLD [3], as baselines.

Author response:
-	*Supervised baseline*: you are partially correct, in that the supervised baseline corresponds to variant FedAvg. Specifically, we use the superior server-side averaging FedAdam proposed by [1], as we explain in Appendix A. In [1] the authors show empirically that this approach is strictly superior to SCAFFOLD for the cross-device setting, which is why we have not included SCAFFOLD. We have experimented with FedProx but did not see any effect of $\mu>0$ for our choice of $E=1$ across experiments unless setting $\mu>>1$ to detrimental effect. FedBN is a proven FL algorithm under covariate shift, however it prescribes the existence of BatchNorm layers in the network, which we choose to omit. FedBN further assumes the existence of a data-set for computing statistics on a new client, which is an assumption standard FedAvg (FedAdam) does not require with GroupNorm. In summary, we believe that FedAdam is a strong supervised baseline under the given assumptions.

Reviewer comment (Experiments):
For the semi-supervised setting, please compare to pseudo-labeling approaches [1,2] as semi-supervised baselines.

Author response:
-	*Relation to pseudo-labelling approaches*: while from our perspective there are no pseudo-labels, we do agree that having these results as well will help providing a better overall picture. To this end, we are running experiments with two pseudo-label approaches, SemiFed and CBAFed, and so far we observe similar outcomes: the UV loss motivated by our theory is beneficial in the case of label skew.  

We also uploaded a revision to our submission that includes in the appendix results with the pseudo-labelling baselines where we generally observe that:

- They tend to underperform compared to the other methods
- They improve by adding the user-verification loss motivated from our mutual information perspective

Reviewer comment (Theory):
The paper lacks convergence analysis of their optimization algorithm, which is quite common in FL papers.

Author response:
We appreciate your fast response to our additional results. Unfortunately, given our limited compute resources, more than preliminary results cannot be done in the timeframe of this rebuttal. We generally observe that the benefit of the UV loss manifests later in training. The federated nature of our simulation makes these experiments take approximately two weeks for 30k rounds and even though we started them as soon as possible, we cannot wait for full convergence in the time for this rebuttal. 

We would kindly ask the reviewer to not dismiss the evidence we bring for the other datasets and - in fact - already show the benefits of our method for TinyImagenet when considering Spectral CL in the unsupervised setting as well as across all methods for the semi-supervised setting, **without any hyper-parameter tuning**. We believe to have addressed SimSiam performance in the main text (section 4), please let us know if there are remaining questions. 

Rest assured that we will update the camera-ready version of the paper with a full suite of numbers to satisfy our own demands for rigorous experimental evidence.  

The benefits of our method can be condensed to allowing the federated training procedure to exploit correlation between the source of non-iid-ness and the downstream task in the semi/un-supervised setting. We make no claims about convergence or communication rounds - in fact, we inherit theoretical guarantees from FedAvg directly.

Thanks for spending the time on our paper

Reviewer comment (Theory):
Discussion on how their objective can be adapted to other centralized pretraining objectives is missing. (See questions)

Author response:
-	*Applying our objective to other centralized pretraining objectives*: we have indeed demonstrated how our insights apply to other pretraining objectives, be it contrastive, i.e., SimCLR (which is an instance of InfoNCE) and Spectral CL, or non-contrastive, i.e., SimSiam, by intuitively using the same structure for the loss (see the paragraph above the “Unsupervised Setting” in the experiment section). We will expand more about this in the appendix. Indeed, in doing so we observe similar patterns: In the fully unsupervised case with Spectral CL we have better performance in the presence of label skew when adding the UV loss. We also extended these baselines in the same way to the semi-supervised setting, i.e., applying the pretraining objective between elements that belong to the same class while adding an auxiliary classification head that predicts the correct class for those datapoints where it is available. In this scenario we also observe similar improvements, with our SimSiam modifications and the UV loss we significantly improve performance. Extending to feature contrastive approaches such as Barlow Twins is an interesting direction for future work.

Reviewer comment (Experiments):
How does the proposed MI LB/relaxation work if we move slightly away from SimCLR and look at related objectives: InfoNCE, or even non-contrastive ones like Barlow Twins? Are federated versions of these similar to Federated SimCLR?

Author response:
-	*Applying our objective to other centralized pretraining objectives*: we have indeed demonstrated how our insights apply to other pretraining objectives, be it contrastive, i.e., SimCLR (which is an instance of InfoNCE) and Spectral CL, or non-contrastive, i.e., SimSiam, by intuitively using the same structure for the loss (see the paragraph above the “Unsupervised Setting” in the experiment section). We will expand more about this in the appendix. Indeed, in doing so we observe similar patterns: In the fully unsupervised case with Spectral CL we have better performance in the presence of label skew when adding the UV loss. We also extended these baselines in the same way to the semi-supervised setting, i.e., applying the pretraining objective between elements that belong to the same class while adding an auxiliary classification head that predicts the correct class for those datapoints where it is available. In this scenario we also observe similar improvements, with our SimSiam modifications and the UV loss we significantly improve performance. Extending to feature contrastive approaches such as Barlow Twins is an interesting direction for future work.