From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This paper studies the use of self-supervised learning (SSL) techniques for decentralized learning, with the emphasis on the understanding of how and why it is effective. It focuses particularly on the task of decentralized SSL that enables representation learning from unlabeled data that are owned separately by multiple clients, and compares it with other standard setups for decentralized supervised learning. The paper demonstrated that decentralized SSL works pretty well even when client data are highly non-iid (i.e., heterogeneous) both empirically and theoretically. This finding holds also for labeled data settings. The model is validated on diverse tasks in computer vision. 


justification_for_why_not_higher_score: The main concern is they only include computer vision tasks in their experimental evaluation. If they could include one or two other domain applications, that will make the paper stronger.





justification_for_why_not_lower_score: Overall, the paper is excellently written and easy to follow. The level of the investigation into SSL in Federated learning is more thorough and deeper compared with other existing works. Therefore, this work has made a great contribution to the federated learning literature.

summary_of_AC-reviewer_meeting: We mainly discussed about the concern from the 5-score reviewer. His main concern is the application of the paper is too narrow focusing only on computer vision tasks. But Federated learning usually focuses on broader applications. After some discussion, we concluded that despite a not wide-enough experimental evaluation, given this is a 9-page conference paper, we don't expect this paper to include to many experiments. Also given that this work provides thorough theoretical and empirical (though only focusing on CV) analyses into why SSL under Federated learning works, we agreed that this paper can be accepted.

note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Please let us know if you have additional comments 
comment: Dear Reviewer AXwb, since the discussion period is ending soon, we would greatly appreciate it if you can let us know whether you have any additional comments. Thanks!

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thanks again for your feedback and suggestions!
comment: Dear Reviewer bG2C, We would like to thank you again for your efforts and time in providing thoughtful feedback and comments. We’ve revised the paper according to your suggestions and replied to all the questions and concerns. Have our responses addressed your comments? We are happy to address any further comments you might have. Thanks a lot!

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thanks again for you feedback and suggestions!
comment: Dear Reviewer 84PM, We would like to thank you again for your efforts and time in providing thoughtful feedback and comments. We’ve revised the paper according to your suggestions and replied to all the questions and concerns. Have our responses addressed your comments? We are happy to address any further comments you might have. Thanks a lot!

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thanks again for your suggestions and feedback! 
comment: Dear Reviewer AXwb,
We would like to thank you again for your efforts and time in providing thoughtful feedback and comments. We’ve revised the paper according to your suggestions and replied to all the questions and concerns. Have our responses addressed your comments? We are happy to address any further comments you might have. Thanks a lot!

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thanks again for your constructive feedback!
comment: Thanks a lot for the prompt reply and acknowledgement! Thanks for the suggestion, and we will definitely move the new results of FedProx to the main body after collecting everyone’s feedback and being stable about what to include in the main body (as currently the organization is a bit dense and delicate to stay perfectly within 9-page). We will make sure the paper present a coherent idea for the robustness of various types of decentralized learning algorithms. Thank you very much again!

From: ICLR.cc/2023/Conference/Paper3160/Reviewer_fhez
title: Thank you for the update
comment: Thank you for your quick update! I have confirmed the responses, revised manuscript, as well as the other reviews. The responses are overall satisfactory and have clarified my concerns. 

One remaining concern:
Although FedAvg is the very basic FL algorithm, I don't think that this necessarily mean that more advanced FL algorithms can equally benefit from (or can contribute to) the proposed self-supervised approach. I still believe that having results of various FL variants such as FedProx in the main body of the paper (i.e., rather than its appendix) is important to demonstrate that the argument of **non-IIDness can be less concerning** holds on various types of FL algorithms.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (Reference)
comment: [1] Moming Duan, Duo Liu, Xianzhang Chen, Renping Liu, Yujuan Tan, and Liang Liang. "Self-balancing federated learning with global  imbalanced data in mobile systems." IEEE Transactions on Parallel and Distributed Systems 32, no. 1 (2020): 59-71.

[2] Alaa Awad Abdellatif, Naram Mhaisen, Amr Mohamed, Aiman Erbad, Mohsen Guizani, Zaher Dawy, and Wassim Nasreddine. "Communication-efficient hierarchical federated learning for IoT heterogeneous systems with imbalanced data." Future Generation Computer Systems 128 (2022): 406-419.

[3] Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. FedBN: Federated learning on non-iid features via local batch normalization. In International Conference on Learning Representations, 2020b

[4] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429–450, 2020a

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (1/1)
comment: Thank you for appreciating the significance and contributions of our work, and for the helpful suggestions. We address your questions below.

> *Q: Other federated learning algorithms that have techniques such as personalization, and remove batch normalization for non-IID data.*

  A: Thanks for the suggestions, and we have compared and cited these works in our updated paper. For our current experiments, we were essentially running FedAvg but without averaging the layers of models that have local batch normalization, which can be viewed as an instance of FedBN in [3]. As per your request, we have also added new experiments using FedProx [4], in Sec. F.7 of our updated draft. It can be seen that the key results and main messages remain unchanged. As mentioned in the General Response, we chose FedAvg because we do not want to “dilute” the finding of the inherent properties of decentralized SSL, even in its most basic and simplest form. Other variants, including the new FeatARC algorithm we developed, would further improve the performance of Dec-SSL. 

  Finally, we would like to re-emphasize that, as you pointed out, “the performance of FedAvg is **known** to degrade when data is highly non-iid”. Meanwhile, our paper was **exactly trying to “contrast” this point** in some sense, by conveying the message that when combined with self-supervisions, **non-IIDness can be less concerning**, even for the very basic FedAvg algorithm (which is **not that known** to the FL community). 

> *Q: The effect of heterogeneity in terms of data size $|D_k|$, in the context of Eq 2.2.*

  A: Thanks for pointing it out. In fact, even in our current experiments (with Dirichlet sampling for creating distribution shifts), see Figure 9, the data size $|D_k|$ for each local dataset $k$ **is** not exactly constant. We have also added new experiments on different amounts of $D_k$ in Sec. F.5 of our updated draft and observe a performance drop when the local data is not as sufficient. In our theoretical analysis, the results still hold as long as the data size $D_k$ satisfies the big O order (they don’t need to be identical across local datasets). In the extremely heterogeneous case, for example, when the data sizes across datasets are orders of magnitude different, the performance of federated/decentralized learning is known to downgrade severely [1,2], even for supervised learning (an active research area!). Studying and mitigating its effect in decentralized “SSL” is definitely a new direction worth further investigation.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (Reference)
comment: [1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017.

[2] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pp. 4387– 4398. PMLR, 2020.

[3] Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4912–4921, 2021.

[4] Chaoyang He, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, and Salman Avestimehr. Ssfl: Tackling label deficiency in federated learning via personalized self supervision. arXiv preprint arXiv:2110.02470, 2021a

[5] Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. "Federated unsupervised representation learning." arXiv preprint arXiv:2010.08982 (2020).

[6] Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. arXiv preprint arXiv:2205.12493, 2022.

[7] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.

[8] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586– 19597, 2020.

[9] Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619, 2020.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (2/2)
comment: > *Q: How hyperparameters are chosen. Reproducibility Unclear. Not all experimental details are presented.*

   A: We note that the hyperparameters for FedAvg and the setting are included in all figures and Appendix C.1. For our new algorithm FeatARC, there are only two hyperparameters where $\lambda=1$ (equal weight) and $C=2$ (two clusters). We also note that our code has been submitted along with this paper and will be released publicly. Feel free to let us know if you have any other questions about hyperparameters.  
> *Q: Clustering idea in Sec. 5 is quite naive and ad-hoc.*

   A: To the best of our knowledge, this “clustering-based” decentralized SSL algorithm is **new** in the literature. While simple, it is more effective than many existing decentralized SSL algorithms, as we have shown in Table 2 in the paper. 

   On the other hand, the idea of “clustering” has been well-supported in decentralized “supervised” learning, so it is not a very “ad-hoc” idea, see e.g., [8,9]. It can be viewed as an efficient way to balance the bias and variance trade-off by learning “multiple” models and aggregating local data sources properly. 

   Finally, as we noted before, our focus is **not** on designing better algorithms, but, as the first-of-its-kind work in this area, on **understanding** the interesting properties of even simple/basic algorithms that exist, to try to answer **why** we should continue work on this nascent but exciting area. 

 > *Q: Reproducibility Unclear. Not all experimental details are presented.*

   A: We would like to note that the code has been submitted and will be released publicly for reproducibility. The implementation details are included in Appendix C.1. We are more than happy to answer any additional questions the reviewer might have about any implementation details.

We thank you very much for the detailed and constructive comments. We hope our responses and new results have addressed your comments. Feel free to let us know if there are any other questions.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (1/2)
comment: Thank you for appreciating our theoretical insights and finding our “Sec. 4 very important for the future of ML”. We address your specific questions as below. 


> *Q: Only computer vision tasks are focused.*

  A: We agree that we mainly focused on computer vision tasks (as we have specified in our abstract, “specifically contrastive visual representation learning”). 

  Please see our justification for the use of vision tasks in General Response. Compared to most existing works on decentralized/federated (supervised) learning [1,2,8], self-supervised learning [7], and decentralized SSL [3,4,5,6], our setup is already more sufficient. In particular, these recent decentralized SSL works [3,4,5,6] mostly only have limited experiments on either CIFAR-10 or MNIST with few-layer CNN for classification tasks. In contrast, we use ResNet trained on Imagenet-100 and MS-COCO, and also the custom Amazon robotics dataset (which is definitely a **real-world** grasping task and new in the literature) with not only classification, but also **detection** and **segmentation** tasks. Moreover, such a choice of vision tasks makes it viable to compare with the supervised learning counterpart, as in our Sec. 4.

  We also agree that extending our framework and analysis to these domains would be beneficial. We are currently working on different domains in federated graph representation learning and federated natural language processing, to further validate our results. These setups, especially NLPs are relatively large-scale and it will take some time to incorporate them into our framework. We will include them in the paper if we are able to finish them on time.

> *Q: Self-supervised learning method is adopted in a standard way.*

  A: Please refer to our General Response for re-emphasizing the focus of our work: the main point of our work is NOT to **design better decentralized SSL algorithms** to handle data heterogeneity, but to reveal the message that, even for the very basic/simple algorithms (FedAvg + SimCLR/BYOL/MAE), there are some interesting inherent properties that benefit decentralized SSL. This is in stark contrast to the supervised learning case, and provides new insights and confidence for the community to continue studying this nascent area. The **understanding** nature of the work, which itself is novel in the literature, drives us to start with the “standard” setting, so that the understandings are not diluted/biased by algorithm-design. As both decentralized learning and SSL are fast-growing areas, it seems not possible to “consider all self-supervised learning algorithms extensively”, but only consider the **most representative** and **widely used** ones, to gain some understanding first (the goal of our work). To the best of our knowledge, our work provides **the first** such understanding of decentralized SSL.

  In addition, we still designed a **new** decentralized SSL algorithm that outperforms many existing ones, and **further** improves the performance of vanilla Dec-SSL.

> *Q: No dataset was used in experiments with features with very different characteristics.*

  A: We believe that the way we created **data heterogeneity** based on class labels is standard in the decentralized/federated (supervised) learning literature [3,4,5,6,8], and has been widely used to model the data heterogeneity across clients. Note that both the **feature** and **label** heterogeneity we created in the paper have already had **very different** characteristics. More importantly, compared to other few works along this line [3,4,5,6], we have already used larger-scale and more diverse datasets such as ImageNet and MS-COCO, as well as real-world Amazon robotics datasets. Note that the Amazon dataset and MS-COCO experiments have **detection** and **segmentation** as the downstream tasks, which have very different characteristics from **classification**. We are more than happy to evaluate other datasets (as long as time permits), if the reviewer can kindly suggest any. 

  Finally, to further address your concern, we have also added new experiments regarding different characteristics, including rotation and stylization as different characteristics for each local data source (the raw data X is changed, see Fig.14 for visualization), in Sec. **F.2**, **F.3**, and **F.4** in the updated paper.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (Reference)
comment: [1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017.

[2] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pp. 4387– 4398. PMLR, 2020.

[3] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz et al. "Advances and open problems in federated learning." Foundations and Trends® in Machine Learning 14, no. 1–2 (2021): 1-210.

[4] Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4912–4921, 2021.

[5] Chaoyang He, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, and Salman Avestimehr. Ssfl: Tackling label deficiency in federated learning via personalized self-supervision. arXiv preprint arXiv:2110.02470, 2021a

[6] Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. "Federated unsupervised representation learning." arXiv preprint arXiv:2010.08982 (2020).

[7] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. "An efficient framework for clustered federated learning." Advances in Neural Information Processing Systems 33 (2020): 19586-19597.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (2/2) 
comment: > *Q: The theoretical contribution is marginal.*

  A: To the best of our knowledge, there **was no** theoretical analysis for decentralized SSL before, even in any simplified setting, to justify **why** one should use “self-supervised learning” in “decentralized learning with unlabeled data”, and what’s the inherent property of decentralized SSL. As the first step towards **understanding** SSL in this decentralized setting, we believe our theoretical insights are not marginal but in fact important and novel. Also, we note that our theoretical insights (**Theorem 3.2 and Proposition 4.1**) are more about the (solution to the) Dec-SSL objective, not about the specific optimization algorithms being used (nor the "rate" as the reviewer mentioned). In the paper, we define the representability in the linear setting and show that the solution learned by each local data source is relatively more uniform and similar to each other, and not affected by the non-IIDness in the decentralized data. These are also insightful even for the fully decentralized case where there is no centralized server and each data source can only communicate with its neighbors (see our response to Reviewer 84PM and our new **Sec. F.1** for more details). Note that a "rate" analysis would not be easily generalized to and be useful in this more decentralized case, but our insights about the solutions still can.

  Moreover, indeed “the analysis of convergence of FeatARC is missing”. However, as we pointed out in General Response, designing better **algorithms** (and their analysis) is not the focus of our paper, but a bonus, as the main message is that even the very basic decentralized SSL algorithms would inherently enjoy nice properties. In fact, our FeatARC is a “clustering-based” FL approach, which involves simultaneously “assigning the cluster identity of clients” and “optimizing the loss”. This leads to a **highly nonconvex** optimization problem that is very challenging to tackle even in the **convex FL loss** and **supervised learning** setting, not to mention that we are dealing with the SSL setting whose theory is still an active and relatively new area (even for decentralized supervised learning). The only complete convergence analysis for “clustering-based FL” was [7], for the convex loss + supervised learning setting, where only **local** convergence around the initialization can be obtained, which cannot capture the global convergence “rate” the reviewer aims to find, either. Given the good performance of our FeatARC, studying its convergence behavior can definitely be valuable future work that is worth highly non-trivial investigation. 

> *Q: Can Dec-SSL be applicable to the case where each client only owns one class?*
   
  A: Indeed, in the paper, we have tried an extreme case where beta=0 with 5 clients, and each client only has two classes. In Figure 7 (d), we have also tried 20 clients with $\alpha=0.02$, where each client has even less data than 10 client cases and is also extremely non-IID.

  To further address your concern, we have added new experiments on this “extreme” case in Sec. F.3 in the updated appendix. We show that Dec-SSL is surprisingly robust to participation ratios in this extreme case and stays robust as we go from very IID to extremely non-IID. As we can see, Dec-SSL still works even in this case. This is consistent with our theoretical insights and the main message of the paper. 

> *Q: Was Dec-SSL designed specifically for unlabeled data?*
   
  A: Initially, our Dec-SSL is designed to address the label deficiency issue in decentralized learning. Then, interestingly, we realized that it has some inherent properties that are suitable for decentralized learning, and can be favorable even in the case when labels are available. See our Sec. 4 for detailed observations and discussions. Hence, we believe Dec-SSL has more potential than just being applied to unlabeled data, but also (partially) labeled data that are **highly heterogeneous** (as our paper has suggested). 

We thank you very much for the detailed and constructive comments. We hope our responses and new results have addressed your questions. Feel free to let us know if there are any other concerns.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (1/2)
comment: Thank you for appreciating that we have “presented a new direction” that is of significance to the community. We also appreciate your constructive comments and address your questions point-by-point as below.

> *Q: The notation is confusing. Why use decentralized learning instead of federated learning?*

  A: Sorry for the confusion. There is indeed some inconsistency issue regarding terminology in the literature of this area. When we say “decentralized learning”, we meant the general definition of “learning from decentralized datasets”, which **includes** federated learning (with a central server, as the reviewer pointed out), probably the most well-known and impactful decentralized learning scheme these days. Confusingly, even for federated learning, from its Wikipedia page, there is “centralized FL” (with a central server) and “decentralized FL” (without a central server). Hence, we did not deliberately distinguish the two concepts while preparing the paper.

  We chose to use “decentralized learning” (but mainly focus on the FL setting) for the following reasons: 
  - FL with a central server is nowadays the most popular and impactful decentralized learning scheme. As among the first step towards developing and understanding **decentralized-SSL**, we started with this simplest setting; 
  - We follow the convention from several high-impact papers in the area, which in fact focused on the **FL setting only**, but still used the term **decentralized learning/learning from decentralized data**, see [2] and even the very first FedAvg paper [1] for example. While the peer-to-peer network setting the reviewer mentioned is sometimes referred to as **fully** decentralized [4], in order to differentiate it further from FL. Note that we did not claim our setting is “fully decentralized”;

  - To fully address **FL**, there are many other questions of interest, other than the “communication efficiency” and “participation rate” we considered, including **privacy** concerns, **personalization** efficiency, etc., which are not the focus of our work. We did not want these additional considerations in algorithm design to “dilute” our study of the fundamental and inherent properties of decentralized SSL;
  - While focusing on the server-client setting, we believe our observations and insights can be extended to the fully decentralized setting, since our results are more about the **objective function** used in SSL, less about how we **average** the iterates. Consensus-based fully decentralized learning can be viewed as a different way to **average** the iterates other than a simple and direct average the iterates. Indeed, our theoretical results are insights about the **solutions** to the global and local objectives, but not about the **optimization algorithms** that achieve them. 
  
  Hope our previous considerations make sense to you. To further address your concern, we have:
    - Edited the title (and related expressions) tentatively from “decentralized learning” to “learning from decentralized data”, the **same expression** as in the FedAvg paper [1]. Feel free to let us know if it sounds better.
   - We have conducted some new experiments for this p2p “fully decentralized” case with more network topology and validated our results further. See the detailed results and discussions in **Sec. F.1** in the updated appendix where we observe consistent robustness in other network topologies.

> *Q: The number of data sources is limited.*
 
  A:  We agree that 5-agent is a small number. In fact, as stated in our Table 3, we do have results for more agents, e.g., 20, with the results in Figure 4 Right, Figure 6c, and Figure 7d. We also would like to point out that in many works on decentralized self-supervised/unsupervised learning with unlabeled data, the number of 5-20 agents is common and standard, see [4,5,6] for example. 

  To further address your concerns, we have added new experiments with 50/100 agents and observed similar robustness to participation ratios and non-IIDness, in Sec. F.6 in the updated paper. We have also added results in the fully decentralized setting, with different network topologies, as discussed in detail in **Sec. F.1**.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (Reference) 
comment: [1] Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4912–4921, 2021.

[2] Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervised learning. In International Conference on Learning Representations, 2022.

[3] Chaoyang He, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, and Salman Avestimehr. Ssfl: Tackling label deficiency in federated learning via personalized selfsupervision. arXiv preprint arXiv:2110.02470, 2021a

[4] Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. arXiv preprint arXiv:2205.12493, 2022.

[5] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.

[6] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017.

[7] Chen, T., Kornblith, S., Norouzi, M. and Hinton, G., 2020, November. A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.

[8] He, K., Fan, H., Wu, Y., Xie, S. and Girshick, R., 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9729-9738).

[9] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M. and Piot, B., 2020. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33, pp.21271-21284.

[10] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P. and Joulin, A., 2020. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33, pp.9912-9924.

[11] He, K., Chen, X., Xie, S., Li, Y., Dollár, P. and Girshick, R., 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16000-16009).

[12] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pp. 4387– 4398. PMLR, 2020.

[13] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz et al. "Advances and open problems in federated learning." Foundations and Trends® in Machine Learning 14, no. 1–2 (2021): 1-210.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (2/2) 
comment: > *Q: Highlighting limitations of more uniform representations learned by Dec-SSL.*

  A: We agree that a more uniform representation may not always be better in certain settings, and there should always be a tradeoff of performance in specific tasks and generality. Regarding uniform features, the SSL community has demonstrated that more uniform features tend to transfer better (SimCLR [7], MoCo [8], BYOL [9], SwAV [10]). However, compared to supervised features which are more concentrated within the same class and more separated between classes, more uniform features could be less effective in some cases. As you suggested, for beta-VAE, higher beta applies a stronger constraint on the latent bottleneck and limits the representation capacity, keeping them disentangled and remaining efficient and more interpretable. But for better transfer learning performance, sometimes uniform features are more preferable. We have added these highlighting points in the paragraph before Section 5 in our updated version.

> *Q: Novelty: Extension/combination v.s. **Understanding**.*

  A: Please see our justification about novelty and our focus in General Response. In particular, we emphasize that our focus is \emph{not} on “algorithmic advances”, i.e., developing “better” decentralized SSL algorithms to “handle data heterogeneity”; instead, we take one step back, and asked and tried to answer “why” do we even do decentralized SSL in the first place. We observe that even the very basic and simple decentralized SSL algorithms can inherently handle data heterogeneity, and lead to better communication and participation efficiency, which are significant metrics in decentralized learning.


  To the best of our knowledge, our work provides *the first* attempt to understand and justify the use of decentralized SSL, and its unique properties. Specifically, compared to the very few and recent/contemporaneous decentralized SSL results [1,2,3,4], we are novel in providing these new insights and understandings, and in fact managed to “shift the research community towards new lines of research”, by justifying why we should do it, instead of a no-brainer “combination” of existing algorithms. Due to the “understanding” nature of our work, our focus was not on algorithm design (though we still developed a new algorithm that outperforms many existing decentralized SSL algorithms), but was on the basic ones in order to be not “distracted” by specific algorithm design, and to identify the interesting underlying property more clearly. Our experiments are also on larger-scale and even real-world datasets, compared to these works.

We thank you again for the very detailed and constructive comments. Please don’t hesitate to let us know of any additional comments on the paper or the planned changes.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: Thank you for your constructive feedback! (1/2)
comment: Thank you for finding the topic of our work is of increasing importance, and appreciating our contribution. We address your questions below. 

> *Q: Unsure whether the theoretical insights are clearly supported by empirical results besides Figure 3.*

  A: First note that the theoretical results are just providing one possible explanation about what we have observed in the experiments, in a simplified setting. In the paper, we define the representability in the linear setting and show that the solution learned by each local agent is more amenable to the averaging operation. The actual experimental results are affected by more factors, e.g., communication, optimization, nonlinearity of the model, etc. So there is no *exact* empirical counterpart. Our theoretical results are insights about the **solutions** to the global and local objectives, but not about the **optimization algorithms** that achieve them. 

  However, we emphasize that the theory in Sec. 4 **is supported** by the empirical observations in Sec. 4, where we see that Dec-SSL is more robust to non-IIDness than Dec-SLRep. The similar theory in Sec. 5 **is also supported** by the empirical observations in Sec. 5. To further address your comment, we have included a visual illustration of the feature representations in **Sec. F.2** in the appendix of our updated paper, which can be viewed as one such **empirical counterpart** of Figure 3.

> *Q: Extending the work to problem settings and domains other than tasks on visual data*

  A: We agree that we mainly focused on computer vision tasks (as we have specified in our abstract, “specifically contrastive visual representation learning”).  Please see our justification for the use of vision tasks in General Response. Compared to most existing works on decentralized/federated (supervised) learning [1,2,8], self-supervised learning [7], and decentralized SSL [3,4,5,6], our setup is already more comprehensive. In particular, these recent decentralized SSL works [3,4,5,6] mostly only have limited experiments on either CIFAR-10 or MNIST with few-layer CNN for classification tasks. In contrast, we use ResNet trained on Imagenet-100 and MS-COCO, and also the custom Amazon robotics dataset (which is definitely a **real-world** grasping task and new in the literature) with not only classification, but also **detection** and **segmentation** tasks. Moreover, such a choice of vision tasks makes it viable to compare with the supervised learning counterpart, as in our Sec. 4.

  We also agree that extending our framework and analysis to these domains would be beneficial. We are currently working on different domains in federated graph representation learning and federated natural language processing, to further validate our results. These setups, especially NLPs are relatively large-scale and it will take some time to incorporate them into our framework. We are still actively working on these experiments on tasks beyond visual data. 

> *Q: Further visualizing and comparing representations learned through Dec-SLRep, Dec-SSL, and FeatARC. Rigorously quantifying these comparisons.*

  A: As per your suggestion, we have added the visualization in **Sec. F.2**  and measured the model weight difference in the updated appendix. We observe that the local clients and the global server have more aligned representations in the feature space and less model drift in the weight space, in the Dec-SSL settings compared to the Dec-SLRep settings. The feature space between local and global data sources in FeatARC is surprisingly similar even under high non-IIDness (alpha=0.1).  

> *Q: Explaining how the Amazon data was curated and whether it is publicly available.*

  A: The details of Amazon data can be found in Sec. D in Appendix. Due to confidential issues, it has not been made public yet. But we know that Amazon researchers are working on publicizing it. We will share the public link in the paper once it is available.

> *Q: Evaluating the performance of FeatARC to other algorithms on more than just CIFAR.*

  A: We have added the experiments on comparing FeatARC to other algorithms on datasets CIFAR-100 in **Sec. F.7** in the appendix of our updated paper and observe similar robustness to non-IIDness.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: General Response to All Reviewers (Reference)
comment: [1] Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4912–4921, 2021.

[2] Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervised learning. In International Conference on Learning Representations, 2022.

[3] Chaoyang He, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, and Salman Avestimehr. Ssfl: Tackling label deficiency in federated learning via personalized self supervision. arXiv preprint arXiv:2110.02470, 2021a

[4] Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. arXiv preprint arXiv:2205.12493, 2022.

[5] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.

[6] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017.

[7] Chen, T., Kornblith, S., Norouzi, M. and Hinton, G., 2020, November. A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.

[8] He, K., Fan, H., Wu, Y., Xie, S. and Girshick, R., 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9729-9738).

[9] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M. and Piot, B., 2020. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33, pp.21271-21284.

[10] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P. and Joulin, A., 2020. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33, pp.9912-9924.

[11] He, K., Chen, X., Xie, S., Li, Y., Dollár, P. and Girshick, R., 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16000-16009).

[12] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pp. 4387– 4398. PMLR, 2020.

[13] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz et al. "Advances and open problems in federated learning." Foundations and Trends® in Machine Learning 14, no. 1–2 (2021): 1-210.

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: General Response to All Reviewers (2/2)
comment: > *Q: Sufficiency of our experiment settings:*

  A:

  **Task domains**:

   We focused on “contrastive visual representation learning” and thus the visual tasks, as already pointed out in the abstract (in order to not overclaim). We followed this convention as these recent works on decentralized SSL [1,2,3,4], and even most works on understanding general SSL [5], have mainly focused on vision tasks (Vision domain indeed has been the touchstone for many milestone SSL methods in the past several years, such as SimCLR [7], MoCo [8], BYOL [9], SwAV [10], MAE [10], and so on). We have in fact already greatly expanded the domains studied in these papers, by considering for example **detection** and **segmentation** tasks with larger networks such as ResNet and Transformer (in Appendix), in contrast with the few-layer CNN on MNIST dataset. Also note that the Amazon detection tasks are actually for suction grasping, which is a novel robotic task in the decentralized learning literature. Vision tasks have oftentimes been studied as the domain for federated/decentralized (supervised) learning and supervised learning.

  **Datasets**: 

  Similar as above, we follow the convention from existing decentralized decentralized/federated (supervised) learning [6,12,13], self-supervised learning [5,7,8,9,10,11], and decentralized SSL [1,2,3,4] literature, and use standard vision datasets as CIFAR, ImageNet, and MS-COCO. In fact, none of the existing decentralized SSL works [1,2,3,4] have even been evaluated under the large-scale datasets as ImageNet and MS-COCO as we did. Also, note that many previous related works only used few-layer CNN for classification. In contrast, we use ResNet trained on Imagenet-100, MS-COCO, and real-world amazon dataset with detection and segmentation tasks (more than just classification), which makes our validation already more comprehensive than those in the literature. The Amazon robotics dataset is definitely new in the literature.  


  **Algorithms**: 

  As mentioned above, our focus was on understanding existing algorithms, not developing a better algorithms for decentralized SSL. Although our comparison with other decentralized SSL algorithms is not perfect, we have compared with most **existing “decentralized SSL”** algorithms we were aware of in **Table 2**.

Please do not hesitate to let us know if you have any additional feedback. We are very happy to address them. Thanks!

From: ICLR.cc/2023/Conference/Paper3160/Authors
title: General Response to All Reviewers (1/2)
comment: We thank all the reviewers for the very valuable and constructive comments. We have revised the paper as suggested by the reviewers, and have addressed their comments point-by-point below (please see each individual response). We have also conducted additional experiments, organized them in the **Section F** of the updated appendix. We will also release our code for reproducibility (which has been submitted already). The section includes the following experiments:

- **Section F.1**: As Reviewer 84PM suggested,  we have added new and interesting experiments on the **fully decentralized** learning, where the local data sources are only allowed to communicate with their neighbors over a peer-to-peer network, without a centralized server.

- **Section F.2**: As suggested by Reviewer AXwb, we further visualize and compare the representation models learned from different approaches. 

- **Section F.3**: As Reviewer 84PM suggested, we have added new experiments on the extremely heterogeneous case, where each local data source only owns one class. 

- **Section F.4**: As per Reviewer bG2C's suggestion, we have added experiments with features with very different characteristics.
 
- **Section F.5**: As per Reviewer fhez's suggestion, we have added experiments with different data sizes for each local data source. 

- **Section F.6**: As per Reviewer 84pM's suggestion, we have added experiments with much more data sources. 

- **Section F.7**: As per Reviewer fhez's suggestion, we have added experiments with other decentralized learning algorithms such as FedProx. 


Overall, we appreciate that most reviewers acknowledged the contributions and significance of our work in a “new direction” (from Reviewer 84PM) of “a growing learning paradigm” (from Reviewer AXwb) and the tasks are “important for the future of ML” (from Reviewer bG2C). We address several common questions from the reviewers as below.

> *Q: The focus and novelty of our work: **Understanding** v.s. **Algorithm Design***

  A: We first clarify on some misunderstanding from some reviewers: as we pointed out in our introduction and Sec. A “Our focus” paragraph, the focus of our work is *not* simply “combining” existing decentralized learning and SSL algorithms, *nor* developing “better” decentralized SSL algorithms. In other words, we focus on **understanding** rather than new **modules** or new **algorithms**. In fact, such a combination has already been done in a few recent related works we cited therein (see Appendix Sec. A for more details). In contrast, our focus is to take one step deeper, and revisit this new paradigm, by asking and carefully answering the question about “if such a combination is really useful, in what sense, and why”. To the best of our knowledge, our work is **the first** one that has proposed and tried to answer these “understanding” questions in decentralized SSL. We hope to discover the unique properties of the problem, and provide new insights into it.

  To this end, we focused on and carefully examined the “most basic/simple” decentralized learning (FedAvg) and SSL (SimCLR, Simsiam, BYOL) approaches, in order to delineate the underlying interesting properties of the problem from advancing the algorithmic design. In particular, while several recent decentralized SSL works have focused on designing better algorithms to “handle/mitigate data heterogeneity”, we observed, via both experiments in “large-scale” synthetic *and* real-world data, as well as theoretical analyses, that such a problem can be **less intimidating** in Dec-SSL, by nature. In turn, this nice property inherently improves the efficiency of decentralized learning even using these very basic/simple algorithms, and sometimes even in the case when “labels are available”. These new insights, understandings, analyses, and implications were not characterized in these recent/contemporaneous works, and we believe them to be novel in this area to the best of our knowledge. Additionally, we developed a new Dec-SSL algorithm, FeatARC, as a bonus (as the main message is that even the very basic decentralized SSL algorithms would inherently enjoy nice properties), which further improves the performance and is better than a number of state-of-the-art algorithms. 

  These perspectives act as the guiding principle to organize our paper structure, experiment setup, and choices of algorithms, leaving a relatively small space for our new algorithm, FeatARC. Still, we appreciate the comments very much, and have improved the writings on these points as well as included more results in the updated version.

From: ICLR.cc/2023/Conference/Paper3160/Reviewer_AXwb
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper shows that decentralized self-supervised learning (Dec-SSL) can be used on unlabeled data and is robust to the heterogeneity of data aggregated over several data sources. It combines FedAvg, a decentralized learning algorithm, and SimCLR, a self-supervised representation learning algorithm, to create a Dec-SSL algorithm and evaluates this on multiple tasks for varying degrees of data heterogeneity against multiple baselines. It provides some theoretical considerations as to why Dec-SSL is robust to data heterogeneity and proposes an extension to the aforementioned Dec-SSL algorithm called FeatARC and shows that FeatARC outperforms baselines, which include ablations to FeatARC.
strength_and_weaknesses: The paper motivates the problem it addresses well--namely, the problem that much real-world data is unlabeled and can differ greatly across data sources, both of which cause issues in existing decentralized learning algorithms when compared to situations in which data is labeled or there is homogeneity across data sources. Decentralized learning is becoming more prevalent over time, so addressing this problem is becoming more important.

The primary objective of the paper is to evaluate a Dec-SSL algorithm that is a combination of two pre-existing algorithms on multiple vision tasks on large-scale visual datasets under varying degrees of heterogeneity. The paper does a good job at this. It conducts several experiments that assess the performance of Dec-SSL with respect to baselines that intuitively should and do act as "upper bounds" to Dec-SSL and show that Dec-SSL is robust to both feature and label non-IIDness (i.e., data hetereogeneity). The paper shows that Dec-SSL outperforms a decentralized supervised learning algorithm (Dec-SLRep) on object detection and semantic segmentation tasks on MS-COCO and segmentation tasks on an Amazon package segmentation dataset on multiple well-accepted metrics. Overall, the benchmarks the paper evaluates Dec-SSL on are large and diverse, and the baselines it compares Dec-SSL to are reasonable.

The paper provides theoretical insights as to why Dec-SSL outperforms Dec-SLRep in a clear, concise manner. However, I'm unsure whether the paper shows that these insights are clearly supported by empirical results. Figure 3 seems to be an artificial, illustrative example, but I don't see empirical results mirroring this example.

The paper further shows that Dec-SSL can be beneficial compared to Dec-SLRep even when labels are available by evaluating both algorithms under communication and participation constraints on ImageNet-100.

The paper finally motivates and introduces a new Dec-SSL algorithm, FeatARC, an extension to Dec-SSL that uses well-known techniques to improve multiple-model performance. FeatARC outperforms Dec-SSL on CIFAR tasks, and the paper shows ablations of the techniques used to extend FeatARC from Dec-SSL to show that these techniques consistently improve FeatARC over varying degrees of heterogeneity.

The paper would benefit from
1. Extending the work to problem settings and domains other than tasks on visual data
2. Further visualizing and comparing representations learned through Dec-SLRep, Dec-SSL, and FeatARC. Rigorously quantifying these comparisons (e.g., comparing Bhattacharyya distances between local models and global model in Dec-SSL vs FeatARC) under varying levels of heterogeneity and constraints would be interesting
3. Explaining how the Amazon data was curated and whether it is publicly available (apologies if I missed this--I didn't see it in the main body or Appendix)
4. Evaluating the performance of FeatARC to other algorithms on more than just CIFAR
5. Highlighting limitations of more uniform representations (i.e., those learned by Dec-SSL vs Dec-SLRep). Are there settings in which having such "spread out" representations leads to worse performance? What about interpretability? Would less uniform representations generally be more interpretable, distinct, or disentangled (for example, comparing VAE to Beta-VAE with beta > 1)


clarity,_quality,_novelty_and_reproducibility: The paper is primarily intended to evaluate the viability of decentralized learning with unlabeled data and data heterogeneity. The clarity of the paper is generally excellent. The quality is very good but is limited to the visual domain and common vision tasks. The novelty is limited, as most of it is in the form of experimental design and theorems of limited importance instead of algorithmic advances (as explained above for Dec-SSL and FeatARC) or deep insights that shifts the research community towards new lines of research. The reproducibility of the work is excellent--the experimental design, algorithms, optimizers, architectures, hyperparameters, etc. are clearly presented in the main body and Appendix.
summary_of_the_review: The paper motivates addressing an important problem in a growing learning paradigm, explores this problem well in a specific domain, and proposes theoretical insights and an extension to an algorithm that's the combination of two existing algorithms. The paper is clearly written and understandable, even for those with limited experience in decentralized learning. However, the paper is limited by its use of only the visual domain and lack of results on representations learned, especially since it is primarily evaluatory of existing algorithms and minor extensions of these algorithms.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3160/Reviewer_84PM
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents a novel decentralized learning algorithm based on the recently developed self-supervised learning algorithms to address the heterogeneous data issues. Specifically, the authors applied the SSL to the Federated Learning settings and presented conclusions based on the empirical findings from a couple popular benchmarking datasets. They claimed that the Dec-SSL approach is robust against the data heterogeneity. Additionally, they provided theoretical insights into understanding why data heterogeneity is not a significant concern for Dec-SSL. They also designed a new algorithm called FeatARC to make learn multiple models based on clusters. The empirical evidences validated their claims in the paper.

strength_and_weaknesses: Strength:

This paper presented a new direction that involves the self-supervised learning techniques to address the unlabeled and heterogeneous data issues in decentralized learning settings. Indeed, unlabeled data in decentralized learning has not sufficiently been explored, while the past years have witnessed the extensive studies on the decentralized supervised learning. The insight from this work seems new for the community as it shows that the proposed Dec-SSL framework is robust against data heterogeneity. Also, the authors provided sufficient and promising empirical evidences. Overall the paper is easy to follow and organized well.

Weaknesses:

1. The notation is confusing. Throughout the whole paper, the authors just focused on Federated Learning settings, which are NOT decentralized. The significant difference between these two settings is that for FL, communication only happens between server and workers, while for decentralized settings, there is no server and communication happens among workers. Even though they might share some similarities, in theory and practice, results can be completely different. So I wonder why the authors emphasized decentralization in this work. Probably FL is more appropriate in this work. Sine the title can be somewhat confusing. Also, if the setting changes, would the conclusions drawn from the results still hold true? The simpliest correction would be to focus on FL.

2. The number of data sources is limited. It looks like throughout the paper, the authors always used 5 agents in the empirical results. Given the current status in FL and decentralized learning, 5 is too small and the relevant results are not that convincing as there is no clue to know if the same conclusions would apply to a large number of agents. For example, if CIFAR 100 and 50 or 100 agents were used, would Dec-SSL still perform robustly? Given that this paper is more applied and that conclusions are more based on experimental results, it is better for the authors to run the same experiments with a large number of agents/workers. Another issue that naturally emerges is that in decentralized learning, different topologies are quite critical. The authors should consider this topic in the paper and see how the topology would affect the performance of Dec-SSL framework. 

3. The theoretical contribution is marginal. Though the authors have provided the theoretical insights for Dec-SSL. However, the analysis of convergence of FeatARC is missing. Even if in the paper we have seen the slight model performance improvement over the baselines, that could be attributed to noise or variance. Then theoretical bound is important here since we can know if the proposed algorithm has the similar rate or not.

4. Can Dec-SSL be applicable to an extreme case, where each agent only owns a unique class in the real decentralized learning, instead of the presented FL in this work? We have seen that many existing algorithms have failed under this scenario. Additionally, was Dec-SSL designed specifically for unlabeled data? Or it might perform better if the data was unlabeled or partially unlabeled?

*******************************Post-rebuttal********************************
I appreciate the detailed responses, careful revisions and additional results from the authors. After carefully reviewing the responses and clarification from the authors, I think most of my concerns have been addressed. Though there is some room for the authors to improve the theoretical analysis in this work, I feel it has provided some good insights for the community on the decentralized learning by leveraging SSL. While I would like to raise my score, the authors should still consider other comments from other reviewers to make the draft more technically solid and sound. 
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of this paper are decent. Particularly, the empirical novelty looks great.
summary_of_the_review: This paper presents a novel decentralized learning algorithm based on the recently developed self-supervised learning algorithms to address the heterogeneous data issues. The authors concluded that in their proposed Dec-SSL framework, the data heterogeneity is less of a concern. The author designed a new algorithm to run multiple models based on clusters and provided some theoretical insights. Overall, this paper provides some new insights on FL with unlabeled data and good empirical evaluation. However, there is confusion in learning settings and the theoretical contribution is marginal, which requires more work from the authors to make it technically solid and sound.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3160/Reviewer_bG2C
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: A decentralized self-supervised learning is proposed that learns representations from non-IID unlabeled data. The paper presents empirical studies on the robustness to different types of heterogeneity, communication constraints, and partial participation of data sources, as well as theoretical findings. 
strength_and_weaknesses: Strengths

- Theoretical insights are provided to support the design choices
- Well written paper
- Extensive ablation studies and analyses
- Sec. 4 is very important for the future of ML.

Weaknesses

- The experimental section is quite weak. Only computer vision tasks are focused, with synthetic distortions. There are many real-world datasets that suffer from being non-i.i.d. with decentralized learning being important, such as from healthcare and finance. 
- Self-supervised learning is adopted in a very standard way, the methodological novelty is low. Not all self-supervised learning methods are considered extensively extensively.
- There are bold claims about robustness to data heterogeneity but demonstrations are quite weak. There is no dataset used in experiments with features with very different characteristics.
- It is unclear how hyperparameters are chosen.
- Clustering idea in Sec. 5 is quite naive and ad-hoc.
clarity,_quality,_novelty_and_reproducibility: Clarity - Overall pretty good but the paper organization can be improved.

Quality - Pretty decent, except experimental sections.

Novelty - Not high, straightforward extension.

Reproducibility - Unclear. Not all experimental details are presented.
summary_of_the_review: Because of the weaknesses mentioned above, primarily on experiments not being extensive in supporting the claims, I do not suggest acceptance of this paper. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: Section D is great but so disconnected from the paper.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3160/Reviewer_fhez
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies the use of self-supervised learning (SSL) techniques for decentralized learning, with the emphasis on the understanding of how and why it is effective. It focuses particularly on the task of decentralized SSL that enables representation learning from unlabeled data that are owned separately by multiple clients, and compares it with other standard setups for decentralized supervised learning. The paper demonstrated that decentralized SSL works pretty well even when client data are highly non-iid (i.e., heterogeneous) both empirically and theoretically. This finding holds also for labeled data settings. Furthermore, the paper presents a new decentralized SSL method named FeatARC, which groups client data into several clusters to train multiple global models, and confirmed that it worked further better than FedAvg.
strength_and_weaknesses: ## Strong points
- Overall, the paper is excellently written and easy to follow. The paper is very well organized, and the connections to related work, proof of theoretical claims, and details of experiments are well documented in the appendix, making the paper complete.
- The proposed work is well motivated. Although enabling SSL in decentralized (or federated) learning setups is a known topic with some existing work, to my knowledge there has been no study at this level of why SSL should work through extensive empirical evaluation and theoretical analysis. This would seem to be a solid technical contribution.
- The fact that SSL works well for representation learning even from highly non-iid and unlabeled data, which was supported both empirically and theoretically, is a nice finding, because collecting labeled data is often identified as difficult in federated learning. 
- Experimental evaluation is thorough. Decentralized SSL is compared against decentralized SL approaches under a variety of conditions, such as multiple datasets, multiple levels of non-iidness, multiple SSL methods (SimCLR, SimSiam, and BYOL), and multiple hyperparameters for federated learning algorithms (FedAvg). 

## Weak points
- Although the experimental result is already extensive, this work could have been further stronger if it confirmed the effectiveness of DecSSL with other federated learning algorithms. Currently, all the results are obtained using FedAvg, a very standard algorithm of federated learning. On the other hand, the performance of FedAvg is known to degrade when data is highly non-iid [a] or when the model includes batch normalization [b]. It remains not perfectly obvious if the current results hold only for FedAvg or various other algorithms such as FedProx and FedBN. This is a critical question to me to judge the significance of this work.
- Another that was not very clear in the paper is the effect of heterogeneity in terms of data size |D_k|. As shown in Eq 2.2, this heterogeneity is considered in the global objective. How does it affect the performance of DecSSL empirically and theoretically? 

[a] Li et al., "Federated Optimization in Heterogeneous Networks", MLSys 2020

[b] Li et al., "FedBN: Federated Learning on Non-IID Features via Local Batch Normalization", ICLR 2021
clarity,_quality,_novelty_and_reproducibility: - **Clarity**: The paper is very clearly written. The motivation, approaches, and results of the work are all clear.
- **Quality**: Overall, I think the work is of very high quality with excellent presentation and sufficient technical contributions.
- **Novelty**: As far as I know, this level of detailed experimental evaluation and theoretical analysis for how SSL works well under decentralized learning setups is novel.
- **Reproducibility**: Details of experimental setups are presented in the appendix. Code is also submitted. 
summary_of_the_review: I think that this seems to be a solid piece of work with sufficient technical contributions and excellent presentation. Nevertheless, I also believe that this work could become further stronger if other federated learning algorithms (e.g., FedProx, FedBN) are also evaluated. Otherwise it remains unclear if the provided results hold just for a specific algorithm of FedAvg or can work in general.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper