From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This work considers decentralized optimization problems where agents' local data sets are non-i.i.d. To mitigate this data heterogeneity issue, an augmentation of a pre-existing decentralized optimization method is developed based on local weighted averaging of agents' gradients, which is termed (somewhat erroneously) as gradient clustering. This method bears a number of resemblances to cross gradients as detailed in the following reference:

[1] Esfandiari, Yasaman, et al. "Cross-gradient aggregation for decentralized learning from non-iid data." International Conference on Machine Learning. PMLR, 2021.

and does not provide any specific theoretical guarantees of the proposed modification on top of prior work. Therefore, the technical novelty and innovation is not very clear. However, experimental evidence demonstrates the merits of the proposed approach in a limited manner. During the rebuttal phase, the authors have significantly expanded the degree of experimental validation, which brings the bar closer to meeting the standard of acceptance. However, without sound conceptual underpinning and an explicit relationship between the proposed methodology and the actual degree of data heterogeneity as quantified by some bias or power law effect of the data distribution, it is unclear what is the technical contribution. For this reason, the work is not yet warranting acceptance.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA

From: ICLR.cc/2023/Conference/Paper1992/Authors
title: Summary of the final additions to the rebuttal (revised paper)
comment: The following are the changes made to the revised version of the paper. We included the highlighted paper with the changes marked in blue in the supplementary folder.
* Updated the contributions to clarify the novelty of the proposed algorithm
* Updated the experimental section and appendix with an exhaustive set of experiments. All the experiments were conducted for three different seeds and the averaged test accuracy and the standard deviation are reported for statistical significance.
  * CIFAR10 on various architectures such as 5layer CNN, VGG11, and Resnet-20 over 5, 10, and 20 agents with ring and torus topologies (existing)
  * CIFAR100 trained on ResNet-20 architecture over 5 and 10 agents ring topology
  * Fashion MNIST trained on LeNet-5 architecture over 5 and 10 agents ring topology
  * Imagenette (low resolution - 32x32) trained on MobileNet-V2 over 5 and 10 agents ring topology.
    * Imagenette is a 10-class subset of the ImageNet dataset.
  * Imagenette (high resolution - 224x224) trained on ResNet-18 over 5 agents with ring and chain topology
  * AGNews (text classification) trained on BERT$_{mini}$ over 4 and 8 agents ring topology
  * AGNews trained on DistilBERT trained on 4 agents ring topology

From: ICLR.cc/2023/Conference/Paper1992/Reviewer_y4qJ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposed a new algorithm that utilizes groups of stochastic gradients collected from neighborhood nodes in a decentralized learning environment with heterogeneous data distributed in different locations. The proposed algorithm is communication efficient and achieves better empirical performance on non-IID data sampled from the CIFAR-10 Dataset.
strength_and_weaknesses: Strength: the paper is quite well-written and easy to follow.
Weakness: the working mechanism of the proposed method is not fully demonstrated. Other than the empirical performance on a benchmark data set, it is not clear to me why this method work or under what circumstances the proposed method will work best.

1. The paper has emphasized the concept of "non-IID" data. But what types of data can this method handle? It would be very ambitious to claim that it will work for any type of setting. So it is necessary to clarify or at least give some concrete examples of the non-IID data types under consideration.

2. What types of models are most suited for the proposed method? Regression problems? Classification problems? or other tasks? Different models have very different structures on the gradients and one needs to be more specific.

3. The term "clustering" in the title implies some sort of unsupervised learning algorithms which I do not see in the paper.  To the best of my understanding, the paper seems to assume that the data distributions among the neighboring nodes are somehow "homogeneous". Otherwise, I do not understand why taking a weighted average of a bunch of inhomogeneous gradients will help improve the model's performance.  Please clarify.

4. Since there are no theoretical justifications for the proposed method, I think carefully designed synthetic experiments are necessary to study the advantages and limitations of the proposed method. Otherwise, I have little confidence in the generalizability of the method.

clarity,_quality,_novelty_and_reproducibility: The presentation is very clear and easy to follow. The proposed methodology is novel but needs a more comprehensive justification.
summary_of_the_review: The paper is very well written but the supporting evidence for the model performance is not sufficient.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1992/Reviewer_LFjx
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies the decentralized distributed training under the non-iid data distribution setting. Different from the baseline method D-PSGD, it leverages self- and cross-gradient information to modify the gradient update for each agent. 
strength_and_weaknesses: Strength
1. The motivation of this paper is clear and interesting.
2. The proposed method looks reasonable.
3. Experiments have some promising results.

Weaknesses
1. As pointed out in Introduction section, the idea of leveraging cross-gradient information has already been used in Esfandiari et al. (2021). Therefore, the novelty of this paper is limited in this sense.
2. It would be great to include some convergence analysis in this paper. Many decentralized learning methods have convergence analysis, such as D-PSGD. Actually, Esfandiari et al. (2021) which also uses cross-gradient information, has convergence analysis as well.
3. The experiments are only conducted on one dataset, i.e., CIFAR10. It’s better to use more datasets. Decentralized learning algorithms are particularly useful for training large models on large distributed datasets. The authors are encouraged to use large datasets (such as ImageNet).

clarity,_quality,_novelty_and_reproducibility: Look good to me.
summary_of_the_review: The paper proposed an interesting method for decentralized learning but it has several weaknesses. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1992/Reviewer_b2jc
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper presents a new decentralized learning algorithm that bases on the proposed cross gradient aggregation (CGA) algorithm. Specifically, the authors leverage the existing self-gradient and cross-gradient concepts to develop the neighborhood gradient clustering (NGC) algorithm. This proposed method replaces the local gradients of the model with the weighted mean of the self-gradients, model-variant cross-gradients, and data-variant cross-gradients, which is better able to handle the parameter variations in non-IID scenarios. To reduce the communication bottleneck, the authors develop the compressed version of NGC, CompNGC. To validate both algorithms, the authors utilize a benchmark dataset with various model architectures and the results show the superiority of the proposed algorithms.
strength_and_weaknesses: Strength: this paper presents a novel algorithm on top of an existing one and shows the better model performance compared to the existing baseline. The investigated topic is quite interesting as non-IID remains a challenging problem in the decentralized learning area. The paper is easy to follow and the presentation is clear.

Weaknesses: the novelty in this work is quite marginal. It looks the only novelty is to replace the gradient calculation step in CGA with the weighted mean of self-gradients, model-variant cross-gradients, and data-variant cross-gradients. Such a change is just a minor incremental work, which fails to provide sufficient contributions. Also, concepts are just based on the existing works. Given this, if the authors could provide detailed theoretical analysis on NGC and CompNGC, that may make this paper technically solid and sound. However, the authors only provide empirical results, which to me is not promising as well. If this paper is posited as an applied paper, then comprehensive results are necessarily required, including diverse model architectures, datasets, tasks, topologies and numbers of agents. Through these, if NGC and CompNGC remain competitive and outperform baselines, then the work will look very technically solid and sound. However, the existing experimental results to me are not very comprehensive and promising. 

*******************************Post-rebuttal************************************
I appreciate the detailed response and additional results in the revised paper from the authors. After carefully reviewing the responses, I think the additional results have made the work better. However, the overall novelties in this work is still low to me and it still has much room for improvement in terms of theory. Though I would raise my score given the much better empirical evidences for the proposed algorithm in the revised version, the paper with the current form is still marginally below the acceptance standard. 
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the presentation in this paper look good, but the big weakness as mentioned above is the novelty, which is quite marginal. The current form of the paper cannot provide sufficient contributions. 
summary_of_the_review: This paper presents a new decentralized learning algorithm called NGC to cope with the issues in decentralized learning settings. The authors provide algorithm design and empirical evidence to show case the superiority. This work seems incremental, failing to providing good novelties. The authors should analyze the proposed algorithms theoretically and present more experimental result to validate.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold