N EIGHBORHOOD G RADIENT C LUSTERING : A N E FFICIENT D ECENTRALIZED L EARNING M ETHOD FOR N ON -IID D ATA D ISTRIBUTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Decentralized learning algorithms enable the training of deep learning models over large distributed datasets generated at different devices and locations, without the need for a central server.
In practical scenarios, the distributed datasets can have significantly different data distributions across the agents.
The current state-of-the-art decentralized algorithms mostly assume the data distributions to be Independent and Identically Distributed (IID).
This paper focuses on improving decentralized learning over non-IID data distributions with minimal compute and memory overheads.
We propose Neighborhood Gradient Clustering (NGC) , a novel decentralized learning algorithm that modifies the local gradients of each agent using self- and cross-gradient information.
Cross-gradients for a pair of neighboring agents are the derivatives of the model parameters of an agent with respect to the dataset of the other agent.
In particular, the proposed method replaces the local gradients of the model with the weighted mean of the self-gradients, model-variant cross-gradients (derivatives of the received neighbors’ model parameters with respect to the local dataset - computed locally) and data-variant cross-gradients (derivatives of the local model with respect to its neighbors’ datasets - received through communication).
The data-variant cross-gradients are aggregated through an additional communication round without breaking the privacy constraints of the decentralized setting.
Further, we present CompNGC , a compressed version of NGC that reduces the communication overhead by 32 × through cross-gradient compression.
We demonstrate the efficiency of the proposed technique over non-IID data sampled from various vision and language datasets trained on diverse model architectures, graph sizes and topologies.
Our experiments demonstrate that NGC and CompNGC either remain competitive or outperform (by 0 − 6% ) the existing state-of-the-art (SoTA) decentralized learning algorithm over non-IID data with significantly less compute and memory requirements.
Further, our experiments show that the model-variant cross-gradients information available locally at each agent can improve the performance over nonIID data by 1 − 35% without any additional communication cost.
I NTRODUCTION
The remarkable success of deep learning is mainly attributed to the availability of humongous amounts of data and compute power.
Large amounts of data is generated on a daily basis at different devices all over the world which could be used to train powerful deep learning models.
Collecting such data for centralized processing is not practical because of the communication and privacy constraints.
To address this concern, a new interest in developing distributed learning algorithms
Agarwal & Duchi (2011) has emerged.
Federated learning (centralized learning) Koneˇcn`y et al.
(2016) is a popular setting in the distributed machine learning paradigm, where the training data is kept locally at the edge devices and a global shared model is learnt by aggregating the locally computed updates through a coordinating central server.
Such a setup requires continuous communication with a central server which becomes a potential bottleneck Haghighat et al.
This has motivated the advancements in decentralized machine learning.
Decentralized machine learning is a branch of distributed learning which focuses on learning from data distributed across multiple agents/devices.
Unlike Federated learning, these algorithms assume that the agents are connected peer to peer without a central server.
It has been demonstrated that decentralized learning algorithms
Lian et al.
(2017) can perform comparable to centralized algorithms on benchmark vision datasets.
Lian et al.
present Decentralised Parallel Stochastic
Gradient Descent (D-PSGD) by combining SGD with gossip averaging algorithm Xiao & Boyd (2004).
Further, the authors analytically show that the convergence rate of D-PSGD is similar to its centralized counterpart Dean et al.
Balu et al.
(2021) propose and analyze Decentralized Momentum Stochastic Gradient Descent (DMSGD) which introduces momentum to D-PSGD.
Assran et al.
(2019) introduce Stochastic Gradient Push (SGP) which extends D-PSGD to directed and time varying graphs.
Tang et al.
; Koloskova et al.
(2019) explore error-compensated compression techniques (Deep-Squeeze and CHOCO-SGD) to reduce the communication cost of
P-DSGD significantly while achieving same convergence rate as centralized algorithms.
Aketi et al.
(2021) combined Deep-Squeeze with SGP to propose communication efficient decentralized learning over time-varying and directed graphs.
Recently, Koloskova et al.
(2020) proposed a unified framework for the analysis of gossip based decentralized SGD methods and provide the best known convergence guarantees.
The key assumption to achieve state-of-the-art performance by all the above mentioned decentralized algorithms is that the data is independent and identically distributed (IID) across the agents.
In particular, the data is assumed to be distributed in a uniform and random manner across the agents.
This assumption does not hold in most of the real-world applications as the data distributions across the agent are significantly different (non-IID) based on the user pool Hsieh et al.
The effect of non-IID data in a peer-to-peer decentralized setup is a relatively under-studied problem.
There are only a few works that try to bridge the performance gap between IID and non-IID data distributions for a decentralized setup.
Note that, we mainly focus on a common type of non-IID data, widely used in prior works Tang et al.
Hsieh et al.
Lin et al.
(2021); Esfandiari et al.
(2021): a skewed distribution of data labels across agents.
Tang et al.
(2018) proposed D 2 algorithm that extends D-PSGD to non-IID data distribution.
However, the algorithm was demonstrated on only a basic LENET model and is not scalable to deeper models with normalization layers.
Lin et al.
(2021) replace local momentum with Quasi-Global Momentum (QGM) and improve the test performance by 1 − 20% .
But the improvement in accuracy is only 1 − 2% in case of highly skewed data distribution as shown in Aketi et al.
Most recently, Esfandiari et al.
(2021) proposed
Cross-Gradient Aggregation (CGA) and a compressed version of CGA (CompCGA), claiming stateof-the-art performance for decentralized learning algorithms over completely non-IID data.
CGA aggregates cross-gradient information, i.e., derivatives of its model with respect to its neighbors’ datasets through an additional communication round.
It then updates the model using projected gradients based on quadratic programming.
The cross-gradient and self-gradient terms are formally defined in Section 3.
CGA and CompCGA require a very slow quadratic programming step Goldfarb & Idnani (1983) after every iteration for gradient projection which is both compute and memory intensive.
This work focuses on the following question: Can we improve the performance of decentralized learning over non-IID data with minimal compute and memory overhead?
In this paper, we propose Neighborhood Gradient Clustering ( NGC ) to handle non-IID data distributions in peer-to-peer decentralized learning setups.
Firstly, we classify the gradients available at each agent into three types, namely self-gradients, model-variant cross-gradients, and data-variant cross-gradients (see Section 3).
The self-gradients (or local gradients) are the derivatives computed at each agent on its model parameters with respect to the local dataset.
The model-variant crossgradients are the derivatives of the received neighbors’ model parameters with respect to the local dataset.
These gradients are computed locally at each agent after receiving the neighbors’ model parameters.
Communicating the neighbors’ model parameters is a necessary step in any gossip based decentralized algorithm Lian et al.
The data-variant cross-gradients are the derivatives of the local model with respect to its neighbors’ datasets.
These gradients are obtained through an additional round of communication.
We then cluster the gradients into a) model-variant cluster with self-gradients and model-variant cross-gradients, and b) data-variant cluster with self-gradients and data-variant cross-gradients.
Finally, the local gradients are replaced with the weighted average of the cluster means.
The main motivation behind this modification is to account for the high variation in the computed local gradients (and in turn the model parameters) across the neighbors due to the non-IID nature of the data distribution.
The proposed technique has two rounds of communication at every iteration to send model parameters and data-variant cross-gradients which incurs 2 × communication cost compared to traditional decentralized algorithms (D-PSGD).
To reduce the communication overhead, we propose compressed version of NGC ( CompNGC ) by compressing the additional round of cross-gradient communication.
Moreover, if the weight associated with data-variant cluster is set to 0 then NGC does not require an additional round of communication.
We validate the performance of proposed algorithm on CIFAR-10 dataset over various model architectures and graph topologies.
We compare the proposed algorithm with D-PSGD, CGA, and CompCGA and show that we can achieve superior performance over non-IID data compared to the current state-of-the-art approach.
We also report the order of communication, memory and compute overheads required for NGC and CGA as compared to D-PSGD.
Contributions:
In summary, we make the following contributions
• We propose Neighborhood Gradient Clustering ( NGC ) for decentralized learning setting that utilizes self-gradients, model-variant
cross-gradients and data-variant cross-gradients to improve the learning over non-IID data distribution (label-wise skew) among agents.
• We present compressed version of Neighborhood Gradient Clustering ( CompNGC ) that reduces the additional round of cross-gradients communication by 32 × .
• Our experiments show that the proposed method either outperforms by 0 − 6% or remains competitive with significantly less compute and memory requirements compared to current state-of-the-art decentralized learning algorithm over non-IID data at iso-communication cost.
• We also show that when the weight associated with data-variant cross-gradients is set to 0 , NGC performs 1 − 35% better than D-PSGD without any communication overhead by utilizing locally available model-variant cross-gradients information.
B ACKGROUND
In this section, we provide the background on decentralized learning algorithm with peer-to-peer connections.
The main goal of the decentralized machine learning is to learn a global model using the knowledge extracted from the locally generated and stored data samples across n edge devices/agents while maintaining the privacy constraints.
In particular, we solve the optimization problem of minimizing global loss function f ( x ) distributed across n agents as given in equation.
Note that
f i is a local loss function (for example, cross entropy loss) defined in terms of the data sampled ( d i ) from the local dataset D i at agent
i with model parameters x i .
[Equation]
This is typically achieved by combining local stochastic gradient descent Bottou (2010) with global consensus based gossip averaging Xiao & Boyd (2004).
The communication topology in this peerto-peer setup is modeled as a graph G = ([ n ] , E ) with edges { i, j } ∈ E if and only if agents i and j are connected by a communication link exchanging the messages directly.
We represent N ( i ) as the neighbors of i including itself.
It is assumed that the graph G is strongly connected with self loops i.e., there is a path from every agent to every other agent.
The adjacency matrix of the graph
G is referred as a mixing matrix W where w ij is the weight associated with the edge { i, j } .
Note that, weight 0 indicates the absence of a direct edge between the agents.
We assume that the mixing matrix is doubly-stochastic and symmetric, similar to all previous works in decentralized learning.
For example, in a undirected ring topology, w ij = 13 if j ∈ { i − 1 , i, i + 1 } .
Further, the initial models and all the hyperparameters are synchronized in the beginning of the training.
Algorithm.in appendix describes the flow of D-PSGD with momentum.
The convergence of the Algorithm.assumes the data-distribution across the agents to be Independent and Identically Distributed (IID).
N EIGHBORHOOD G RADIENT C LUSTERING
We propose the Neighborhood Gradient Clustering (NGC) algorithm and a compressed version of
NGC which improve the performance of decentralized learning over non-IID data distribution.
NGC utilizes the concepts of self-gradient and cross-gradient Esfandiari et al.
The following are the definitions of self-gradient and cross-gradient.
Self-Gradient:
For an agent i with the local dataset D i and model parameters x i , the self-gradient is the gradient of the loss function f
i with respect to the model parameters x i , evaluated on mini-batch d i sampled from dataset D i .
[Equation]
Cross-Gradient: For an agent i with model parameters x i connected to neighbor j that has local dataset D j ,
the cross-gradient is the gradient of the loss function f j with respect to the model parameters x i , evaluated on mini-batch d j sampled from dataset D j .
[Equation]
Note that the cross-gradient g ij is computed on agent j using its local data after receiving the model parameters x i from its neighbouring agent i and is then communicated to agent i .
3. T HE NGC ALGORITHM
The flow of the Neighborhood Gradient Clustering ( NGC ) is shown in Algorithm.
1 and the form of the algorithm is similar to D-PSGD Lian et al.
(2017) presented in Algorithm. Algorithm
Input: Each agent i ∈ [1 , n ] initializes model weights x (0) i , learning rate η , averaging rate γ , mixing matrix W = [ w ij ] i,j ∈ [1 ,n ] , NGC mixing weight α , and I ij are elements of n × n identity matrix, N ( i ) represents neighbors of i including itself. Each agent simultaneously implements the T RAIN ( ) procedure procedure T RAIN ( ) 2.
for t= 0 , 1 , . . .
, T − 1 do 3.
d ti ∼ D
i
// sample data from training dataset
[Equation]
[Equation]
[Equation]
The main contribution of the proposed NGC algorithm is the local gradient manipulation step (line 12 in Algorithm. 1).
In the t th iteration of NGC , each agent i calculates its self-gradient g ii .
Then, agent i ’s model parameters are transmitted to all other agents ( j ) in its neighborhood, and the respective cross-gradients are calculated by the neighbors and transmitted back to agent i .
At every iteration after the communication rounds, each agent i has access to self-gradients ( g ii ) and two sets of cross-gradients: 1) Model-variant cross-gradients :
The derivatives that are computed locally using its local data on the neighbors’ model parameters ( g ji ).
Data-variant cross-gradients :
The derivatives (received through communication) of its model parameters on the neighbors’ dataset ( g ij ).
Note that each agent i computes and transmits cross-gradients ( g ji ) that act as model-variant cross-gradients for i and as data-variant cross-gradients for j .
We then cluster the gradients into two groups namely: a) Model-variant cluster { g ji ∀ j ∈ N ( i ) } that includes self-gradients and modelvariant cross-gradients, and b) Data-variant cluster { g ij ∀ j ∈ N
( i ) } that includes self-gradients and data-variant
cross-gradients.
The local gradients at each agent is replaced with the weighted average of the above defined cluster means as shown in Equation.
4, which assumes uniform mixing matrix ( w ij = 1 /m ; m = |N ( i ) | ).
The mean of model-variant cluster is weighted by (1 − α ) and the mean of data-variant cluster is weighed by α where α ∈ [0 , 1] is a hyper-parameter referred as NGC mixing weight.
[Equation]
The motivation for this modification is to reduce the variation of the computed local gradients across the agents.
In IID settings, the local gradients should statistically resemble the cross-gradients and hence simple gossip averaging is sufficient to reach convergence.
However, in the non-IID case, the local gradients across the agents are significantly different due to the variation in datasets and hence the model parameters on which the gradients are computed.
The proposed algorithm reduces this variation in the local-gradients as it is equivalent to adding two bias terms ϵ and ω with weights ( 1 − α ) and α respectively as shown in Equation.
[Equation]
The bias term ϵ compensates for the difference in a neighborhood’s self-gradients caused due to variation in the model parameters across the neighbors.
Whereas, the bias term ω compensates for the difference in a neighborhood’s self-gradients caused due to variation in the data distribution across the neighbors.
We hypothesis and show through our experiments that addition of these bias terms to the local gradients improves the performance of decentralized learning over non-IID data by accelerating the global convergence.
Note that if we set α = 0 in the NGC algorithm then it does not require an additional communication round (no communication overhead compared to D-PSGD).
3. T HE C OMPRESSED NGC A LGORITHM
The NGC algorithm at every iteration involves two-steps of communication with the neighbors: 1) communicate the model parameters, and 2) communicate the cross-gradients.
This communication overhead can be a bottleneck in a resource-constrained environment.
Hence we propose a compressed version of NGC using Error Feedback SGD (EF-SGD)
Karimireddy et al.
(2019) to compress gradients.
We compress the error-compensated self-gradients and cross-gradients frombits (full-precision) to 1 bit by using scaled signed gradients.
The error between the compressed and non-compressed gradient of the current iteration ( e tji in the algorithm) is added as feedback to the gradients in the next iteration before compression.
The pseudo code for CompNGC is shown in
Algorithm.
−
E XPERIMENTS
In this section, we analyze the performance of the proposed NGC and CompNGC techniques and compare them with the baseline D-PSGD algorithm Lian et al.
and state-of-the-art CGA and
CompCGA methods Esfandiari et al.
Experimental Setup: The efficiency of the proposed method is demonstrated through our experiments on diverse set of datasets, model architectures, tasks, topologies and numbers of agents.
We present the analysis on – (a) Datasets (Appendix A.2): vision datasets (CIFAR-10, CIFAR100, Fashion MNIST and Imagenette Husain (2018)) and language datasets (AGNews Zhang et al. (b) Model architectures (Appendix A.3): 5-layer CNN, modified VGG-11, ResNet-20, LeNet-5, MobileNet-V2 and BERT mini (c) Tasks: Image and Text Classification. (d) Topologies:
Ring and Torus. (e) Number of agents: varying from 4 to 20.
Note that we use low resolution ( 32 × 32 ) images of Imagenette dataset for the experiments in Table.
The results for high resolution ( 224 × 224 ) Imagenette are presented in Appendix.
A.6 We consider an extreme case of non-IID distribution where no two neighboring agents have same class.
This is referred as complete label-wise skew or 100% label-wise non-IIDness Hsieh et al.
In particular for a 10-class dataset such as CIFAR-10 - each agent in a 5 agents system has data from 2 distinct classes, each agent in a 10 agents system has data from an unique class.
For a 20 agent system two agents that are maximally apart share the samples belonging to a class.
We report the test accuracy of the consensus model averaged over three randomly chosen seeds. The details of the hyperparameters for all the experiments are present in Appendix.
A.4.
We compare the propose method with isocommunication baselines.
The experiments on NGC ( α = 0 ) are compared with D-PSGD, NGC with CGA, and CompNGC with CompCGA.
The communication cost for each experiment in this section is presented in Appendix A.8.
Results: We evaluate variants of NGC and CompNGC and compare it with respective baselines in
Table.
1, for training different model architectures on CIFAR-10 over various graph sizes and topologies.
We observe that NGC with α = 0 consistently outperforms D-PSGD for all model architecture, graph sizes and topologies with a significant performance gain varying from 3 − 35% . Our experiments show that NGC and CompNGC has superior performance over CGA and CompCGA and the performance gains are more pronounced when considering larger graphs (with 20 agents) and compact models such as ResNet-20.
We further demonstrate the generalizability of the proposed method by evaluating its performance on classification tasks from various data distributions such as Fashion MNIST, Imagenette and on challenging dataset such as CIFAR-100.
Table. 2 shows that
NGC with α = 0 outperforms D-PSGD by 2 − 13% across various datasets where as NGC and CompNGC remain competitive with an average improvement of ∼ 1% . To show the effectiveness of the proposed method across different modalities, we present results on text classification task in
Table 3. We train on BERT mini model with AGNews dataset distributed over 4 and 8 agents. For
NGC α = {+
 we see an maximum improvement+} of 2 . 1 % over the baseline D-PSGD algorithm. Even for the text classification task, we observe NGC and CompNGC to be competitive with CGA and CompCGA methods.
These observations are consistent with the results on the image classification tasks.
We presents experiments with a larger transformer model (DistilBert base ) in Appendix A.7.
Finally, through these exhaustive set of experiments, we demonstrate that the weighted averaging of data-variant and model-variant cross-gradients can be served as a simple plugin to boost the performance of the decentralized learning over label-wise non-IID data.
Further, locally available model-variant cross-gradients information at each agent can be efficiently utilized to improve the decentralized learning with no communication overhead.
[Table] Analysis: We show the convergence characteristics of the proposed algorithm over IID and Non-IID data distributions in Figure.
1a, and 1b respectively.
For Non-IID distribution, we observe that there is a slight difference in convergence (as expected) with slower rate for sparser topology (ring graph) compared to denser counterpart (fully connected graph).
Figure.
1c shows the comparison of the convergence characteristics of the NGC algorithm with the current state-of-the-art CGA algorithm.
We observe that NGC has lower validation loss than CGA for same decentralized setup.
Analysis for 10 agents is presented in Appendix A.5.
The change in average validation accuracy with NGC mixing weight α is shown in Figure.
We observe that α close to 1 has the best performance as the model variance bias is taken care in the gossip averaging step.
We also plot the model variance and data variance bias terms for both NGC and CGA techniques as shown in Figure.
3a, and 3b [Table] [Table] respectively.
We observe that both the model variance and the data variance bias for NGC are significantly lower than CGA.
This is because CGA gives more importance to self-gradients as it updates in the descent direction that is close to self-gradients and is positively correlated to datavariant cross-gradients.
In contrast, NGC accounts for the biases directly and gives equal importance to self-gradients and data-variant cross-gradients, thereby achieving superior performance.
[Figure] Hardware Benefits: The proposed NGC algorithm is superior in terms of memory and compute efficiency (see Table. 4), while having equal communication cost as compared to CGA .
Since NGC involves weighted averaging, we do not need any additional buffer to store the cross-gradients.
Weighted cross-gradients can be added to the the self-gradient buffer.
CGA stores all the cross- [Figure] gradients in a matrix form for quadratic programming projection of local gradient.
Therefore, NGC has no memory overhead compared to the baseline D-PSGD algorithm, while CGA requires additional memory equivalent to the number of neighbors times model size.
Moreover, the quadratic programming projection step Goldfarb & Idnani (1983) in CGA is much more expensive in terms of compute and latency as compared to weighted averaging step of cross-gradients in NGC .
Our experiments clearly show NGC is superior to CGA in terms of test accuracy, memory efficiency, compute efficiency and latency.
[Table]
O [Table] m s N
[Table] O ( N i m s ) O (3 N i FP + QP )
CompCGA
O ( m s N i b )
O ((1 + 1 b ) N i m s ) O (3 N i FP + QP ) NGC α = 0 (ours) O (3 N i FP + m s N i )
NGC (ours) O ( m s N i )
O (3 N i FP + m s N i )
CompNGC (ours) O ( m s N i b ) O ( N i m s ) O (3 N i FP + m s N i )
C ONCLUSION
Enabling decentralized training over non-IID data is key for ML applications to efficiently leverage the humongous amounts of user-generated private data.
In this paper, we propose the Neighborhood Gradient Clustering ( NGC ) algorithm that improves decentralized learning over non-IID data distributions.
Further, we present a compressed version of our algorithm ( CompNGC ) to reduce the communication overhead associated with NGC .
We validate the performance of the proposed techniques ( NGC and CompNGC ) over different model architectures, graph sizes and topologies.
Finally, we compare the proposed algorithms with the current state-of-the-art decentralized learning algorithm over non-IID data and show superior performance of our algorithm with significantly less compute and memory requirements.
A A PPENDIX A. D ECENTRALIZED L EARNING S ETUP
The traditional decentralized learning algorithm
d-psgd) is described as Algorithm.
For the decentralized setup, we use undirected ring and undirected torus graph topologies with uniform mixing matrix.
The undirected ring topology for any graph-size has 3 peers per agent including itself and each edge has a weight of 13 .
The undirected torus topology with 10 agents has 4 peers per agent including itself and each edge has a weight of 14 .
The undirected torus topology 20 agents has 5 peers per agent including itself and each edge has a weight of 15 .
n ] initializes model weights x (0)
, learning rate η , averaging rate γ mixing matrix W = [ w ij ] i,j ∈ [1 ,n ] , and I ij are elements of n × n identity matrix.
simultaneously implements the procedure
We highlight the various assumptions required for the convergence of the proposed NGC and CompNGC algorithm.
Graph Structure:
The graph topology of the agents is strongly connected and symmetric.
Doubly stochastic mixing matrix:
The mixing matrix W is a real doubly stochastic matrix i.e. 1 T W = W 1 = W , where 1 is the vector of all 1 ′ s .
Lipschitz Gradients: Local loss functions f i ( . ) has L-lipschitz gradients for all i ∈ [1 , n ] i.e. || F i ( x ) − F i ( y ) ||
≤ L || x − y
||
∀ x, y ∈ R d 4.
Bounded Variance: The variance of the stochastic gradients are assumed to be bounded.
E d i ∼ D i ||∇ f
i ( x ; d i ) − ∇ F i ( x ) || 2 ≤ σ 2 , (Inner variance).n (cid:80) ni
=1 ||∇ F i ( x ) − ∇ f ( x ) || 2 ≤ ζ 2 ∀
i, x (Outer variance).
Initialization: The model parameters on all the agents are initialized to the same random values.
It may be noted that assumptions 1-5 are similar to that of CGA algorithm Esfandiari et al.
and are commonly used in most decentralized training algorithms.
A. D ATASETS
In this section, we give a brief description of the datasets used in our experiments.
We use a diverse set of datasets each originating from a different distribution of images to show the generalizability of the proposed techniques.
CIFAR-10:
CIFAR-10 Krizhevsky et al.
(2014) is a image classification dataset with 10 classes.
The image samples are
colored
(3 input channels) and have a resolution of 32 ×
There
are
, 000 training
samples
with
5000 samples per class and 10 ,
000 test
samples
with
1000 samples per class.
CIFAR-100: CIFAR-100 Krizhevsky et al.
(2014) is a image classification dataset with 100 classes.
The image samples
are colored (3 input channels) and have a resolution of 32 × 32 .
There are 50 ,
training samples with 500 samples per class and 10 , 000 test samples with 100 samples per class.
CIFAR-100 classification is a harder task compared to CIFAR-10 as it has 100 classes with very less samples per class to learn from.
Fashion MNIST:
Fashion MNIST Xiao et al.
(2017) is a image classification dataset with 10 classes.
The image samples are in grey scale (1 input channel) and have a resolution of 28 × 28 .
There are 60 , 000 training samples with 6000 samples per class and 10 , 000 test samples with 1000 samples per class.
Imagenette: Imagenette Husain (2018) is a 10 class subset of the ImageNet dataset.
The image samples are in colored (3 input channel) and have a resolution of 224 × 224 .
There are 9469 training samples with roughly 950 samples per class and 3925 test samples.
We conduct our experiments on two different resolutions of Imagenette dataset – a) a resized low resolution of 32 × 32 and, b) a full resolution of 224 × 224 .
The Imagenette experiments reported in Table.
2 use the low resolution images where as experiments in Table.
10 use the full resolution images.
AGNews: We use AGNews Zhang et al.
(2015) dataset for Natural Language Processing (NLP) task.
This is a text classification dataset where the given text news is classified into 4 classes, namely ”World”, ”Sport”, ”Business” and ”Sci/Tech”.
The dataset has a total of 120000 and 7600 samples for training and testing respectively, which are equally distributed across each class.
A. N ETWORK A RCHITECTURE
We replace ReLU+BatchNorm layers of all the model architectures with EvoNorm-S0 Liu et al.
(2020) as it was shown to be better suited for decentralized learning over non-IID distributions Lin et al.
5 layer CNN: The 5 layer CNN consists of 4 convolutional with EvoNorm-S0 Liu et al.
as activation-normalization layer, 3 max pooling layer and one linear layer.
In particular, it hasconvolutional layers with 32 filters, a max pooling layer, then 2 more convolutional layers withfilters each followed by another max pooling layer and a dense layer with 512 units.
It has a total of 76 K trainable parameters.
Modified VGG-11: We modify the standard VGG-11 Simonyan & Zisserman (2014) architecture by reducing the number of filters in each convolutional layer by 4 × and use only one dense layer with 128 units.
Each convolutional layer is followed by EvoNorm-S0 as activation-normalization layer and it has 0 .
58 M trainable parameters.
Modified ResNet-20: For ResNet-20 He et al. (2016), we use the standard architecture with 0 .
27 M trainable parameters except that BatchNorm+ReLU layer is replaced by EvoNorm-S0. LeNet-5: For LeNet-5 LeCun et al.
(1998), we use the standard architecture with 61 , 706 trainable parameters. MobileNet-V2: We use the the standard MobileNet-V2 Sandler et al. (2018) architecture used for
CIFAR dataset with 2 .
3 M parameters except that BatchNorm+ReLU layer is replaced by EvoNormS0.
BERT mini : For BERT mini Devlin et al.
we use the standard model from the paper.
We restrict the sequence length of the model to 128.
The model used in the work hence has 11 .
07 M parameters.
DistilBERT base : For DistilBERT base Sanh et al.
we use the standard model from the paper.
We restrict the sequence length of the model to 128.
The model used in the work hence has 66 .
67 M parameters.
All the experiments were run for three randomly chosen seeds. We decay the learning-rate by 10x after 50% and 75% of the training, unless mentioned otherwise. Hyper-parameters for CIFAR-10 on 5 layer CNN: All the experiments that involve 5layer CNN model (Table. 1) have stopping criteria set to 100 epochs.
We decay the learning rate by 10 × in a multiple steps at 50 th and 75 th epoch.
Table 5 presents the α , β , η , and γ corresponding to the ngc mixing weight, momentum, learning-rate and gossip averaging rate. For all the experiments, we use a mini-batch size of 32 per agent. The stopping criteria is a fixed number of epochs.
We have used
Nesterov momentum of 0.9 for all CGA and NGC experiments where as D-PSGD and NGC with α = 0 has no momentum.
[Table]
Hyper-parameters for CIFAR-10 on VGG-11 and ResNet-20: All the experiments for CIFARdataset trained on VGG-11 and ResNet-20 architectures (Table. 1) have stopping criteria set toepochs. We decay the learning rate by 10 × in a multiple steps at 100 th and 150 th epoch.
Tablepresents the α , β , η , and γ corresponding to the ngc mixing weight, momentum, learning-rate and gossip averaging rate. For all the experiments, we use a mini-batch size of 32 per agent.
Hyper-parameters used for Table. 2: All the experiments in Table. 3 have stopping criteria set to 100 epochs.
We decay the learning rate by 10 × in a multiple steps at 50 th and 75 th epoch.
Table 7 presents the α , β , η , and γ corresponding to the ngc mixing weight, momentum, learningrate and gossip averaging rate. For all the experiments related to Fashion MNIST and Imagenette (low resolution of (32 × 32) ), we use a mini-batch size of 32 per agent.
For all the experiments related to CIFAR-100, we use a mini-batch size of 20 per agent.
Hyper-parameters used for Table. 3: All the experiments in Table. 3 have stopping criteria set to 3 epochs.
We decay the learning rate by 10 × at 2 nd epoch.
Table 8 presents the α , β , η , and γ corresponding to the ngc mixing weight, momentum, learning-rate and gossip averaging rate. For all the experiments, we use a mini-batch size of 32 per agent on AGNews dataset.
Hyper-parameters for Figures: The simulations for all the figures are run for 300 epochs.
We scale the learning-rate by a factor or 0 .
981 after each epoch to obtain a smoother curve.
All the experiment in Figure 1, 2, 3, 4 and 5 use 5 layer CNN network.
Experiments in Figure 1, and 3 use 5 agents while the experiments in Figure 2, 4 and 5 use 10 agents.
The hyper-parameters for the simulations for all the plots are mentioned in Table
A. A NALYSIS FOR 10 A GENTS
We show the convergence characteristics of the proposed NGC algorithm over IID and Non-IID data sampled from CIFAR-10 dataset in Figure.
4a, and 4b respectively.
For Non-IID distribution, we observe that there is a slight difference in convergence rate (as expected) with slower rate for sparser topology (undirected ring graph) compared to its denser counterpart (fully connected graph).
Figure.
4c shows the comparison of the convergence characteristics of the NGC technique with the [Table] [Table] [Table]
[Table]
[Table] 1 . 0 ) ( − , [Table] [Table]
[Table]
[Table]
[Table]
( − ,
[Table]
[Table]
[Table]
) ( − ,
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
( − , 0 . 9 , 0 . 01 ,
[Table]
0 ) ( − , 0 . 9 , 0 . 1 , 0 . 5 ) ( 0 . 0 , 0 . 0 , 0 . 01 , 0 . 5 )
[Table]
ours) ( 1 . 0 , 0 . 9 , 0 .
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
( − ,
[Table]
[Table]
[Table]
1 ) ( −
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
( − , 0 . 9 , 0 . 01 , 0 . 1 ) ( − , 0 . 9 ,
[Table]
[Table]
[Table]
[Table]
[Table]
ours
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] current state-of-the-art CGA algorithm. We observe that NGC has lower validation loss than CGA for same decentralized setup indicating its superior performance over CGA. We also plot the model variance and data variance bias terms for both NGC and CGA techniques as shown in Figure.
5a, and 5b respectively.
We observe that both model variance and data variance bias for NGC are significantly lower than CGA. A. R ESULTS ON FULL RESOLUTION I MAGENETTE
We present the results related to full resolution (224 × 224) Imagenette in this sub-section (shown in Table. 10).
We train the full resolution 10-class subset of ImageNet on ResNet-18 architecture with 11 M parameters over a 5-agent directed ring topology.
Our experiments show that NGC outperforms CGA and CompNGC remain competitive to CompCGA for the lar-scale models as well.
We also show that NGC with α = 0 improves the performance by 27% over the baseline without any communication overhead.
This shows that the proposed methods outperform their respective baselines even for large-scale models and datasets. [Table] [Table] A. R ESULTS ON AGN EWS ON D ISTIL B ERT
Table 11 shows the results for AGNews dataset with DistilBERT base model.
The model description for DistilBERT base can be found in Appendix A.3.
The hyperparamerters for the experiments can be found in Table 8 and Communication cost can be found in Table 14.
A. C OMMUNICATION C OST :
In this section we present the communication cost per agent in terms of Gigabytes of data transferred during the entire training process (refer Tables.
The D-PSGD and NGC with α = 0 have the lowest communication cost ( 1 × ).
We emphasize that NGC with α = 0 outperforms D-PSGD in decentralized learning over label-wise non-IID data for same communication cost.
NGC and CGA have 2 × communication overhead compared to D-PSGD where as CompNGC and CompCGA have 1 .
03 × communication overhead compared to D-PSGD.
The compressed version of NGC and CGA compresses the second round of cross-gradient communication to 1 bit.
We assume the full-precision
[Table] [Figure] [Figure]
cross-gradients to be of 32-bit precision and hence the CompNGC reduces the communication cost by 32 × compared to NGC .
A. R ESOURCE C OMPARISON
The communication cost, memory overhead and compute overhead for various decentralized algorithms are shown in Table.
The D-PSGD algorithm requires each agent to communicate model parameters of size m s with all the N i neighbors for the gossip averaging step and hence has a communication cost of O (
m s N i ) .
In case of NGC and CGA, there is an additional communication round [Table] [Table] D-PSGD
[Table]
[Table]
[Table] and [Table] [Table]
[Table]
[Table]
NGC α
= 4.
[Table]
CGA
[Table]
[Table]
and
[Table]
NGC
[Table]
[Table]
[Table]
CompCGA 18.
[Table]
[Table]
and 9.
[Table]
[Table]
CompNGC
[Table]
[Table]
[Table]
[Table]
[Table]
D
[Table]
and
[Table]
NGC
[Table]
[Table]
CGA and
[Table]
NGC (ours)
[Table]
CompCGA and
[Table]
[Table]
CompNGC (ours)
[Table]
[Table]
for sharing data-variant cross gradients apart from sharing model parameters for gossip averaging step.
So, both these techniques incur a communication cost of O (2 m s N i ) and therefor an overhead of O ( m s N i ) compared to D-PSGD.
CompNGC compresses the additional round of communication involved with NGC from b bits to 1 bit.
This reduces the communication overhead from O ( m s N i ) to O ( m s N i b ) .
CGA algorithm stores all the received data-variant cross-gradients in the form of a matrix for quadratic projection step.
Hence, CGA has a memory overhead of O ( m s N i ) compared to DPSGD.
NGC does not require any additional memory as it averages the received data-variant crossgradients into self-gradient buffer.
The compressed version of NGC requires an additional memory of O ( m s N i ) to store the error variables e ji (refer Algorith. 2).
CompCGA also needs to store error variables along with the projection matrix of compressed gradients.
Therefore, CompCGA has a memory overhead of O ( m s N i + m s N i b ) .
Note that memory overhead depends on the type of graph topology and model architecture but not on the size of the graph.
The memory overhead for different model architectures trained on undirected ring topology is shown in Table.
The computation of the cross-gradients (in both CGA and NGC algorithms) requires N i forward and backward passes through the deep learning model at each agent. This is reflected as O (3 N i FP )
[Table] [Table]
in the compute overhead in Table. 4.
We assume that the compute effort required for the backward pass is twice that of the forward pass .
CGA algorithm involves quadratic programming projection step Goldfarb & Idnani (1983) to update the local gradients.
Quadratic programming solver (quadprog) uses Goldfarb/Idnani dual algorithm.
CGA uses quadratic programming to solve the following (Equation 6 -see Equation 5a in Esfandiari et al.
(2021)) optimization problem in an iterative manner:
[Equation]
where, G is the matrix containing cross-gradients, g is the self-gradient and the optimal gradient direction g ∗ in terms of the optimal solution of the above equation u ∗ is g ∗ =
G T u ∗ + g .
The above optimization takes multiple iterations which results in compute and time complexity to be of polynomial(degree ≥ 2 ) order.
In contrast, NGC involves simple averaging step that requires
O ( m s N i ) addition operations.