GAL: Gradient Assisted Learning for Decentralized Multi-Organization Collaborations
Abstract
Collaborations among multiple organizations, such as ﬁnancial institutions, medical centers, and retail markets in decentralized settings are crucial to providing improved service and performance.
However, the underlying organizations may have little interest in sharing their local data, models, and objective functions.
These requirements have created new challenges for multi-organization collaboration.
In this work, we propose Gradient Assisted Learning (GAL), a new method for multiple organizations to assist each other in supervised learning tasks without sharing local data, models, and objective functions.
In this framework, all participants collaboratively optimize the aggregate of local loss functions, and each participant autonomously builds its own model by iteratively ﬁtting the gradients of the overarching objective function.
We also provide asymptotic convergence analysis and practical case studies of GAL.
Experimental studies demonstrate that
GAL can achieve performance close to centralized learning when all data, models, and objective functions are fully disclosed.
Our code is available here 1 .
Introduction
One of the main challenges in harnessing the power of big data is the fusion of knowledge from numerous decentralized organizations that may have proprietary data, models and objective functions.
Due to various ethical and regulatory constraints it may not be feasible for decentralized organizations to centralize their data and fully collaborate to learn a shared model.
Thus, a large-scale autonomous decentralized learning method that can avoid data, models and objective functions transparency may be of critical interest.
Cooperative learning may have various scientiﬁc and business applications [1].
As illustrated in
Figure 1, a medical institute may be helped by multiple clinical laboratories and pharmaceutical entities to improve clinical treatment and facilitate scientiﬁc research [2,3].
Financial organizations may collaborate with universities and insurance companies to predict loan default rates [4].
The organizations can match the correspondence with common identiﬁers, such as user identiﬁcation associated with the registration of different online platforms, timestamps associated with different
[Figure]
PrivateData clinics and health providers, and geo-locations associated with map-related trafﬁc and agricultural data.
With the help of our framework, they can form a community of shared interest to provide better
Machine-Learning-as-a-Service (MLaaS)
[5,6] without transmitting their local data, models, and objective functions.
The main idea of Gradient Assisted Learning (GAL) is outlined below.
In the training stage, the organization to be assisted, denoted by Alice, will calculate a set of ‘residuals’ and broadcast these to other organizations.
These residuals approximate the fastest direction of reducing the training loss in hindsight.
Subsequently, other organizations will ﬁt the residuals using their local data, models, and objective functions and send the ﬁtted values back to Alice.
Alice will then assign weights to each organization to best approximate the fastest direction of learning.
Next, Alice will line-search for the optimal gradient assisted learning rate along the calculated direction of learning.
The above procedure is repeated until Alice accomplishes sufﬁcient learning.
In the inference stage, other organizations will send their locally predicted values to Alice, who will then assemble them to generate the ﬁnal prediction.
We show that the number of assistance rounds needed to attain the centralized performance is often small (e.g., fewer than ten).
This is appealing since GAL is primarily developed for large organizations with rich computation resources.
A small number of interactions will reduce communication and networking costs.
Our main contributions are summarized below.
• We propose a Gradient Assisted Learning (GAL) algorithm that is suitable for large-scale autonomous decentralized learning.
It can effectively exploit task-relevant information preserved by vertically decentralized organizations.
Our method enables simultaneous collaboration among organizations without sharing data, models, and objective functions.
• We provide asymptotic convergence analysis and practical case studies of GAL.
For the case of vertically distributed data, GAL generalizes the classical Gradient Boosting algorithm.
• Our proposed method can signiﬁcantly outperform learning baselines and achieve nearoracle performance on various benchmark datasets.
Compared with existing works, GAL does not need frequent synchronization of organizations.
It also signiﬁcantly
reduces the computation and communication overhead.
Related work
Multimodal Data Fusion Vertically distributed data can be viewed as multimodal data with modalities provided in a distributed manner to different learners/organizations.
Standard multimodal data fusion methods include the early, intermediate, and late data fusions [7,8].
These methods concatenate different modes of data at the input, intermediate representation, and ﬁnal prediction levels.
However, these data fusion methods in decentralized settings often require organizations to share the task labels to train their local models synchronously.
In contrast, our method presented below only requires that organizations asynchronously ﬁt some task-related statistics named pseudo-residuals to approximate the direction of reducing the global training loss in hindsight.
Gradient Boosting Our approach was inspired by Gradient Boosting [9,10], where weak learners are sequentially trained from the same dataset and aggregated into a strong learner.
In our learning context, each organization uses side information from heterogeneous data sources to improve a particular learner’s performance.
Our method can be regarded as a generalization of Gradient Boosting to address decentralized learning with vertically distributed data.
Federated Learning Federated learning [11–16] is a popular distributed learning framework.
Its main idea is to learn a joint model by averaging locally learned model parameters.
It avoids the need for the transmission of local training data.
Conceptually, the goal of Federated Learning is to exploit the resources of edge devices with communication efﬁciency.
Vertical Federated Learning methods split sub-networks for local clients to jointly optimize a global model [17–22].
These methods can be viewed as federated learning with an intermediate data fusion method, and the central server will have access to the true labels.
In order to converge, these methods typically require very frequent batchwise synchronization of backward gradients [22].
Frequent batchwise synchronization is critical for vertical federated learning because the local model at each client constitutes a part of the globally backpropable model, and one client’s local update may not decrease the overall loss.
In contrast, our proposed method trains multiple autonomous local models with pseudo-residuals, each contributing to a small portion of the overarching loss.
Each round of updates will decrease the loss.
Consequently, our method can achieve desirable performance with signiﬁcantly fewer communication rounds without a global backpropable model.
Assisted Learning Assisted Learning (AL)
[23] is a decentralized collaborative learning framework for organizations to improve their learning quality.
In that context, neither the organization being assisted nor the assisting organizations share their local models and data.
The original AL methodology applies to regression tasks.
It is derived from a linear projection perspective, and its convergence to the oracle performance was theoretically justiﬁed for linear regression models with quadratic loss.
Inspired by Gradient Boosting, the proposed Gradient Assisted Learning (GAL) is a general method for multiple organizations to assist each other in supervised learning scenarios.
Overall, AL and GAL share similar motivations and concepts but signiﬁcantly differ from methodological and theoretical perspectives.
More speciﬁcally, the novelties of GAL include: 1) generalization from regression loss to any differentiable loss for supervised learning, 2) allowing for local objective functions at each organization, 3) generalization from a sequential protocol between two organizations to parallel aggregation across multiple organizations, 4) introduction of deep model sharing for reducing computation and memory costs, 5) introduction of the assisted learning rate for fast convergence, and 6) theoretical analysis of GAL’s convergence properties.
Gradient Assisted Learning 3.1 Notation
Suppose that there are N data observations independently drawn from a joint distribution p xy =
p x p y | x , where y ∈ Y and x ∈ R d respectively represent the task label and feature variables, and d is the number of features.
For regression tasks, we have Y = R .
For K -class classiﬁcation tasks, Y = { e 1 , . . .
, e K } , where e k is the canonical vector representing the class k , k = 1 , . . .
, K .
Let E and E N denote the expectation and empirical expectation, respectively.
Thus, E N g ( y, x ) ∆ =
N − 1 (cid:80) Ni =1 g ( y i , x i ) for any measurable function g , where ( y i , x i ) are i.i.d.
observations from p xy .
Suppose that there are M organizations.
Each organization m only holds X m , a sub-vector of
X (illustrated in Figure 2).
In general, we assume that the variables in X 1 , . . .
, X M are disjoint in the presentation of our algorithm, although our method also allows the sharing of some variables.
For example, one organization may observe demographic features for a mobile user cohort, and another organization holds health-related features of that cohort.
Without loss of generality, we suppose that Alice, the organization to be assisted, has local data x 1 and task label y 1 , while other M −organizations are collaborators which assist Alice and have local data x 2 . . .
x M .
We use 1 : m and 1 : t to represent from the ﬁrst to the m th organization and the t th assistance round, respectively.
3.2 Problem
For m = 1 , . . .
, M , let { x i,m } Ni =1 denote the available data to the organization m .
Thus, N objects are simultaneously observed by M organizations, each observing a subset of features from the x ∈ R d .
Alice also has local task labels { y i, 1 } Ni =1 for training purposes.
[Figure]
Let F m and L m respectively denote supervised function class (such as generalized linear functions or neural networks) and the local objective function of organization m .
We will assume that L m is differentiable.
Without loss of generality, we assume that Alice denotes organization 1 , who will be assisted.
Without assistance from other organizations, Alice would learn a model that minimizes the following empirical risk,
[Equation]
Note that the above formulation only involves Alice’s local data x 1 and local model (as represented by
F 1 and L 1 ).
In an oracle case, Alice would be able to operate on other organizations’ data x 2 , . . .
, x M as well.
Recall that x represents the ensemble of all the available data variables.
In general, the ideal case for Alice is to minimize the following empirical risk,
[Equation]
where F is a supervised function class deﬁned with the space of x as its domain.
In reality, Alice has no access to the complete data and model resources of other organizations.
In this light, she shall be happy to crowdsource the learning task to other organizations to cooperatively build a model in hindsight, without the need to share any organization’s local data or models.
In the prediction stage, Alice can collect the information needed to form a ﬁnal prediction to hopefully achieve a performance that signiﬁcantly improves over her single-organization performance.
To this end, we will develop a solution for Alice to achieve such a goal.
We will include detailed derivations and discussions of our solution in Subsection 3.3.
For readability, we summarize notations that will be frequently used in the exposition below.
Our method will require Alice to occasionally send continuous-valued vectors r 1 =
[ r i, 1 ] Ni =1 ∈ R N × K to each organization m at each communication round.
These residuals, to be elaborated in the next subsection, approximate the fastest direction of reducing the training loss in hindsight, namely a sample version of ∂L 1 ( y 1 , F ( x )) /∂F
( x ) given Alice’s estimation of F at a particular time (round).
Upon the input of these residual vectors, the organization m will locally learn a supervised function f m that maps from its feature space to the residual space.
With a slight abuse of notation, we also refer to f m as the learned model.
To this end, the organization m will perform the empirical risk minimization
[Equation]
to obtain a locally trained model f m .
Here, F m and (cid:96) m respectively denote the supervised function class and loss function of the organization m .
We note that (cid:96) m are local regression loss functions for ﬁtting the pseudo-residual r 1 and may not necessarily be the same as L 1 for ﬁtting true labels.
For example, L 1 may be the cross-entropy loss for classiﬁcation of label y , while (cid:96) 1: M could be the squared loss for regression of the response r 1 .
The above local training (optimization) is often performed using the stochastic gradient descent (SGD) algorithm.
In our assisted learning context,
L 1 , F m , and (cid:96) m are proprietary local resources that cannot be shared across organizations.
3.3 The GAL Algorithm
We ﬁrst introduce the derivation of the GAL algorithm from a functional gradient descent perspective.
Then, we cast the algorithm into pseudocode and discuss each step.
Consider the unrealistic case that
Alice has all the data x needed for a centralized supervised function F : x (cid:55)→ F ( x ) .
Recall that the goal of Alice is to minimize the population loss E p x,y L 1 ( y, F ( x )) over a data distribution p x,y .
If
Input: M decentralized organizations, each holding data { x i,m } Ni =1 (local) corresponding to N objects, the task label { y i, 1 } Ni =1 initially held by the service receiver (Alice local), model class F m (local), gradient assistance weights w (Alice local), assistance rate η (Alice local), overarching loss function L 1 (Alice local), regression loss function (cid:96) m to ﬁt pseudo-residual (local), assistance rounds T .
Learning Stage:
Intialization:
Let t = 0 , and initialize F 0 ( x ) =
E N ( y 1 ) for assistance round t from 1 to T do
Compute pseudo-residual r t
[Equation]
= − (cid:20) ∂L 1 ( y 1 ,F t − 1 ( x ) )
∂F t − 1 ( x ) (cid:21)
Broadcast pseudo-residual r t 1 to other organizations for organization m from 1 to M in parallel do f
[Equation]
[Equation] [Equation]
( x m )) end
Gather predictions f t m ( x m ) , m = 1 , . . .
M , from all the organizations
Optimize the gradient assistance weights
[Equation]
Line-search for the gradient assisted learning rate
[Equation]
end
Prediction Stage:
For each data observation x ∗ , of which x ∗ m is held by organization m :
Gather predictions f tm ( x ∗ m ) , t = 1 , . . .
, T from each organization m , m = 1 , . . .
, M
Predict with
[Equation]
p x,y is known, starting with an initial guess F 0 ( x ) , Alice would have performed a gradient descent step in the form of
[Equation]
where the equality holds under the standard regularity conditions of exchanging integration and differentiation.
Note that the second term in (4) is a function on R d .
However, because Alice only has access to her own data x 1 , the expectation E p x,y cannot be realistically evaluated.
Therefore, we need to approximate it with functions in a pre-speciﬁed function set.
In other words, we will ﬁnd f from F M that ‘best’ approximates E p x,
y ∂∂F L 1 ( y, F ( x )) .
We will show that this is actionable without requiring the organizations to share proprietary data, models, and objective functions.
Recall that F m is the function set locally used by the organization m , and x m is a correspondingly observed portion of x .
The function class that we propose to approximate the second term in (4) is
[Equation]
where P M = { w ∈ R M : (cid:80) Mm =1 w m = 1 , w m ≥ 0 } denotes the probability simplex.
The gradient assistance weights w m ’s are interpreted as the contributions of each organization at a particular greedy update step.
The gradient assistance weights are constrained to sum to one to ensure the function space is compact and the solutions exist.
We propose the following solution so that each organization can operate on its own local data, model, and objective function .
Alice initializes with a startup model, denoted by F 0 ( x ) =
F 0 ( x 1 , y 1 ) , based only on her local data and labels.
Alice broadcasts r 1 (named ‘pseudo residuals’) to each organization
User ID
ProprietaryModel
PrivateData
End User ServiceReceiver
Collaborator
B
I I
I
Residual
B
Residual
M
H
B
H
M
Residual
B
H
M
H H
H
M M
M M
M
M 3 (a) Learning Stage (b) Prediction Stage
B
B
B 3 B
User ID m, m = 2 , · · · , M , who will then ﬁt a local model f m using r 1 .
Each organization will then send the ﬁtted values from f m to Alice, who will train suitable gradient assistance weights w m .
Subsequently,
Alice ﬁnds the η in (4) that minimizes her current empirical risk.
The above procedure is iterated for a ﬁnite number of rounds until Alice obtains a satisfactory performance (e.g., on validation data).
The validation will be based on the same technique as the prediction stage to be described below.
This training stage is described under the ‘learning stage’ of Algorithm 1.
Note that the pseudocode is from the perspective of Alice, the service receiver.
Each organization m will only need to perform the empirical risk minimization using the label r t 1 sent by Alice at each round t .
In the Prediction/Inference stage (given above in Algorithm 1), other organizations send prediction results generated from their local models to Alice, who will calculate a prediction result F T ( x ) that is implicitly operated on x , where T is the number of iteration steps.
The idea of approximating functional derivatives with regularized functions was historically used to develop the seminal work of gradient boosting [9,10].
The above method reduces to the standard gradient boosting algorithm when there is only one organization.
Organizations in our learning framework form a shared community of interest.
Each serviceproviding organization can provide end-to-end assistance for an organization without sharing anyone’s proprietary data, models, and objective functions.
In practice, the participating organizations may receive ﬁnancial rewards from the one to assist.
Moreover, every organization in this framework can provide its own task and seek help from others.
As a result, all organizations become mutually beneﬁcial to each other.
We provide a realistic example in Figure 3 to demonstrate each step of
Algorithm 1.
We elaborate on the learning and prediction procedures in the Appendix.
We also provide an asymptotic convergence analysis for the GAL algorithm, where the goal is to minimize a loss f
(cid:55)→ L ( f ) over a function class through step-wise function aggregations.
Because of the greedy nature of GAL, we consider the function class to be the linear span of organization-speciﬁc
F m .
The following result states that the GAL can produce a solution that attains the inﬁmum of L ( f ) .
More technical details are included in the Appendix.
Theorem 1 Assume that the loss (functional) f
(cid:55)→ L ( f ) is convex and differentiable on F , the function u (cid:55)→ L ( f + ug ) has an upper-bounded second-order derivative ∂ 2 L ( f + ug ) /∂u
2 for all f ∈ span ( F 1 , . .
. , F M ) and g ∈ ∪ Mm =1 F m , and the ranges of learning rates { a t } t =1 , 2 ,...
satisfy (cid:80) ∞ t =1 a t = ∞ , (cid:80) ∞
t =1 a 2 t < ∞ .
Then, the GAL algorithm satisﬁes L ( F t ) → inf f ∈ span ( F 1 ,..., F M ) L ( f ) as t → ∞ , with a convergence rate at the order of O ( (cid:80) tτ =
1 ( a 1: τ /a 1: t ) a 2 τ ) .
Experimental Studies
Baselines Our experiments are performed with four baselines, including ‘Interm’, ‘Late’, ‘Joint’, ‘Alone’, and ‘AL’.
‘Interm’ and ‘Late’ refer to intermediate and late data fusions [7,8], respectively.
The intermediate data fusion (‘Interm’) sums up the intermediate features (output before the last layer) from each separate feature extractor, and then the aggregated feature is passed into a shared last layer to output the ﬁnal prediction.
The late data fusion (‘Late’) sums up the ﬁnal prediction of each separate local model.
Thus, ‘Interm’ works only for deep learning models such as CNN and LSTM by averaging the hidden representation of each local model, while ‘Late’ also works for Linear models as it aggregates the output of each local model.
‘Joint’ is the case where all the data are held by Alice and trained with the Gradient Boosting reduced from GAL.
‘Alone’ is the single-agent scenario, where only Alice’s data are used for learning and prediction.
‘AL’ represents the performance of Assisted Learning [23].
GAL is expected to perform close to the centralized baselines, including ‘Interm’, ‘Late’, and ‘Joint’ cases, while signiﬁcantly outperforming the ‘Alone’ and ‘AL’ cases.
The summary statistics of each dataset are elaborated in Table 7 of the Appendix.
Details of learning hyper-parameters are included in Table 9 of the Appendix.
We conducted four random experiments for all datasets with different seeds, and the standard errors are shown in the brackets of all tables.
4.1 Model Autonomy
Recall that GAL allows each organization to choose its own local model.
We demonstrate the performance of autonomous local models with UCI datasets downloadable from the scikit-learn package [24], including Diabetes [25], Boston Housing [26], Blob [24], Iris [27], Wine [28], Breast
Cancer [29], and QSAR [30] datasets, where we randomly partition the features into 2 , 4 , or 8 subsets.
For all the UCI datasets, we train on 80% of the available data and test on the remaining.
[Table]
We experiment with Linear, Gradient Boosting (GB)
[Table]
and Support Vector Machine (SVM) for local models f m ( · ) with the UCI datasets.
We also demonstrate the performance of the scenario (GB-SVM) where half of the organizations use GB and the other half uses SVM.
The experimental results are shown in Tables 1.
↓ indicates the smaller the better, while ↑ indicates the larger the better.
Our method signiﬁcantly outperforms the baselines ‘Alone’ and ‘AL.’
The results also demonstrate that with GAL, an organization with little informative data and free choice of its local model (model autonomy) can leverage others’ local data and models and even achieve near-oracle performance.
We point out that although ‘Interm,’ ‘Late,’ and ‘Joint’ marginally outperform our method, they require training from centralized data.
Our GAL algorithm replaces the true labels used in ‘Interm,’ ‘Late,’ and ‘Joint’ centralized cases with pseudo-residuals.
The results from both regression and classiﬁcation datasets with various model settings lead to similar conclusions.
4.2 Deep Model Sharing
[Table]
We demonstrate that our method is effective for deep models by using MNIST [31] and CIFAR10 [32] image datasets, where we split each image into patches as depicted in Figure 6.
We use
Convolutional Neural Networks (CNN) for both datasets, and the model architecture can be found in Table 8 of the Appendix.
We visualize the performance of CIFAR10 at each assistance round in Figure 4 (a-c).
The number of assistance rounds needed to approach the centralized performance is small (e.g., often within ten).
The experimental results are shown in Tables 2.
GAL signiﬁcantly outperforms the bottom line ‘Alone’ in all the settings.
This is expected since the ﬁrst organization holds partial data and does not receive any assistance under ‘Alone.’
Interestingly, the performance of MNIST for M = 8 drops signiﬁcantly under ‘Alone’ because the organization only holds the left upper image patch, which is usually completely dark, as shown in Figure 6.
Because local deep learning models such as CNN can consume extensive computation space, we propose Deep Model Sharing (DMS) to allow sharing feature extractors of deep models across all iterations to save memory.
In particular, we propose to jointly train the feature extractors with residuals from previous iterations as well as from the current iteration by adding an additional last prediction layer.
The local deep model f tm ( · ) is composed of f tm,o ( f m,e ( · ))) , where f tm,o ( · ) is the last output layer at assistance round t and f m,e ( · ) is the deep feature extractor shared across multiple assistance rounds.
For each assistance round, local organizations ﬁts pseudo-residuals across previous assistance rounds r 1: t 1 with f 1: tm = argmin
f m ∈F m E N (cid:96) m (cid:0) r 1: t 1 , f 1: tm,o ( f m,e ( x m )) (cid:1) .
It is worth mentioning that we do not expect such trade-off GAL DMS to consistently outperform AL and GAL because a single feature extractor may not well ﬁt residuals across many iterations.
The results in Table 2 show that sharing the feature extractor across multiple assistance rounds can still outperform the ‘Alone’ case.
Thus, DMS can provide a trade-off between predictive performance and computation space.
The detailed comparisons can be found in Table 14.
4.3 Comparison with AL
As demonstrated in our extensive experiments, the proposed GAL outperforms AL in terms of predictive performance.
In particular, AL converges not only worse but also slower than GAL.
This is due to the fact that AL uses a constant assisted learning rate and trains participating organizations in a sequential manner.
Moreover, sequentially training participating organizations also requires much more computation and communication overhead.
We compare the computation and communication complexity between AL and GAL under the constraint of the same communication cost as demonstrated in Table 14.
Because AL sequentially trains each organization while GAL allows organizations to train locally in parallel, the computation time and communication round of AL is
M × those of GAL.
The GAL with Deep Model Sharing (GAL DMS ) saves T × computation space by sharing the feature extractor of deep models.
In summary, GAL generalizes the problem scope, reduces the computation and communication complexity, and achieves signiﬁcantly better results.
[Figure]
A constant gradient assisted learning rate ( η = 1 ) converges much slower.
The gradient assistance weights exhibits interpretability of the importance of organizations as the weights of the central image patches ( m = { 2 , 3 , 6 , 7 } ) of CIFAR10 dataset are larger than the boundary patches ( m = { 1 , 4 , 5 , 8 } ) in the ﬁrst few rounds.
More results can be found in the Appendix.
4.4 Case Studies
The results in Table 3 demonstrate the utility of GAL in various practical applications.
We illustrate the results across multiple assistance rounds of MIMICL in Figure 4 (d-f).
GAL signiﬁcantly outperforms ‘Alone’ and ‘AL.’
Our method also performs close to the centralized baselines.
[Table]
[Figure]
[Table]
Three-dimensional object recognition We use the shape representation of 3D objects for recognition from a collection of rendered views on 2D images.
We generate M = 12 2D camera views of
ModelNet40 [33] and ShapeNet55 [34] datasets following [35].
Cameras are treated as decentralized learners in our experiments.
We adopt the same CNN architecture used for MNIST and CIFAR datasets.
It is worth mentioning that ‘Joint’ of ModelNet40 and ShapeNet55 performs considerably worse than other baselines because ‘Joint’ uses a single CNN feature extractor to process all images from twelve angles [35].
Medical time series forecasting We use the in-hospital dataset MIMIC3 [36], where the task aims to predict the Length-of-stay (MIMICL) and Mortality rate (MIMICM) of the ICU stays of patients.
We follow the benchmark work [37] to process raw data and split the features into four organizations, including 1) microbiology measurement, 2) demographic information, 3) body measurement, and 4) International Classiﬁcation of Diseases (ICD).
We use MAD to evaluate the result of MIMICL (regression) and the Area Under the Curve-Receiver Operating Characteristics (AUROC) to evaluate the result of MIMICM (imbalanced binary classiﬁcation).
Our backbone model (LSTM) is the same as the one used in the benchmark work [38].
It is worth noting that the data features among the four organizations are aligned with time stamps which is a natural identiﬁer for time series forecasting.
4.5 Ablation Studies
Local objective functions Our method does not require local regression loss functions (cid:96) m to be shared with other organizations, but they are supposed to be beneﬁcial for assisting Alice.
We conduct ablation studies on the choice of various local regression loss functions.
In this study shown in
Table 4, we use different regression functions (cid:96) q ( y, ˆ y ) =
| y − ˆ y | q where q ∈ { 1 , 1 . 5 , 2 , 4 } .
( (cid:96) q 1 , (cid:96) q 2 ) allows half of the organizations to use (cid:96) q 1 while the other half to use l q 2 .
The results show that the classiﬁcation task generally works better with q > 1 .
[Table]
Privacy
enhancement Our learning framework does not require organizations to share local data, models
[Table]
and objective functions.
One potential limitation of our approach is that assisting organizations may infer Alice’s information based on the shared pseudo-residuals.
Therefore, we suggest to further enhancing privacy by adopting the framework of Differential Privacy (DP) [39] or Interval
Privacy (IP) [40].
We use the Laplace mechanism with α = 1 for DP and set the number of intervals of IP to be 1 .
We add a moderate amount of noise to the pseudo-residuals in hindsight.
In Tables 5, we demonstrate that privacy-enhanced GAL can still outperform the ‘Alone’ case.
[Table]
Gradient assisted learning rate We illustrate the gradient assisted learning rate of CIFAR10 and
MIMICL datasets at each assistance round in Figure 4(b,e).
We perform a line search for the gradient assisted learning rate with the Limited-Memory BFGS optimizer, which improves the convergence rates compared with SGD and Adam.
We conduct an ablation study using a constant gradient assisted learning rate ( η = 1 ).
As shown in Figure 4(a,d), the constant gradient assisted learning rate leads to a convergence much slower than the line-search method.
Fast convergence is desirable since the computation and communication cost increases with the number of assistance rounds.
To determine the maximal number of assistance rounds T for the service receiver, we can run the GAL procedure until the gradient assisted learning rate becomes small.
When the gradient assistance rate is low, as shown in Figure 4(e), the overarching loss converges to zero.
In this light, an organization may stop receiving assisted learning when the gradient assisted learning rate is below a threshold.
Gradient assistance weights We show the gradient assistance weights of CIFAR10 and MIMICL datasets at each assistance round in Figure 4(c,f).
The results of MNIST and CIFAR10 show that the gradient assistance weights exhibit interpretability of the importance of organizations because the image patches with dominant contributions are m = (2 , 3 , 6 , 7) (colored in red).
These image patches correspond to the center of the original image, which matches our intuition appealingly.
The weights converge to uniform as the residuals of later iterations are small.
We also conduct an ablation study of the gradient assistance weights in Figure 5 by adding noises (Gaussian with zero mean and σ 2 variance, σ ∈ { 1 , 5 } ) to the output of a randomly chosen half of the clients during learning and prediction.
Adding noise simulates realistic scenarios where some assisting organizations may be noninformative or inject noise.
We summarize the ablation study results in Table 6.
The results show that the GAL with gradient assistance weights is more robust than the GAL with a direct average
[Table] [Table]
Conclusion
We proposed Gradient Assisted Learning, a decentralized learning method for multiple organizations to collaborate without sharing data, models, and objective functions.
The proposed method can signiﬁcantly outperform the local learning baselines and achieve near-oracle performance as if data were centralized on various datasets.
All participants form a shared community of interest by autonomously building their own model and iteratively ﬁtting the gradients of the overarching objective function.
We also demonstrate asymptotic convergence analysis and practical case studies of
GAL.
Moreover, this is achieved without any constraints on the models selected by the collaborating organizations.
Acknowledgments
The work of Enmao Diao and Vahid Tarokh was supported by the Ofﬁce of Naval Research (ONR) under grant number N00014-18-1-2244.
The work of Jie Ding was supported by the National Science
Foundation (NSF) under grant number ECCS-2038603.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 4.2 for computation space and Section B for theoretical analysis.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
We do not foresee any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section B. (b) Did you include complete proofs of all theoretical results?
[Yes] See Section B. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide source codes in the supplementary material.
We use publicly available datasets.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section C in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The numerical standard error are shown in brackets of tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] One Nvidia 1080TI is enough for one experiment run.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the publicly available datasets we use.
(b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We use publicly available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] We use publicly available datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[-Suppose that F t −admits the representation F t − 
-]
[N/A]