GAL: Gradient Assisted Learning for Decentralized Multi-Organization Collaborations
Abstract
Collaborations among multiple organizations, such as ﬁnancial institutions, medical centers, and retail markets in decentralized settings are crucial to providing improved service and performance.
However, the underlying organizations may have little interest in sharing their local data, models, and objective functions.
These requirements have created new challenges for multi-organization collaboration.
In this work, we propose Gradient Assisted Learning (GAL), a new method for multiple organizations to assist each other in supervised learning tasks without sharing local data, models, and objective functions.
In this framework, all participants collaboratively optimize the aggregate of local loss functions, and each participant autonomously builds its own model by iteratively ﬁtting the gradients of the overarching objective function.
We also provide asymptotic convergence analysis and practical case studies of GAL.
Experimental studies demonstrate that
GAL can achieve performance close to centralized learning when all data, models, and objective functions are fully disclosed.
Our code is available here 1 .
Introduction
One of the main challenges in harnessing the power of big data is the fusion of knowledge from numerous decentralized organizations that may have proprietary data, models and objective functions.
Due to various ethical and regulatory constraints it may not be feasible for decentralized organizations to centralize their data and fully collaborate to learn a shared model.
Thus, a large-scale autonomous decentralized learning method that can avoid data, models and objective functions transparency may be of critical interest.
Cooperative learning may have various scientiﬁc and business applications [1].
As illustrated in
Figure 1, a medical institute may be helped by multiple clinical laboratories and pharmaceutical entities to improve clinical treatment and facilitate scientiﬁc research [2,3].
Financial organizations may collaborate with universities and insurance companies to predict loan default rates [4].
The organizations can match the correspondence with common identiﬁers, such as user identiﬁcation associated with the registration of different online platforms, timestamps associated with different [Figure] PrivateData clinics and health providers, and geo-locations associated with map-related trafﬁc and agricultural data.
With the help of our framework, they can form a community of shared interest to provide better
Machine-Learning-as-a-Service (MLaaS)
[5,6] without transmitting their local data, models, and objective functions.
The main idea of Gradient Assisted Learning (GAL) is outlined below.
In the training stage, the organization to be assisted, denoted by Alice, will calculate a set of ‘residuals’ and broadcast these to other organizations.
These residuals approximate the fastest direction of reducing the training loss in hindsight.
Subsequently, other organizations will ﬁt the residuals using their local data, models, and objective functions and send the ﬁtted values back to Alice.
Alice will then assign weights to each organization to best approximate the fastest direction of learning.
Next, Alice will line-search for the optimal gradient assisted learning rate along the calculated direction of learning.
The above procedure is repeated until Alice accomplishes sufﬁcient learning.
In the inference stage, other organizations will send their locally predicted values to Alice, who will then assemble them to generate the ﬁnal prediction.
We show that the number of assistance rounds needed to attain the centralized performance is often small (e.g., fewer than ten).
This is appealing since GAL is primarily developed for large organizations with rich computation resources.
A small number of interactions will reduce communication and networking costs.
Our main contributions are summarized below.
• We propose a Gradient Assisted Learning (GAL) algorithm that is suitable for large-scale autonomous decentralized learning.
It can effectively exploit task-relevant information preserved by vertically decentralized organizations.
Our method enables simultaneous collaboration among organizations without sharing data, models, and objective functions.
• We provide asymptotic convergence analysis and practical case studies of GAL.
For the case of vertically distributed data, GAL generalizes the classical Gradient Boosting algorithm.
• Our proposed method can signiﬁcantly outperform learning baselines and achieve nearoracle performance on various benchmark datasets.
Compared with existing works, GAL does not need frequent synchronization of organizations.
It also signiﬁcantly
reduces the computation and communication overhead.
Related work
Multimodal Data Fusion Vertically distributed data can be viewed as multimodal data with modalities provided in a distributed manner to different learners/organizations.
Standard multimodal data fusion methods include the early, intermediate, and late data fusions [7,8].
These methods concatenate different modes of data at the input, intermediate representation, and ﬁnal prediction levels.
However, these data fusion methods in decentralized settings often require organizations to share the task labels to train their local models synchronously.
In contrast, our method presented below only requires that organizations asynchronously ﬁt some task-related statistics named pseudo-residuals to approximate the direction of reducing the global training loss in hindsight.
Gradient Boosting Our approach was inspired by Gradient Boosting [9,10], where weak learners are sequentially trained from the same dataset and aggregated into a strong learner.
In our learning context, each organization uses side information from heterogeneous data sources to improve a particular learner’s performance.
Our method can be regarded as a generalization of Gradient Boosting to address decentralized learning with vertically distributed data.
Federated Learning Federated learning [11–16] is a popular distributed learning framework.
Its main idea is to learn a joint model by averaging locally learned model parameters.
It avoids the need for the transmission of local training data.
Conceptually, the goal of Federated Learning is to exploit the resources of edge devices with communication efﬁciency.
Vertical Federated Learning methods split sub-networks for local clients to jointly optimize a global model [17–22].
These methods can be viewed as federated learning with an intermediate data fusion method, and the central server will have access to the true labels.
In order to converge, these methods typically require very frequent batchwise synchronization of backward gradients [22].
Frequent batchwise synchronization is critical for vertical federated learning because the local model at each client constitutes a part of the globally backpropable model, and one client’s local update may not decrease the overall loss.
In contrast, our proposed method trains multiple autonomous local models with pseudo-residuals, each contributing to a small portion of the overarching loss.
Each round of updates will decrease the loss.
Consequently, our method can achieve desirable performance with signiﬁcantly fewer communication rounds without a global backpropable model.
Assisted Learning Assisted Learning (AL)
[23] is a decentralized collaborative learning framework for organizations to improve their learning quality.
In that context, neither the organization being assisted nor the assisting organizations share their local models and data.
The original AL methodology applies to regression tasks.
It is derived from a linear projection perspective, and its convergence to the oracle performance was theoretically justiﬁed for linear regression models with quadratic loss.
Inspired by Gradient Boosting, the proposed Gradient Assisted Learning (GAL) is a general method for multiple organizations to assist each other in supervised learning scenarios.
Overall, AL and GAL share similar motivations and concepts but signiﬁcantly differ from methodological and theoretical perspectives.
More speciﬁcally, the novelties of GAL include: 1) generalization from regression loss to any differentiable loss for supervised learning, 2) allowing for local objective functions at each organization, 3) generalization from a sequential protocol between two organizations to parallel aggregation across multiple organizations, 4) introduction of deep model sharing for reducing computation and memory costs, 5) introduction of the assisted learning rate for fast convergence, and 6) theoretical analysis of GAL’s convergence properties.
Gradient Assisted Learning 3.1 Notation
Suppose that there are N data observations independently drawn from a joint distribution p xy =
p x p y | x , where y ∈ Y and x ∈ R d respectively represent the task label and feature variables, and d is the number of features.
For regression tasks, we have Y = R .
For K -class classiﬁcation tasks, Y = { e 1 , . . .
, e K } , where e k is the canonical vector representing the class k , k = 1 , . . .
, K .
Let E and E N denote the expectation and empirical expectation, respectively.
Thus, E N g ( y, x ) ∆ =
N − 1 (cid:80) Ni =1 g ( y i , x i ) for any measurable function g , where ( y i , x i ) are i.i.d.
observations from p xy .
Suppose that there are M organizations.
Each organization m only holds X m , a sub-vector of
X (illustrated in Figure 2).
In general, we assume that the variables in X 1 , . . .
, X M are disjoint in the presentation of our algorithm, although our method also allows the sharing of some variables.
For example, one organization may observe demographic features for a mobile user cohort, and another organization holds health-related features of that cohort.
Without loss of generality, we suppose that Alice, the organization to be assisted, has local data x 1 and task label y 1 , while other M −organizations are collaborators which assist Alice and have local data x 2 . . .
x M .
We use 1 : m and 1 : t to represent from the ﬁrst to the m th organization and the t th assistance round, respectively.
3.2 Problem
For m = 1 , . . .
, M , let { x i,m } Ni =1 denote the available data to the organization m .
Thus, N objects are simultaneously observed by M organizations, each observing a subset of features from the x ∈ R d .
Alice also has local task labels { y i, 1 } Ni =1 for training purposes.
[Figure]
Let F m and L m respectively denote supervised function class (such as generalized linear functions or neural networks) and the local objective function of organization m .
We will assume that L m is differentiable.
Without loss of generality, we assume that Alice denotes organization 1 , who will be assisted.
Without assistance from other organizations, Alice would learn a model that minimizes the following empirical risk,
[Equation]
Note that the above formulation only involves Alice’s local data x 1 and local model (as represented by
F 1 and L 1 ).
In an oracle case, Alice would be able to operate on other organizations’ data x 2 , . . .
, x M as well.
Recall that x represents the ensemble of all the available data variables.
In general, the ideal case for Alice is to minimize the following empirical risk,
[Equation]
where F is a supervised function class deﬁned with the space of x as its domain.
In reality, Alice has no access to the complete data and model resources of other organizations.
In this light, she shall be happy to crowdsource the learning task to other organizations to cooperatively build a model in hindsight, without the need to share any organization’s local data or models.
In the prediction stage, Alice can collect the information needed to form a ﬁnal prediction to hopefully achieve a performance that signiﬁcantly improves over her single-organization performance.
To this end, we will develop a solution for Alice to achieve such a goal.
We will include detailed derivations and discussions of our solution in Subsection 3.3.
For readability, we summarize notations that will be frequently used in the exposition below.
Our method will require Alice to occasionally send continuous-valued vectors r 1 =
[ r i, 1 ] Ni =1 ∈ R N × K to each organization m at each communication round.
These residuals, to be elaborated in the next subsection, approximate the fastest direction of reducing the training loss in hindsight, namely a sample version of ∂L 1 ( y 1 , F ( x )) /∂F
( x ) given Alice’s estimation of F at a particular time (round).
Upon the input of these residual vectors, the organization m will locally learn a supervised function f m that maps from its feature space to the residual space.
With a slight abuse of notation, we also refer to f m as the learned model.
To this end, the organization m will perform the empirical risk minimization
[Equation]
to obtain a locally trained model f m .
Here, F m and (cid:96) m respectively denote the supervised function class and loss function of the organization m .
We note that (cid:96) m are local regression loss functions for ﬁtting the pseudo-residual r 1 and may not necessarily be the same as L 1 for ﬁtting true labels.
For example, L 1 may be the cross-entropy loss for classiﬁcation of label y , while (cid:96) 1: M could be the squared loss for regression of the response r 1 .
The above local training (optimization) is often performed using the stochastic gradient descent (SGD) algorithm.
In our assisted learning context,
L 1 , F m , and (cid:96) m are proprietary local resources that cannot be shared across organizations.
3.3 The GAL Algorithm
We ﬁrst introduce the derivation of the GAL algorithm from a functional gradient descent perspective.
Then, we cast the algorithm into pseudocode and discuss each step.
Consider the unrealistic case that
Alice has all the data x needed for a centralized supervised function F : x (cid:55)→ F ( x ) .
Recall that the goal of Alice is to minimize the population loss E p x,y L 1 ( y, F ( x )) over a data distribution p x,y .
If
Input: M decentralized organizations, each holding data { x i,m } Ni =1 (local) corresponding to N objects, the task label { y i, 1 } Ni =1 initially held by the service receiver (Alice local), model class F m (local), gradient assistance weights w (Alice local), assistance rate η (Alice local), overarching loss function L 1 (Alice local), regression loss function (cid:96) m to ﬁt pseudo-residual (local), assistance rounds T .
Learning Stage:
Intialization:
Let t = 0 , and initialize F 0 ( x ) =
E N ( y 1 ) for assistance round t from 1 to T do
Compute pseudo-residual r t
[Equation]
= − (cid:20) ∂L 1 ( y 1 ,F t − 1 ( x ) )
∂F t − 1 ( x ) (cid:21)
Broadcast pseudo-residual r t 1 to other organizations for organization m from 1 to M in parallel do f
[Equation]
[Equation] [Equation]
( x m )) end
Gather predictions f t m ( x m ) , m = 1 , . . .
M , from all the organizations
Optimize the gradient assistance weights
[Equation]
Line-search for the gradient assisted learning rate
[Equation]
end
Prediction Stage:
For each data observation x ∗ , of which x ∗ m is held by organization m :
Gather predictions f tm ( x ∗ m ) , t = 1 , . . .
, T from each organization m , m = 1 , . . .
, M
Predict with
[Equation]
p x,y is known, starting with an initial guess F 0 ( x ) , Alice would have performed a gradient descent step in the form of
[Equation]
where the equality holds under the standard regularity conditions of exchanging integration and differentiation.
Note that the second term in (4) is a function on R d .
However, because Alice only has access to her own data x 1 , the expectation E p x,y cannot be realistically evaluated.
Therefore, we need to approximate it with functions in a pre-speciﬁed function set.
In other words, we will ﬁnd f from F M that ‘best’ approximates E p x,
y ∂∂F L 1 ( y, F ( x )) .
We will show that this is actionable without requiring the organizations to share proprietary data, models, and objective functions.
Recall that F m is the function set locally used by the organization m , and x m is a correspondingly observed portion of x .
The function class that we propose to approximate the second term in (4) is
[Equation]
where P M = { w ∈ R M : (cid:80) Mm =1 w m = 1 , w m ≥ 0 } denotes the probability simplex.
The gradient assistance weights w m ’s are interpreted as the contributions of each organization at a particular greedy update step.
The gradient assistance weights are constrained to sum to one to ensure the function space is compact and the solutions exist.
We propose the following solution so that each organization can operate on its own local data, model, and objective function .
Alice initializes with a startup model, denoted by F 0 ( x ) =
F 0 ( x 1 , y 1 ) , based only on her local data and labels.
Alice broadcasts r 1 (named ‘pseudo residuals’) to each organization
User ID
ProprietaryModel
PrivateData
End User ServiceReceiver
Collaborator
B
I I
I
Residual
B
Residual
M
H
B
H
M
Residual
B
H
M
H H
H
M M
M M
M
M 3 (a) Learning Stage (b) Prediction Stage
B
B
B 3 B
User ID m, m = 2 , · · · , M , who will then ﬁt a local model f m using r 1 .
Each organization will then send the ﬁtted values from f m to Alice, who will train suitable gradient assistance weights w m .
Subsequently,
Alice ﬁnds the η in (4) that minimizes her current empirical risk.
The above procedure is iterated for a ﬁnite number of rounds until Alice obtains a satisfactory performance (e.g., on validation data).
The validation will be based on the same technique as the prediction stage to be described below.
This training stage is described under the ‘learning stage’ of Algorithm 1.
Note that the pseudocode is from the perspective of Alice, the service receiver.
Each organization m will only need to perform the empirical risk minimization using the label r t 1 sent by Alice at each round t .
In the Prediction/Inference stage (given above in Algorithm 1), other organizations send prediction results generated from their local models to Alice, who will calculate a prediction result F T ( x ) that is implicitly operated on x , where T is the number of iteration steps.
The idea of approximating functional derivatives with regularized functions was historically used to develop the seminal work of gradient boosting [9,10].
The above method reduces to the standard gradient boosting algorithm when there is only one organization.
Organizations in our learning framework form a shared community of interest.
Each serviceproviding organization can provide end-to-end assistance for an organization without sharing anyone’s proprietary data, models, and objective functions.
In practice, the participating organizations may receive ﬁnancial rewards from the one to assist.
Moreover, every organization in this framework can provide its own task and seek help from others.
As a result, all organizations become mutually beneﬁcial to each other.
We provide a realistic example in Figure 3 to demonstrate each step of
Algorithm 1.
We elaborate on the learning and prediction procedures in the Appendix.
We also provide an asymptotic convergence analysis for the GAL algorithm, where the goal is to minimize a loss f
(cid:55)→ L ( f ) over a function class through step-wise function aggregations.
Because of the greedy nature of GAL, we consider the function class to be the linear span of organization-speciﬁc
F m .
The following result states that the GAL can produce a solution that attains the inﬁmum of L ( f ) .
More technical details are included in the Appendix.
Theorem 1 Assume that the loss (functional) f
(cid:55)→ L ( f ) is convex and differentiable on F , the function u (cid:55)→ L ( f + ug ) has an upper-bounded second-order derivative ∂ 2 L ( f + ug ) /∂u 2 for all f ∈ span ( F 1 , . .
. , F M ) and g ∈ ∪ Mm =1 F m , and the ranges of learning rates { a t } t =1 , 2 ,...
satisfy (cid:80) ∞ t =1 a t = ∞ , (cid:80) ∞
t =1 a 2 t < ∞ .
Then, the GAL algorithm satisﬁes L ( F t ) → inf f ∈ span ( F 1 ,..., F M ) L ( f ) as t → ∞ , with a convergence rate at the order of O ( (cid:80) tτ =
1 ( a 1: τ /a 1: t ) a 2 τ ) .
Experimental Studies
Baselines Our experiments are performed with four baselines, including ‘Interm’, ‘Late’, ‘Joint’, ‘Alone’, and ‘AL’.
‘Interm’ and ‘Late’ refer to intermediate and late data fusions [7,8], respectively.
The intermediate data fusion (‘Interm’) sums up the intermediate features (output before the last layer) from each separate feature extractor, and then the aggregated feature is passed into a shared last layer to output the ﬁnal prediction.
The late data fusion (‘Late’) sums up the ﬁnal prediction of each separate local model.
Thus, ‘Interm’ works only for deep learning models such as CNN and LSTM by averaging the hidden representation of each local model, while ‘Late’ also works for Linear models as it aggregates the output of each local model.
‘Joint’ is the case where all the data are held by Alice and trained with the Gradient Boosting reduced from GAL.
‘Alone’ is the single-agent scenario, where only Alice’s data are used for learning and prediction.
‘AL’ represents the performance of Assisted Learning [23].
GAL is expected to perform close to the centralized baselines, including ‘Interm’, ‘Late’, and ‘Joint’ cases, while signiﬁcantly outperforming the ‘Alone’ and ‘AL’ cases.
The summary statistics of each dataset are elaborated in Table 7 of the Appendix.
Details of learning hyper-parameters are included in Table 9 of the Appendix.
We conducted four random experiments for all datasets with different seeds, and the standard errors are shown in the brackets of all tables.
4.1 Model Autonomy
Recall that GAL allows each organization to choose its own local model.
We demonstrate the performance of autonomous local models with UCI datasets downloadable from the scikit-learn package [24], including Diabetes [25], Boston Housing [26], Blob [24], Iris [27], Wine [28], Breast
Cancer [29], and QSAR [30] datasets, where we randomly partition the features into 2 , 4 , or 8 subsets.
For all the UCI datasets, we train on 80% of the available data and test on the remaining. [Table]
We experiment with Linear, Gradient Boosting (GB) [Table] and Support Vector Machine (SVM) for local models f m ( · ) with the UCI datasets.
We also demonstrate the performance of the scenario (GB-SVM) where half of the organizations use GB and the other half uses SVM.
The experimental results are shown in Tables 1.
↓ indicates the smaller the better, while ↑ indicates the larger the better.
Our method signiﬁcantly outperforms the baselines ‘Alone’ and ‘AL.’
The results also demonstrate that with GAL, an organization with little informative data and free choice of its local model (model autonomy) can leverage others’ local data and models and even achieve near-oracle performance.
We point out that although ‘Interm,’ ‘Late,’ and ‘Joint’ marginally outperform our method, they require training from centralized data.
Our GAL algorithm replaces the true labels used in ‘Interm,’ ‘Late,’ and ‘Joint’ centralized cases with pseudo-residuals.
The results from both regression and classiﬁcation datasets with various model settings lead to similar conclusions.
4.2 Deep Model Sharing
[Table] We demonstrate that our method is effective for deep models by using MNIST [31] and CIFAR10 [32] image datasets, where we split each image into patches as depicted in Figure 6.
We use
Convolutional Neural Networks (CNN) for both datasets, and the model architecture can be found in Table 8 of the Appendix.
We visualize the performance of CIFAR10 at each assistance round in Figure 4 (a-c).
The number of assistance rounds needed to approach the centralized performance is small (e.g., often within ten).
The experimental results are shown in Tables 2.
GAL signiﬁcantly outperforms the bottom line ‘Alone’ in all the settings.
This is expected since the ﬁrst organization holds partial data and does not receive any assistance under ‘Alone.’
Interestingly, the performance of MNIST for M = 8 drops signiﬁcantly under ‘Alone’ because the organization only holds the left upper image patch, which is usually completely dark, as shown in Figure 6.
Because local deep learning models such as CNN can consume extensive computation space, we propose Deep Model Sharing (DMS) to allow sharing feature extractors of deep models across all iterations to save memory.
In particular, we propose to jointly train the feature extractors with residuals from previous iterations as well as from the current iteration by adding an additional last prediction layer.
The local deep model f tm ( · ) is composed of f tm,o ( f m,e ( · ))) , where f tm,o ( · ) is the last output layer at assistance round t and f m,e ( · ) is the deep feature extractor shared across multiple assistance rounds.
For each assistance round, local organizations ﬁts pseudo-residuals across previous assistance rounds r 1: t 1 with f 1: tm = argmin
f m ∈F m E N (cid:96) m (cid:0) r 1: t 1 , f 1: tm,o ( f m,e ( x m )) (cid:1) .
It is worth mentioning that we do not expect such trade-off GAL DMS to consistently outperform AL and GAL because a single feature extractor may not well ﬁt residuals across many iterations.
The results in Table 2 show that sharing the feature extractor across multiple assistance rounds can still outperform the ‘Alone’ case.
Thus, DMS can provide a trade-off between predictive performance and computation space.
The detailed comparisons can be found in Table 14.
4.3 Comparison with AL
As demonstrated in our extensive experiments, the proposed GAL outperforms AL in terms of predictive performance.
In particular, AL converges not only worse but also slower than GAL.
This is due to the fact that AL uses a constant assisted learning rate and trains participating organizations in a sequential manner.
Moreover, sequentially training participating organizations also requires much more computation and communication overhead.
We compare the computation and communication complexity between AL and GAL under the constraint of the same communication cost as demonstrated in Table 14.
Because AL sequentially trains each organization while GAL allows organizations to train locally in parallel, the computation time and communication round of AL is
M × those of GAL.
The GAL with Deep Model Sharing (GAL DMS ) saves T × computation space by sharing the feature extractor of deep models.
In summary, GAL generalizes the problem scope, reduces the computation and communication complexity, and achieves signiﬁcantly better results.
[Figure]
A constant gradient assisted learning rate ( η = 1 ) converges much slower.
The gradient assistance weights exhibits interpretability of the importance of organizations as the weights of the central image patches ( m = { 2 , 3 , 6 , 7 } ) of CIFAR10 dataset are larger than the boundary patches ( m = { 1 , 4 , 5 , 8 } ) in the ﬁrst few rounds.
More results can be found in the Appendix.
4.4 Case Studies
The results in Table 3 demonstrate the utility of GAL in various practical applications.
We illustrate the results across multiple assistance rounds of MIMICL in Figure 4 (d-f).
GAL signiﬁcantly outperforms ‘Alone’ and ‘AL.’
Our method also performs close to the centralized baselines.
[Table] [Figure] [Table] Three-dimensional object recognition We use the shape representation of 3D objects for recognition from a collection of rendered views on 2D images.
We generate M = 12 2D camera views of
ModelNet40 [33] and ShapeNet55 [34] datasets following [35].
Cameras are treated as decentralized learners in our experiments.
We adopt the same CNN architecture used for MNIST and CIFAR datasets.
It is worth mentioning that ‘Joint’ of ModelNet40 and ShapeNet55 performs considerably worse than other baselines because ‘Joint’ uses a single CNN feature extractor to process all images from twelve angles [35].
Medical time series forecasting We use the in-hospital dataset MIMIC3 [36], where the task aims to predict the Length-of-stay (MIMICL) and Mortality rate (MIMICM) of the ICU stays of patients.
We follow the benchmark work [37] to process raw data and split the features into four organizations, including 1) microbiology measurement, 2) demographic information, 3) body measurement, and 4) International Classiﬁcation of Diseases (ICD).
We use MAD to evaluate the result of MIMICL (regression) and the Area Under the Curve-Receiver Operating Characteristics (AUROC) to evaluate the result of MIMICM (imbalanced binary classiﬁcation).
Our backbone model (LSTM) is the same as the one used in the benchmark work [38].
It is worth noting that the data features among the four organizations are aligned with time stamps which is a natural identiﬁer for time series forecasting.
4.5 Ablation Studies
Local objective functions Our method does not require local regression loss functions (cid:96) m to be shared with other organizations, but they are supposed to be beneﬁcial for assisting Alice.
We conduct ablation studies on the choice of various local regression loss functions.
In this study shown in
Table 4, we use different regression functions (cid:96) q ( y, ˆ y ) =
| y − ˆ y | q where q ∈ { 1 , 1 . 5 , 2 , 4 } .
( (cid:96) q 1 , (cid:96) q 2 ) allows half of the organizations to use (cid:96) q 1 while the other half to use l q 2 .
The results show that the classiﬁcation task generally works better with q > 1 .
[Table] Privacy enhancement Our learning framework does not require organizations to share local data, models [Table]
and objective functions.
One potential limitation of our approach is that assisting organizations may infer Alice’s information based on the shared pseudo-residuals.
Therefore, we suggest to further enhancing privacy by adopting the framework of Differential Privacy (DP) [39] or Interval
Privacy (IP) [40].
We use the Laplace mechanism with α = 1 for DP and set the number of intervals of IP to be 1 .
We add a moderate amount of noise to the pseudo-residuals in hindsight.
In Tables 5, we demonstrate that privacy-enhanced GAL can still outperform the ‘Alone’ case.
[Table]
Gradient assisted learning rate We illustrate the gradient assisted learning rate of CIFAR10 and
MIMICL datasets at each assistance round in Figure 4(b,e).
We perform a line search for the gradient assisted learning rate with the Limited-Memory BFGS optimizer, which improves the convergence rates compared with SGD and Adam.
We conduct an ablation study using a constant gradient assisted learning rate ( η = 1 ).
As shown in Figure 4(a,d), the constant gradient assisted learning rate leads to a convergence much slower than the line-search method.
Fast convergence is desirable since the computation and communication cost increases with the number of assistance rounds.
To determine the maximal number of assistance rounds T for the service receiver, we can run the GAL procedure until the gradient assisted learning rate becomes small.
When the gradient assistance rate is low, as shown in Figure 4(e), the overarching loss converges to zero.
In this light, an organization may stop receiving assisted learning when the gradient assisted learning rate is below a threshold.
Gradient assistance weights We show the gradient assistance weights of CIFAR10 and MIMICL datasets at each assistance round in Figure 4(c,f).
The results of MNIST and CIFAR10 show that the gradient assistance weights exhibit interpretability of the importance of organizations because the image patches with dominant contributions are m = (2 , 3 , 6 , 7) (colored in red).
These image patches correspond to the center of the original image, which matches our intuition appealingly.
The weights converge to uniform as the residuals of later iterations are small.
We also conduct an ablation study of the gradient assistance weights in Figure 5 by adding noises (Gaussian with zero mean and σ 2 variance, σ ∈ { 1 , 5 } ) to the output of a randomly chosen half of the clients during learning and prediction.
Adding noise simulates realistic scenarios where some assisting organizations may be noninformative or inject noise.
We summarize the ablation study results in Table 6.
The results show that the GAL with gradient assistance weights is more robust than the GAL with a direct average [Table] [Table]
Conclusion
We proposed Gradient Assisted Learning, a decentralized learning method for multiple organizations to collaborate without sharing data, models, and objective functions.
The proposed method can signiﬁcantly outperform the local learning baselines and achieve near-oracle performance as if data were centralized on various datasets.
All participants form a shared community of interest by autonomously building their own model and iteratively ﬁtting the gradients of the overarching objective function.
We also demonstrate asymptotic convergence analysis and practical case studies of GAL.
Moreover, this is achieved without any constraints on the models selected by the collaborating organizations. Acknowledgments The work of Enmao Diao and Vahid Tarokh was supported by the Ofﬁce of Naval Research (ONR) under grant number N00014-18-1-2244. The work of Jie Ding was supported by the National Science
Foundation (NSF) under grant number ECCS-2038603.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 4.2 for computation space and Section B for theoretical analysis.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We do not foresee any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section B. (b) Did you include complete proofs of all theoretical results?
[Yes] See Section B. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide source codes in the supplementary material.
We use public available datasets.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section C in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The numerical standard error are shown in brackets of tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] One Nvidia 1080TI is enough for one experiment run.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the public available datasets we use.
(b) Did you mention the license of the assets?
[No] They should be available publicly.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We use public available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] We use public available datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
A Additional Discussions A.1 Application scenario
As shown in Figure 1, Alice, the service receiver (bank) squared in red dashed line, is the organization to be assisted.
Before learning, it broadcasts identiﬁcation (ID) to locate and align vertically distributed data held by other organizations.
At the beginning of the Learning Stage, the bank deterministically initializes the values of F 0 ( x ) to be the unbiased estimate of y 1 , namely
F 0 ( x ) =
E N ( y 01 ) .
For the regression task, F 0 ( x ) is a single scalar.
For classiﬁcation task, F 0 ( x ) is a point in the K -dimensional simplex P K .
During the ﬁrst assistance round in the Learning Stage, the bank computes pseudo-residual rand broadcasts it to other organizations (e.g., hospital, mall, and insurance company).
Then, all the organizations, including the bank, will ﬁt a new local model with 1) their local data, 2) the pseudo-residual r 11 , and 3) their local regression loss function (cid:96) m
(e.g., (cid:96) 2 -loss) to ﬁt the pseudoresidual.
We note that organizations have complete autonomy on model ﬁtting.
In particular, they can choose their own learning algorithms and models by considering their resources (e.g., computation power).
Next, the bank will aggregate all the predictions from each organization’s local models by optimizing a weight vector w 1: M referred to as gradient assistance weights.
As previously discussed in Equation (5), we approximate the oracle gradient (operated on centralized data, in hindsight) with a weighted average of those predictions from organizations.
We then numerically search for the learning rate η .
This process can be iterated multiple times until the learning rate is low or the validation loss is satisfactory.
During the Prediction Stage, organizations will predict with trained models at every assistance round and transmit their predictions to the bank.
Similar to the Learning Stage, the synchronization of each organization is unnecessary .
The bank computes the ﬁnal prediction with gradient assistance weights, learning rates, and received predictions.
A.2 Future work on adversarial learning
In this work, we have considered settings where the participating organizations are cooperative, or some of them receive noisy inputs (pseudo-residuals) or create noisy outputs (ﬁtted pseudo-residuals).
More experimental studies are included in Section D.4 of the Appendix.
Nevertheless, we have not considered adversarial scenarios during Assisted Learning.
In adversarial scenarios, one or more organizations may be malicious or subject to an adversarial attack, e.g., in training data, test data, or models at training or prediction stages.
Compared with conventional adversarial learning settings that often involve one learner, the proposed decentralized learning framework potentially offers more avenues for adversarial behavior.
Inspired by the existing literature on adversarial learning, we brieﬂy comment on the following adversarial GAL problems that may deserve future study.
• Adversarial examples [41–45] refer a type of prediction-stage attack that the intended input data (e.g., an image) is slightly perturbed to cause an already-trained model to make a false prediction.
In GAL, if a participant, Bob, has a large assistance weight (at one or more rounds), it will contribute non-trivially to the ﬁnal prediction of Alice.
In this case, Bob’s adversarially perturbed future input will also affect Alice’s prediction accuracy, especially when the weights are large.
To enhance robustness against such an attack, the participants may use a minimax-based robust local training [46].
• Backdoor attacks [47–56] aim to disrupt the prediction performance on speciﬁc sub-populations or target labels (e.g., from a stop sign to a speed sign) without degrading accuracy on most of the input data regimes.
Backdoor attacks often assume the adversary can inject crafted perturbations into the training data (also known as a “backdoor trigger”), and the learning task is classiﬁcation.
While this attack may occur to any participating organization at the training stage, it is unclear how to devise backdoor triggers for GAL participants that only solve regression problems at each round.
• Data poisoning attacks [57–62] aim to deteriorate the overall prediction performances of Alice.
Compared with backdoor attacks, a poisoning attack is untargeted and often occurs in the training stage.
We conjecture that this type of attack is relatively easier to address in a practical GAL system since the gradient assistance weights may assign small weights for those participants that are not trained well, in contrast with the conventional setting where there is only one learner and one dataset. • Model-stealing attacks [63–68] (also known as model extractions) refer to the unwanted reconstruction of a trained machine learning model through information exchanges.
In GAL, Alice may receive assistance from Bob to steal Bob’s local model using queries and responses in the prediction stage.
Likewise, Bob may steal Alice’s local model by participating in the GAL system initialized by Alice.
Apart from single-model stealing, Alice may also perform multi-model stealing, aiming to learn Bob’s capability to generate predictive models for different pseudo-labels across rounds.
If successful, Alice can imitate Bob’s functionality and assist other learners as if she were Bob
B Theoretical Analysis
To develop a convergence analysis of the GAL algorithm, we use the following notations.
We still let
F m (for each m = 1 , . . .
, M ) denote a set of real-valued functions deﬁned on organization m ’s data x m .
For notational simplicity, for each f m ∈ F m , we also treat it as a function of the (artiﬁcially) extended variable x
= [ x 1 , . .
. , x M ] .
So, we may write a function in the form of f 1 + f 2 , which basically means [ x 1 , x 2 ] (cid:55)→ f 1 ( x 1 ) + f 2 ( x 2 ) .
Let L denote the overarching loss function to minimize (for the agent to assist), and P M the probability simplex.
As a summary, we abstract the core steps in Algorithm 1 below.
For each organization m at assistance round t , it optimizes each local model by solving
[Equation]
Alice then gathers predictions f tm , m = 1 , . . .
M from all the organizations and optimizes the gradient assistance weights and learning rate by solving
[Equation]
At round t = 0 , we initialize with any F 0 ∈ F 1 .
At each round t , each organization ﬁrst runs a greedy boosting step to obtain ( α t , f tm ) .
The f tm will be sent to us (the organization to assist).
Then, we run another greedy step to optimize the assistance weights ˆ w t and learning rate ˆ η t , with ﬁxed f tm , m = 1 , . . .
, M .
The weighted function will be added to F t − 1 to generate the latest F t .
For each m , we let
[Equation]
which is the function space formed from linear combinations of elements in F m .
Let
[Equation]
denote the linear span of the union of F 1 , . .
F M .
An equivalent way to write it is span ( ∪ Mm =1 F m ) .
We will show the following convergence result.
With a suitable choice of step parameters a t and regularity conditions of the loss L , the abstract form of GAL can produce F t that asymptotically attains the minimum loss within the function class span ( F 1 , . .
. , F M ) .
We make the following technical assumptions.
(A1)
The loss (functional) f
(cid:55)→ L ( f ) is convex and differentiable on F , with gradient ∇L .
Also, for all f ∈ span ( F 1 , . . .
, F M ) and g ∈ ∪ M m =1 F m , the function u (cid:55)→ L ( f + ug ) has a second order derivative ∂ 2 L ( f + ug ) /∂u
2 , and it is upper bounded by a ﬁxed constant C
Theorem 1 : Under Assumptions (A1) and (A2), the GAL algorithm satisﬁes L ( F t ) → inf f ∈ span ( F 1 ,..., F M ) L ( f ) as t → ∞ , with a convergence rate at the order of O ( (cid:80) tτ =
1 ( a 1: τ /a 1: t ) a 2 τ ) .
Remarks on Theorem 1 : The result says that with suitable control of the learning rates, the greedy procedure in Algorithm 1 can converge to the oracle one could obtain within span ( F 1 , . .
. , F M ) .
Suppose that an organization, say the one indexed by m = 1 , does not collaborate with others.
Likewise, we have the convergence for that particular organization, lim t →∞ L ( F t ) =
inf f ∗ ∈ span ( F 1 ) L ( f ∗ ) .
It can be seen that the GAL will produce a signiﬁcant gain for this organization as long as
[Equation]
It is conceivable that (8) is easy to meet in many practical scenarios since each F m is operated on a particular modality of data that belongs to organization m .
On the other hand, a skeptical reader may wonder how the GAL solution compares with a function learned from the pulled data.
It is possible that the global minimum of L (over functions that operate on the pulled data) does not belong to span ( F 1 , . .
. , F M ) .
If that is the case, the best we can do is to ﬁnd f that attains the limit inf f ∈ span ( F 1 ,...,
F M ) L ( f ) .
This is a limitation due to the constraint that organizations cannot share data and the additive structure of span ( F 1 , . . .
, F M ) .
Fortunately, in various real-data experiments we performed, the GAL often performs close to the centralized learning within only a few assistance rounds.
In the technical result, we could allow the approximate minimization of (6) and (7), meaning that the loss of the produced solution is δ t -away from the optimal loss.
In that case, it can be veriﬁed that (cid:80) ∞ t =1 δ t < ∞ is sufﬁcient to derive the same asymptotic result in Theorem 1.
The proof of Theorem 1 uses the same technique as was used in [69].
The technical result here is nontrivial, because f tm ( m = 1 , . .
, M ) in each round t are not jointly minimized with ˆ w t and ˆ η t in (7), and thus their linear combination may not be the most greedy solution of minimizing
L ( F t − 1 + f ) within f ∈ span ( F 1 , . .
. , F M ) .
Proof of Theorem 1 :
Let f ∗ ∈ span ( F 1 , . .
. , F M ) be an arbitrary ﬁxed function.
It is introduced for technical convenience and can be treated as the function that (approximately) attains the inﬁmum of L ( f ) .
For every f ∈ span ( F 1 , . . .
, F M ) , we deﬁne the following norm with respect to the basis functions,
[Equation]
where (cid:107) µ (cid:107) 1 denotes the abstract sum of its entries, namely (cid:80) m
=1 ,...,M,j =1 ,...,K m | µ m,j | .
For each t , let S t
⊂
∪ Mm =1 F m denote the ﬁnite set of functions such that
1) f τm ∈ S t for all 0 < τ < t , and 2) f ∗ = (cid:80) g ∈ S t µ gf ∗ g ( µ g ∈ R ), with (cid:107) µ f ∗ (cid:107) 1 ≤ (cid:107) f ∗ (cid:107) 1 + ε .
Note that S t exists due to the deﬁnition of (cid:107)·(cid:107) 1 and the construction of each f τm .
Suppose that F t −admits the representation F t − {+
+} = (cid:80) g ∈ S t µ g F t − 1 g .
From (7), we have
[Equation]
(cid:1) ,
[Equation]
Meanwhile, it follows from (6) that for each m , and each g ∈ S t ∩ F m ,
[Equation]
(10) where s g ∆ = sign ( µ gf ∗ − µ gF t − 1 ) .
Combining (9) and (10), we obtain
L
[Equation]
F
[Equation]
Applying Taylor expansion to f (cid:55)→ L ( f ) at f = F t − 1 , and invoking (11) and Assumption (A1), we have
[Equation]
for all sufﬁciently small a t > 0 .
Let (cid:107) µ f ∗ − µ F
[Equation] [Equation]
=
[Equation]
S
[Equation]
| µ gf ∗ − µ gF t − 1 | .
Multiplying both sides by | µ gf ∗ − µ gF t − 1 | , and add up all the g ∈ S t , we have
[Equation]
where the last inequality is due to the convexity of L .
If (cid:107) µ f ∗ − µ F t − 1 (cid:107) 1 = 0 , F t − 1 already converges to f ∗ .
Otherwise, we rearrange (13) to obtain
[Equation]
where the last inequality is due to the triangle inequality, the way F t − 1 is constructed, and the fact that ε can be arbitrarily chosen.
Here, we deﬁned a 0 ∆ = 0 .
Let a 1: t = (cid:80) tτ
=1 a τ for each t ≥ 1 .
Applying (15) and the Lemma 4.2 in [69], we have
[Equation]
Since f ∗ is arbitrarily chosen, it can be seen from Inequality (16) and Assumption (A2) that lim t [Equation]
L [Equation] F [Equation]
) =
inf [Equation]
[Equation]
L
[Equation]
f [Equation] ) , and the rate of convergence is at the order of
O ( (cid:80)
[Equation]
C Experimental Setup C.1 Dataset
In Table
[Table]
we illustrate the statistics of datasets used in our experiments.
In Figure
[Table]
we show how
MNIST and CIFAR10 images are split into 2 , 4 , and 8 image patches.
The left upper image patch (labeled as [1] ) of the MNIST image is less informative, which demonstrates that an organization with little informative data can leverage other organizations’ local data and models.
The central image patches (labeled [2 , 3 , 6 , 7] ) of MNIST and CIFAR10 images are more informative than others, which leads to larger corresponding gradient assistance weights [Table] number
[Table]
M = M =
M =
[Table] Collaborator
M = M =
M =
ServiceReceiver Collaborator    2            2   48 6(a) (b)
The left upper image patch (labeled [1] ) of MNIST images is less informative in general.
In contrast, the central image patches (labeled [2 , 3 , 6 , 7] ) of MNIST and CIFAR10 images are more informative.
C.2 Model and hyperparameters
Table 8 summarizes the deep neural network architecture used for the MNIST, CIFAR10, ModelNet40, and ShapeNet55 datasets.
Table 9 shows the hyperparameters used in our experiments.
[Table] [Table]
D Experimental Results D.1 Model Autonomy
In Tables 10 and
[Table]
we demonstrate the results of our experiments related to model autonomy for
M = 2 and 4 respectively.
Our method signiﬁcantly outperforms the baselines ‘Alone’ and ‘AL.’
The results also demonstrate that with GAL, an organization with little informative data and free choice of its local model (model autonomy) can leverage other organizations’ local data and models and even achieve near-oracle performance.
[Table]
[Table]
[Table]
D.2 Deep Model Sharing
In Tables 12 and
[Table]
we demonstrate the results of our experiments related to deep model sharing for M = 2 and 4 respectively.
The results show that sharing the feature extractor across multiple assistance rounds can still outperform the ‘Alone’ case.
Thus [Table] DMS can provide a trade-off between predictive performance and computation space.
[Table] [Table]
D.3 Comparison with AL
In Table
[Table]
we demonstrate the comparison of computation and communication complexity between
GAL and AL.
We compare the computation and communication complexity between AL and GAL under the constraint of the same communication cost as demonstrated.
Because AL sequentially trains each organization while GAL allows organizations to train locally in parallel [Table] the computation time and communication round of AL is M × those of GAL.
The GAL with Deep Model Sharing (GAL DMS ) saves T × computation space by sharing the feature extractor of deep models.
In summary,
GAL generalizes the problem scope, reduces the computation and communication complexity
[Table]
and achieves signiﬁcantly better results.
[Table]
D.4 Ablation studies D.4.1 Privacy enhancement
Our learning framework does not require organizations to share local data, models, and objective functions.
One potential limitation of our approach is that assisting organizations may infer Alice’s information based on shared the pseudo-residuals.
Therefore, we suggest further enhancing privacy by adopting the framework of Differential Privacy (DP) [39] or Interval Privacy (IP) [40].
We use the Laplace mechanism with α = 1 for DP and set the number of intervals of IP to be 1 .
We add a moderate amount of noise to the pseudo-residuals in hindsight.
In Tables 15 and 16, we demonstrate that privacy-enhanced GAL can still outperform the ‘Alone’ case
[Table] [Table]
[Table]
D.4.2 Noisy training with gradient assistance weights
To optimize gradient assistance weights, we use the Adam optimizer and enforce the parameters to sum to one by using the softmax function.
The cost to optimize the gradient assistance weights w and gradient assisted learning rate η is often negligible compared with the cost to ﬁt the pseudo-residuals since the number of parameters involved in w ∈ R M and η ∈ R 1 is small.
In Tables 17 and
[Table]
we demonstrate the results of our ablation studies of gradient assistance weights by adding noise to the predicted pseudo-residuals (namely the outputs) from half of the organizations.
In Tables 19-21, we demonstrate the results of our ablation studies of gradient assistance weights when half of the organizations have no predictive power for the target,
i
[Table]
e., data features sampled from N (0 , 1)
[Table]
[Table] [Table]
[Table]
[Table]
[Table] [Table]
[Table]
D.5 Additional Results
In Figure 7- [Figure]
we illustrate results of all datasets (maximal M )
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]