GAL: Gradient Assisted Learning for Decentralized Multi-Organization Collaborations
Anonymous Author(s)
Abstract
Collaborations among multiple organizations, such as ﬁnancial institutions, medicalcenters, and retail markets in decentralized settings are crucial to providing improved service and performance.
However, the underlying organizations may havelittle interest in sharing their local data, models, and objective functions.
Theserequirements have created new challenges for multi-organization collaboration.
In this work, we propose Gradient Assisted Learning (GAL), a new method formultiple organizations to assist each other in supervised learning tasks withoutsharing local data, models, and objective functions.
In this framework, all participants collaboratively optimize the aggregation of local loss functions, and eachparticipant autonomously builds its own model by iteratively ﬁtting the gradientsof the overarching objective function.
We also provide asymptotic convergenceanalysis and practical case studies of GAL.
Experimental studies demonstrate that
GAL can achieve performance close to centralized learning when all data, models,and objective functions are fully disclosed.
Introduction
One of the main challenges in harnessing the power of big data is the fusion of knowledge fromnumerous decentralized organizations that may have proprietary data, models, and objective functions.
Due to various ethical and regulatory constraints, it may not be feasible for decentralized organizationsto centralize their data and fully collaborate to learn a shared model.
Thus, a large-scale autonomousdecentralized learning method that can avoid data, models, and objective functions transparency maybe of critical interest.
Cooperative learning may have various scientiﬁc and business applications [1].
As illustrated in
Figure 1, a medical institute may be helped by multiple clinical laboratories and pharmaceuticalentities to improve clinical treatment and facilitate scientiﬁc research [2,3].
Financial organizationsmay collaborate with universities and insurance companies to predict loan default rates [4].
Theorganizations can match the correspondence with common identiﬁers such as user identiﬁcationassociated with the registration of different online platforms, timestamps associated with differentclinics and health providers, and geo-locations associated with map-related trafﬁc and agriculturaldata.
With the help of our framework, they can form a community of shared interest to provide better
Machine-Learning-as-a-Service (MLaaS) [5,6] without transmitting their local data, models, andobjective functions.
The main idea of Gradient Assisted Learning (GAL) is outlined below.
In the training stage, theorganization to be assisted, denoted by Alice, will calculate a set of ‘residuals’ and broadcast theseto other organizations.
These residuals approximate the fastest direction of reducing the trainingloss in hindsight.
Subsequently, other organizations will ﬁt the residuals using their local data,models, and objective functions and send the ﬁtted values back to Alice.
Alice will then assignweights to each organization to best approximate the fastest direction of learning.
Next, Alice will
ProprietaryModel
PrivateData
line-search for the optimal gradient assisted learning rate along the calculated direction of learning.
The above procedure is repeated until Alice accomplishes sufﬁcient learning.
In the inference stage,other organizations will send their locally predicted values to Alice, who will then assemble themto generate the ﬁnal prediction.
We show that the number of assistance rounds needed to attain thecentralized performance is often small (e.g., fewer than ten).
This is appealing since GAL is primarilydeveloped for large organizations with rich computation resources.
A small number of interactionswill reduce the communications and networking costs.
Our main contributions are summarized below.
• We propose a Gradient Assisted Learning (GAL) algorithm that is suitable for large-scale
autonomous decentralized learning.
It can effectively exploit task-relevant information
preserved by vertically decentralized organizations.
Our method enables simultaneous
collaboration among organizations without sharing data, models, and objective functions.
• We provide asymptotic convergence analysis and practical case studies of GAL.
For the case
of vertically distributed data, GAL generalizes the classical Gradient Boosting algorithm.
• Our proposed method can signiﬁcantly outperform learning baselines and achieve near-
oracle performance on various benchmark datasets.
Compared with existing works, GAL
does not need frequent synchronization of organizations.
It also signiﬁcantly reduces the
computation and communication overhead.
Related work
Multimodal Data Fusion Vertically distributed data can be viewed as multimodal data with modalities provided in a distributed manner to different learners/organizations.
Standard multimodal datafusion methods include the early, intermediate, and late data fusions [7,8].
These methods concatenatedifferent modes of data at the input, intermediate representation, and ﬁnal prediction levels.
However,these data fusion methods in decentralized settings often require organizations to share the task labelsto train their local models synchronously.
In contrast, our method presented below only requires thatorganizations asynchronously ﬁt some task-related statistics named pseudo-residuals to approximatethe direction of reducing the global training loss in hindsight.
Gradient Boosting Our approach was inspired by Gradient Boosting [9,10], where weak learners aresequentially trained from the same dataset and aggregated into a strong learner.
In our learning context,each organization uses side-information from heterogeneous data sources to improve a particularlearner’s performance.
Our method can be regarded as a generalization of Gradient Boosting toaddress decentralized learning with vertically distributed data.
Federated Learning Federated learning [11–16] is a popular distributed learning framework.
Itsmain idea is to learn a joint model by averaging locally learned model parameters.
It avoids theneed for the transmission of local training data.
Conceptually, the goal of Federated Learning is toexploit the resources of edge devices with communication efﬁciency.
Vertical Federated Learningmethods split sub-networks for local clients to jointly optimize a global model [17–22].
Thesemethods can be viewed as federated learning with an intermediate data fusion method, and the central server will have access to the true labels.
In order to converge, these methods typically require veryfrequent batchwise synchronization of backward gradients [22].
Frequent batchwise synchronizationis critical for vertical federated learning because the local model at each client constitutes a part ofthe globally backpropable model, and one client’s local update may not decrease the overall loss.
Incontrast, our proposed method trains multiple autonomous local models with pseudo-residuals, eachcontributing to a small portion of the overarching loss.
Each round of updates will decrease the loss.
Consequently, our method can achieve desirable performance with significantly fewer communicationrounds without a global backpropable model.
Assisted Learning Assisted Learning (AL)
[23] is a decentralized collaborative learning frameworkfor organizations to improve their learning quality.
In that context, neither the organization beingassisted nor the assisting organizations share their local models and data.
The original AL methodology applies to regression tasks. It is derived from a linear projection perspective, and its convergenceto the oracle performance was theoretically justiﬁed for linear regression models with quadratic loss.
Inspired by Gradient Boosting, the proposed Gradient Assisted Learning (GAL) is a general methodfor multiple organizations to assist each other in supervised learning scenarios.
Overall, AL and GALshare similar motivations and concepts but significantly differ from methodological and theoreticalperspectives. More speciﬁcally, the novelties of GAL include: 1) generalization from regressionloss to any differentiable loss for supervised learning, 2) allowing for local objective functionsat each organization, 3) generalization from a sequential protocol between two organizations toparallel aggregation across multiple organizations, 4) introduction of deep model sharing for reducingcomputation and memory costs, 5) introduction of the assisted learning rate for fast convergence, and6) theoretical analysis of GAL’s convergence properties.
Gradient Assisted Learning
3.1 Notation
Suppose that there are N data observations independently drawn from a joint distribution p xy =
p x p y | x , where y ∈ Y and x ∈ R d respectively represent the task label and feature variables, andd is the number of features.
For regression tasks, we have Y = R .
For K -class classiﬁcationtasks, Y = { e 1 , . . .
, e K } , where e k is the canonical vector representing the class k , k = 1 , . . .
, K .
Let E and E N denote the expectation and empirical expectation, respectively.
Thus, E N g ( y, x ) ∆ =
N − 1 (cid:80) Ni =1 g ( y i , x i ) for any measurable function g , where ( y i , x i ) are i.i.d.
observations from p xy .
Suppose that there are M organizations.
Each organization m only holds X m , a sub-vector of
X (illustrated in Figure 2).
In general, we assume that the variables in X 1 , . . .
, X M are disjoint in thepresentation of our algorithm, although our method also allows for the sharing of some variables.
Forexample, one organization may observe demographic features for a mobile user cohort, and anotherorganization holds health-related features of that cohort.
Without loss of generality, we supposethat Alice, the organization to be assisted, has local data x 1 and task label y 1 , while other M − 1organizations are collaborators which assist Alice and have local data x 2 . . .
x M .
[Figure]
[Figure]
For m = 1 , . . .
, M , let { x i,m } Ni =1 denote the available data to the organization m .
Thus, N objectsare simultaneously observed by M organizations, each observing a subset of features from thex ∈ R d .
Alice also has local task labels { y i, 1 } Ni =1 for training purposes.
Let F m and L m respectively denote supervised function class (such as generalized linear functionsor neural networks) and the local objective function of organization m .
We will assume that L m isdifferentiable.
Without loss of generality, we assume that Alice denotes organization 1 , who will beassisted.
Without assistance from other organizations, Alice would learn a model that minimizes the following empirical risk,
[Equation]
Note that the above formulation only involves Alice’s local data x 1 and local model (as represented by
F 1 and L 1 ).
In an oracle case, Alice would be able to operate on other organizations’ data x 2 , . . .
, x Mas well.
Recall that x represents the ensemble of all the available data variables.
In general, the mostideal case for Alice is to minimize the following empirical risk,
[Equation]
where F is a supervised function class deﬁned on the space of x .
In reality, Alice has no access to the complete data and model resources of other organizations.
Inthis light, she shall be happy to crowdsource the learning task to other organizations to cooperativelybuild a model in hindsight, without the need to share any organization’s local data or models.
In theprediction stage, Alice can collect the information needed to form a ﬁnal prediction to hopefullyachieve a performance that signiﬁcantly improves over her single-organization performance.
To thisend, we will develop a solution for Alice to achieve such a goal.
We will include detailed derivations and discussions of our solution in Subsection 3.3.
For readability,we summarize notations that will be frequently used in the exposition below.
Our method willrequire Alice to occasionally send continuous-valued vectors r 1 =
[ r i, 1 ] Ni =1 ∈ R N × K to eachorganization m at each communication round.
These residuals, to be elaborated in the next subsection,approximate the fastest direction of reducing the training loss in hindsight, namely a sample versionof ∂L 1 ( y 1 , F ( x ))
/∂F
( x ) given Alice’s estimation of F at a particular time (round).
Upon the inputof these residual vectors, the organization m will locally learn a supervised function f m that mapsfrom its feature space to the residual space.
With a slight abuse of notation, we also refer to f m as thelearned model.
To this end, the organization m will perform the empirical risk minimization
[Equation]
to obtain a locally trained model f m .
Here, F m and (cid:96) m respectively denote the supervised functionclass and loss function of the organization m .
We note that (cid:96) m are local regression loss functionsfor ﬁtting the pseudo-residual r 1 and may not necessarily be the same as L 1 for ﬁtting true labels.
For example, L 1 may be the cross-entropy loss for classiﬁcation of label y , while (cid:96) 1: M could bethe squared loss for regression of the response r 1 .
The above local training (optimization) is oftenperformed using the stochastic gradient descent (SGD) algorithm.
In our assisted learning context,
L 1 , F m , and (cid:96) m are proprietary local resources that cannot be shared across organizations.
3.3 The GAL Algorithm
We ﬁrst introduce the derivation of the GAL algorithm from a functional gradient descent perspective.
Then, we cast the algorithm into pseudocode and discuss each step.
Consider the unrealistic case that
Alice has all the data x needed for a centralized supervised function F : x (cid:55)→ F ( x ) .
Recall that thegoal of Alice is to minimize the population loss E p x,y L 1 ( y, F ( x )) over a data distribution p x,y .
Ifp x,y is known, starting with an initial guess F 0 ( x ) , Alice would have performed a gradient descentstep in the form of
[Equation]
where the equality holds under the standard regularity conditions of exchanging integration anddifferentiation.
Note that the second term in (4) is a function on R d .
However, because Alice onlyhas access to her own data x 1 , the expectation E p x,y cannot be realistically evaluated.
Therefore,we need to approximate it with functions in a pre-speciﬁed function set.
In other words, we willﬁnd f from F M that ‘best’ approximates E p x,y ∂∂F L 1 ( y, F ( x )) .
We will show that this is actionablewithout requiring the organizations to share proprietary data, models, and objective functions.
Recall that F m is the function set locally used by the organization m , and x m is correspondinglyobserved portion of x .
The function class that we propose to approximate the second term in (4) is
[Equation]
Input: M decentralized organizations, each holding data { x i,m } Ni =1 (local) corresponding to N objects, the task label { y i, 1 } Ni =1 initially held by the service receiver (Alice local), model class F m (local), gradient assistance weights w (Alice local), assistance rate η (Alice local), overarching loss function L 1 (Alice local), regression loss function (cid:96) m to ﬁt pseudo-residual (local), assistance rounds T .
Learning Stage:
Intialization:
Let t = 0 , and initialize F 0 ( x ) =
E N ( y 1 ) for assistance round t from 1 to T do
Compute pseudo-residual r t
[Equation]
= − (cid:20) ∂L 1 ( y 1 ,F t − 1 ( x ) )
∂F t − 1 ( x ) (cid:21)
Broadcast pseudo-residual r t 1 to other organizations for organization m from 1 to M in parallel do f
[Equation]
[Equation] [Equation]
( x m )) end
Gather predictions f t m ( x m ) , m = 1 , . . .
M , from all the organizations
Optimize the gradient assistance weights
[Equation]
Line-search for the gradient assisted learning rate
[Equation]
end
Prediction Stage:
For each data observation x ∗ , of which x ∗ m is held by organization m :
Gather predictions f tm ( x ∗ m ) , t = 1 , . . .
, T from each organization m , m = 1 , . . .
, M
Predict with
[Equation]
where P M = { w ∈ R M : (cid:80) Mm =1 w m = 1 , w m ≥ 0 } denotes the probability simplex.
The gradientassistance weights w m ’s are interpreted as the contributions of each organization at a particular greedyupdate step.
The gradient assistance weights are constrained to sum to one to ensure the functionspace is compact, and the solutions exist.
User ID
ProprietaryModel
PrivateData
End User ServiceReceiver
Collaborator
B
I I
I
Residual
B
Residual
M
H
B
H
M
Residual
B
H
M
H H H
M M
M M
M
M 3 (a) Learning Stage (b) Prediction Stage
B
B
B 3 B
User ID
We propose the following solution so that each organization can operate on its own local data, model,and objective function .
Alice initializes with a startup model, denoted by F 0 ( x ) =
F 0 ( x 1 , y 1 ) , basedonly on her local data and labels.
Alice broadcasts r 1 (named ‘pseudo residuals’) to each organizationm, m = 2 , · · · , M , who will then ﬁt a local model f m using r 1 .
Each organization will then send theﬁtted values from f m to Alice, who will train suitable gradient assistance weights w m .
Subsequently,
Alice ﬁnds the η in (4) that minimizes her current empirical risk.
The above procedure is iteratedfor a ﬁnite number of rounds until Alice obtains a satisfactory performance (e.g., on validation data).
The validation will be based on the same technique as the prediction stage to be described below.
This training stage is described under the ‘learning stage’ of Algorithm 1.
Note that the pseudocodeis from the perspective of Alice, the service receiver.
For each organization m , it will only need toperform the empirical risk minimization using the label r t 1 sent by Alice at each round t .
In the Prediction/Inference stage (given above in Algorithm 1), other organizations send predictionresults generated from their local models to Alice, who will calculate a prediction result F T ( x ) thatis implicitly operated on x , where T is the number of iteration steps.
The idea of approximating functional derivatives with regularized functions was historically usedto develop the seminal work of gradient boosting [9,10].
When there is only one organization, theabove method reduces to the standard gradient boosting algorithm.
Organizations in our learning framework form a shared community of interest.
Each serviceproviding organization can provide end-to-end assistance for an organization without sharing anyone’sproprietary data, models, and objective functions.
In practice, the participating organizations mayreceive ﬁnancial rewards from the one to assist.
Moreover, every organization in this frameworkcan provide its own task and seek help from others.
As a result, all organizations become mutuallybeneﬁcial to each other.
We provide a realistic example in Figure 3 to demonstrate each step of
Algorithm 1.
We elaborate on the learning and prediction procedures in the Appendix.
We also provide an asymptotic convergence analysis for the GAL algorithm, where the goal is tominimize a loss f
(cid:55)→ L ( f ) over a function class through step-wise function aggregations.
Because ofthe greedy nature of GAL, we consider the function class to be the linear span of organization-speciﬁc
F m .
The following result states that the GAL can produce a solution that attains the inﬁmum of L ( f ) .
More technical details are included in the Appendix.
Theorem 1 Assume that the loss (functional) f
(cid:55)→ L ( f ) is convex and differentiable on F , thefunction u
(cid:55)→ L ( f + ug ) has an upper-bounded second-order derivative ∂ 2 L ( f + ug ) /∂u 2 forall f ∈ span ( F 1 , . .
. , F M ) and g ∈ ∪ Mm =1 F m , and the ranges of learning rates { a t } t =1 , 2 ,...
satisfy (cid:80) ∞ t =1 a t = ∞ , (cid:80) ∞
t =1 a 2 t < ∞ .
Then, the GAL algorithm satisﬁes L ( F t ) →inf f ∈ span ( F 1 ,..., F M ) L ( f ) as t → ∞ , with a convergence rate at the order of O ( (cid:80) tτ = 1 ( a 1: τ /a 1: t ) a 2 τ ) .
Experimental Studies
Baselines Our experiments are performed with four baselines, including ‘Interm’, ‘Late’, ‘Joint’,‘Alone’, and ‘AL’.
‘Interm’ and ‘Late’ refer to intermediate and late data fusions [7,8], respectively.
The intermediate data fusion (‘Interm’) sums up the intermediate features (output before the lastlayer) from each separate feature extractor, and then the aggregated feature is passed into a sharedlast layer to output the ﬁnal prediction.
The late data fusion (‘Late’) sums up the ﬁnal predictionof each separate local model.
Thus, ‘Interm’ works only for deep learning models such as CNNand LSTM by averaging the hidden representation of each local model, while ‘Late’ also worksfor Linear models as it aggregates the output of each local model.
‘Joint’ is the case where all thedata are held by Alice and trained with the Gradient Boosting reduced from GAL.
‘Alone’ is thesingle-agent scenario, where only Alice’s data are used for learning and prediction.
‘AL’ representsthe performance of Assisted Learning [23].
GAL is expected to perform close to the centralizedbaselines including ‘Interm’, ‘Late’, and ‘Joint’ cases while signiﬁcantly outperforming the ‘Alone’and ‘AL’ cases.
The summary statistics of each dataset are elaborated in Table 5 of the Appendix.
Details of learning hyper-parameters are included in Table 7 of the Appendix.
We conducted fourrandom experiments for all datasets with different seeds, and the standard errors are shown in thebrackets of all tables.
4.1 Model Autonomy
Recall that GAL allows each organization to choose its own local model.
We demonstrate theperformance of autonomous local models with UCI datasets downloadable from the scikit-learnpackage [24], including Diabetes [25], Boston Housing [26], Blob [24], Iris [27], Wine [28], Breast
Cancer [29], and QSAR [30] datasets, where we randomly partition the features into 2 , 4 , or 8 subsets.
For all the UCI datasets, we train on 80% of the available data and test on the remaining.
We experiment with Linear, Gradient Boosting (GB), and Support Vector Machine (SVM) for localmodels f m ( · ) with the UCI datasets.
We also demonstrate the performance of the scenario (GB-SVM)where half of the organizations use GB and the other half uses SVM.
The experimental results areshown in Tables 1.
↓ indicates the smaller the better, while ↓ indicates the larger the better. Our
[Table]
method signiﬁcantly outperforms the baselines ‘Alone’ and ‘AL.’
The results also demonstrate thatwith GAL, an organization with little informative data and free choice of its local model (modelautonomy) can leverage others’ local data and models and even achieve near-oracle performance.
We point out that although ‘Interm,’ ‘Late,’ and ‘Joint’ marginally outperform our method, theyrequire training from centralized data.
Our GAL algorithm replaces the true labels used in ‘Interm,’‘Late,’ and ‘Joint’ centralized cases with pseudo-residuals.
The results from both regression andclassiﬁcation datasets with various model settings lead to similar conclusions.
4.2 Deep Model Sharing
We demonstrate our method is effective for deep models by using MNIST [31] and CIFAR10 [32]image datasets, where we split each image into patches as depicted in Figure 6.
We use Convolutional
Neural Networks (CNN) for both datasets, and the model architecture can be found in Table 6 of the
Appendix.
We visualize the performance of CIFAR10 at each assistance round in Figure 4 (a-c).
Thenumber of assistance rounds needed to approach the centralized performance is small (e.g., oftenwithin ten).
The experimental results are shown in Tables 2.
GAL signiﬁcantly outperforms thebottom line ‘Alone’ in all the settings.
This is expected since the ﬁrst organization holds partial dataand does not receive any assistance under ‘Alone’.
Interestingly, the performance of MNIST for
M = 8 drops signiﬁcantly under ‘Alone’ because the organization only holds the left upper imagepatch, which is usually completely dark, as shown in Figure 6.
with CNN model.
The MNIST and CIFAR10 are evaluated with
Accuracy.
GAL DMS represents the results with Deep Model
Sharing.
More results for M = 2 and 4 are in the Appendix.
[Table]
Because local deep learning models such as CNN can consumeextensive computation space, we propose Deep Model Sharing(DMS) to allow sharing feature extractors of deep models acrossall iterations to save memory.
In particular, we propose to jointlytrain the feature extractors with residuals from previous iterationsas well as from the current iteration by adding an additional lastprediction layer.
The local deep model f tm ( · ) is composed off tm,o ( f m,e ( · ))) , where f tm,o ( · ) is the last output layer at assistanceround t and f m,e ( · ) is the deep feature extractor shared acrossmultiple assistance rounds.
For each assistance round, local organizations ﬁts pseudo-residuals across previous assistance roundsr 1: t 1 with f 1: tm = argmin
f m ∈F m E N (cid:96) m (cid:0) r 1: t 1 , f 1: tm,o ( f m,e ( x m )) (cid:1) .
Itis worth mentioning that we do not expect such trade-off GAL DMSto consistently outperform AL and GAL because a single featureextractor may not well ﬁt residuals across many iterations.
Theresults in Table 2 show that sharing the feature extractor acrossmultiple assistance rounds can still outperform the ‘Alone’ case.
Thus, DMS can provide a trade-off between predictive performance and computation space.
Thedetailed comparisons can be found in Table 12.
4.3 Comparison with AL
As demonstrated in our extensive experiments, the proposed GAL outperforms AL in terms ofpredictive performance.
In particular, AL converges not only worse but also slower than GAL.
Thisis due to that AL uses a constant assisted learning rate and trains participating organizations in asequential manner.
Moreover, sequentially training participating organizations also requires muchmore computation and communication overhead.
We compare the computation and communication complexity between AL and GAL under the constraint of the same communication cost as demonstrated in Table 12.
Because AL sequentially trains each organization while GAL allowsorganizations to train locally in parallel, the computation time and communication round of AL is
M × those of GAL.
The GAL with Deep Model Sharing (GAL DMS ) saves T × computation spaceby sharing the feature extractor of deep models.
In summary, GAL generalizes the problem scope,reduces the computation and communication complexity, and achieves signiﬁcantly better results.
[Figure]
A constant gradient assisted learning rate ( η = 1 ) converges much slower.
The gradient assistance weights exhibits interpretability of the importance of organizations as the weights of the central image patches ( m = { 2 , 3 , 6 , 7 } ) of CIFAR10 dataset are larger than the boundary patches ( m = { 1 , 4 , 5 , 8 } ) in the ﬁrst few rounds.
More results can be found in the Appendix.
[Table]
4.4 Case Studies
The results in Table 3 demonstrate the utility of GAL in various practical applications.
We illustratethe results across multiple assistance rounds of MIMICL in Figure 4 (d-f).
GAL signiﬁcantlyoutperforms ‘Alone’ and ‘AL.’
Our method also performs close to the centralized baselines.
Three-dimensional object recognition We use the shape representation of 3D objects for recognition from a collection of rendered views on 2D images.
We generate M = 12 2D camera views of
ModelNet40 [33] and ShapeNet55 [34] datasets following [35].
Cameras are treated as decentralizedlearners in our experiments.
We adopt the same CNN architecture used for MNIST and CIFAR10datasets.
It is worth mentioning that ‘Joint’ of ModelNet40 and ShapeNet55 performs considerablyworse than other baselines because ‘Joint’ uses a single CNN feature extractor to process all imagesfrom twelve angles [35].
Medical time series forecasting We use the in-hospital dataset MIMIC3 [36], where the task aimsto predict the Length-of-stay (MIMICL) and Mortality rate (MIMICM) of the ICU stays of patients.
We follow the benchmark work [37] to process raw data and split the features into four organizations,including 1) microbiology measurement, 2) demographic information, 3) body measurement, and4) International Classiﬁcation of Diseases (ICD).
We use MAD to evaluate the result of MIMICL(regression) and the Area Under the Curve-Receiver Operating Characteristics (AUROC) to evaluatethe result of MIMICM (imbalanced binary classiﬁcation).
Our backbone model (LSTM) is the sameas the one used in the benchmark work [38].
It is worth noting that the data features among fourorganizations are aligned with time stamps which is a natural identiﬁer for time series forecasting.
[Figure]
[Table]
4.5 Ablation Studies
Local objective functions Our method does not require local regression loss functions (cid:96) m to beshared with other organizations, but they are supposed to be beneﬁcial for assisting Alice.
We conductablation studies on the choice of various local regression loss functions.
In this study shown in
Table 4, we use different regression functions (cid:96) q ( y, ˆ y ) =
| y − ˆ y | q where q ∈ { 1 , 1 . 5 , 2 , 4 } .
( (cid:96) q 1 , (cid:96) q 2 )allows half of organizations including to use (cid:96) q 1 while the other half to use l q 2 .
The results show thatclassiﬁcation task generally works better with q > 1 .
Gradient assisted learning rate We illustrate the gradient assisted learning rate of CIFAR10 and
MIMICL datasets at each assistance round in Figure 4(b,e).
We perform a line-search for the gradientassisted learning rate with the Limited-Memory BFGS optimizer which improves the convergencerates compared with SGD and Adam.
We conduct ablation study using a constant gradient assistedlearning rate ( η = 1 ).
As shown in Figure 4(a,d), the constant gradient assisted learning rate leadsto a convergence much slower than the line-search method.
Fast convergence is desirable since thecomputation and communication cost increases with the number of assistance rounds.
To determinethe maximal number of assistance rounds T for the service receiver, we can run the GAL procedureuntil the gradient assisted learning rate becomes small.
When the gradient assistance rate is low, asshown in Figure 4(e), the overarching loss converges to zero.
In this light, an organization may stopreceiving assisted learning when the gradient assisted learning rate is below a threshold.
Gradient assistance weights We show the gradient assistance weights of CIFAR10 and MIMICLdatasets at each assistance round in Figure 4(c,f).
The results of MNIST and CIFAR10 show thatthe gradient assistance weights exhibit interpretability of the importance of organizations becausethe image patches with dominant contributions are m = (2 , 3 , 6 , 7) (colored in red).
These imagepatches correspond to the center of the original image, which matches our intuition appealingly.
Theweights converge to uniform as the residuals of later iterations are small.
We also conduct ablationstudy of the gradient assistance weights in Figure 5 by adding noises (Gaussian with zero meanand σ 2 variance, σ ∈ { 1 , 5 } ) to the output of a randomly chosen half of the clients during learningand prediction.
Adding noise simulates realistic scenarios where some assisting organizations maybe noninformative or inject noise.
We summarize the ablation study results in Tables 16-18 in the Appendix.
The results show that the GAL equipped with gradient assistance weights is more robustthan the GAL with a direct average.
Conclusion
We proposed Gradient Assisted Learning, a decentralized learning method for multiple organizationsto collaborate without sharing data, models, and objective functions.
The proposed method cansigniﬁcantly outperform the local learning baselines and achieve near-oracle performance as ifdata were centralized on various datasets.
All participants form a shared community of interest byautonomously building its own model and iteratively ﬁtting the gradients of the overarching objectivefunction.
We also demonstrate asymptotic convergence analysis and practical case studies of GAL.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 4.2 for computationspace and Section B for theoretical analysis.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We do not foresee any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section B.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section B.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We providesource codes in the supplementary material.
We use public available datasets.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section C in Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] The numerical standard error are shown in brackets of tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] One Nvidia 1080TI is enough for [Table]
one experiment run.
[Table] If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the public
[Table]
available datasets we use.
[Table]
(b) Did you mention the license of the assets?
[No] They should be available publicly. [Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A] We use public available datasets. [Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A] We use public available datasets.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
Appendix for
The Appendix contains additional discussions, technical results, and further experimental results.
A Additional DiscussionsA.1 Application scenario
As shown in Figure 1, Alice, the service receiver (bank) squared in red dashed line, is the organization to be assisted.
Before learning, it broadcasts identiﬁcation (ID) to locate and alignvertically distributed data held by other organizations.
At the beginning of the Learning Stage, thebank deterministically initializes the values of F 0 ( x ) to be the unbiased estimate of y 1 , namely
F 0 ( x ) =
E N ( y 01 ) .
For the regression task, F 0 ( x ) is a single scalar.
For classiﬁcation task, F 0 ( x ) isa point in the K -dimensional simplex P K .
During the ﬁrst assistance round in the Learning Stage, the bank computes pseudo-residual r 11and broadcasts it to other organizations (e.g., hospital, mall, and insurance company).
Then, allthe organizations, including the bank, will ﬁt a new local model with 1) their local data, 2) thepseudo-residual r 11 , and 3) their local regression loss function (cid:96) m
(e.g., (cid:96) 2 -loss) to ﬁt the pseudoresidual.
We note that organizations have complete autonomy on model ﬁtting.
In particular, they canchoose their own learning algorithms and models by considering their resources (e.g., computationpower).
Next, the bank will aggregate all the predictions from each organization’s local models byoptimizing a weight vector w 1: M referred to as gradient assistance weights.
As previously discussedin Equation (5), we approximate the oracle gradient (operated on centralized data, in hindsight)with a weighted average of those predictions from organizations.
We then numerically search forthe learning rate η .
This process can be iterated multiple times until the learning rate is low or thevalidation loss is satisfactory.
During the Prediction Stage, organizations will predict with trained models at every assistance roundand transmit their predictions to the bank.
Similar to the Learning Stage, the synchronization of eachorganization is unnecessary .
The bank computes the ﬁnal prediction with gradient assistance weights,learning rates, and received predictions.
A.2 Future work on adversarial learning
In this work, we have considered settings where the participating organizations are cooperative, orsome of them receive noisy inputs (pseudo-residuals) or create noisy outputs (ﬁtted pseudo-residuals). More experimental studies are included in Section D.4 of the Appendix. Nevertheless, we have notconsidered adversarial scenarios during Assisted Learning.
In adversarial scenarios, one or moreorganizations may be malicious or subject to an adversarial attack, e.g., in training data, test data, ormodels at training or prediction stages.
Compared with conventional adversarial learning settingsthat often involve one learner, the proposed decentralized learning framework potentially offers moreavenues for adversarial behavior.
Inspired by the existing literature on adversarial learning, we brieﬂycomment on the following adversarial GAL problems that may deserve future study.
• Adversarial examples [39–43] refer a type of prediction-stage attack that the intended input data
(e.g., an image) is slightly perturbed to cause an already-trained model to make a false prediction.
In
GAL, if a participant, Bob, has a large assistance weight (at one or more rounds), it will contribute
non
-trivially to the ﬁnal prediction of Alice.
In this case, Bob’s adversarially perturbed futureinput will also affect Alice’s prediction accuracy, especially when the weights are large.
To enhance
robustness against such an attack, the participants may use a minimax-based robust local training [44].
• Backdoor attacks [45–54] aim to disrupt the prediction performance on speciﬁc sub-populations or
target labels (e.g., from a stop sign to a speed sign) without degrading accuracy on most of the input
data regimes.
Backdoor attacks often assume the adversary can inject crafted perturbations into the
training data (also known as a “backdoor trigger”), and the learning task is classiﬁcation.
While thisattack may occur to any participating organization at the training stage, it is unclear how to devisebackdoor triggers for GAL participants that only solve regression problems at each round.
• Data poisoning attacks [55–60] aim to deteriorate the overall prediction performances of Alice.
Compared with backdoor attacks, a poisoning attack is untargeted and often occurs in the trainingstage.
We conjecture that this type of attack is relatively easier to address in a practical GAL systemsince the gradient assistance weights may assign small weights for those participants that are nottrained well (in contrast with the conventional setting where there is only one learner and one dataset).• Model-stealing attacks [61–66] (also known as model extractions) refer to the unwanted reconstruction of a trained machine learning model through information exchanges.
In GAL, Alice mayreceive assistance from Bob to steal Bob’s local model using queries and responses in the predictionstage.
Likewise, Bob may steal Alice’s local model by participating in the GAL system initializedby Alice.
Apart from single-model stealing, Alice may also perform multi-model stealing, aimingto learn Bob’s capability to generate predictive models for different pseudo-labels across rounds.
Ifsuccessful, Alice can imitate Bob’s functionality and assist other learners as if she were Bob.
B Theoretical Analysis
To develop a convergence analysis of the GAL algorithm, we use the following notations.
We still let
F m (for each m = 1 , . . .
, M ) denote a set of real-valued functions deﬁned on organization m ’s datax m .
For notational simplicity, for each f m ∈ F m , we also treat it as a function of the (artiﬁcially)extended variable x =
[ x 1 , . .
. , x M ] .
So, we may write a function in the form of f 1 + f 2 , whichbasically means [ x 1 , x 2 ] (cid:55)→ f 1 ( x 1 ) + f 2 ( x 2 ) .
Let L denote the overarching loss function to minimize(for the agent to assist), and P M the probability simplex.
As a summary, we abstract the core steps in Algorithm 1 below.
For each organization m at assistanceround t , it optimizes each local model by solving
[Equation]
Alice then gathers predictions f tm , m = 1 , . . .
M , from all the organizations, and optimize thegradient assistance weights and learning rate by solving
[Equation]
At round t = 0 , we initialize with any F 0 ∈ F 1 .
At each round t , each organization ﬁrst runs agreedy boosting step to obtain ( α t , f tm ) .
The f tm will be sent to us (the organization to assist).
Then,we run another greedy step to optimize the assistance weights ˆ w t and learning rate ˆ η t , with ﬁxed f tm ,m = 1 , . . . , M . The weighted function will be added to F t − 1 to generate the latest F t .
For each m , we let
[Equation]
which is the function space formed from linear combinations of elements in F m .
Let
[Equation]
[Equation]
[Equation]
denote the linear span of the union of F 1 , . .
F M .
An equivalent way to write it is span ( ∪ Mm =1 F m ) .
We will show the following convergence result.
With a suitable choice of step parameters a t andregularity conditions of the loss L , the abstract form of GAL can produce F t that asymptoticallyattains the minimum loss within the function class span ( F 1 , . .
. , F M ) .
We make the followingtechnical assumptions.
(A1)
The loss (functional) f
(cid:55)→ L ( f ) is convex and differentiable on F , with gradient ∇L .
Also,for all f ∈ span ( F 1 , . . .
, F M ) and g ∈ ∪ Mm =1 F m , the function u (cid:55)→ L ( f + ug ) has a second orderderivative ∂ 2 L ( f + ug ) /∂u 2 , and it is upper bounded by a ﬁxed constant C .(A2)
The ranges of learning rates { a t } t =1 , 2 ,... satisfy (cid:80) ∞
t =1 a t = ∞ , (cid:80) ∞
t =1 a 2 t < ∞ .
Theorem 1 : Under Assumptions (A1) and (A2), the GAL algorithm satisﬁes L ( F t ) →inf f ∈ span ( F 1 ,..., F M ) L ( f ) as t → ∞ , with a convergence rate at the order of O ( (cid:80) tτ = 1 ( a 1: τ /a 1: t ) a 2 τ ) .
Remarks on Theorem 1 : The result says that with suitable control of the learning rates, the greedyprocedure in Algorithm 1 can converge to the oracle one could obtain within span ( F 1 , . .
. , F M ) .
Suppose that an organization, say the one indexed by m = 1 , does not collaborate with others.
Likewise,we have the convergence for that particular organization, lim t →∞ L ( F t ) =
inf f ∗ ∈ span ( F 1 ) L ( f ∗ ) .
Itcan be seen that the GAL will produce a signiﬁcant gain for this organization as long as
[Equation]
It is conceivable that (8) is easy to meet in many practical scenarios since each F m is operated ona particular modality of data that belongs to organization m .
On the other hand, a skeptical readermay wonder how the GAL solution compares with a function learned from the pulled data.
It ispossible that the global minimum of L (over functions that operate on the pulled data) does notbelong to span ( F 1 , . .
. , F M ) .
If that is the case, the best we can do is to ﬁnd f that attains the limitinf f ∈ span ( F 1 ,...,
F M ) L ( f ) .
This is a limitation due to the constraint that organizations cannot sharedata and the additive structure of span ( F 1 , . . .
, F M ) .
Fortunately, in various real-data experimentswe performed, the GAL often performs close to the centralized learning within only a few assistancerounds.
In the technical result, we could allow the approximate minimization of (6) and (7), meaning that theloss of the produced solution is δ t -away from the optimal loss.
In that case, it can be veriﬁed that 656 (cid:80) ∞ t =1 δ t < ∞ is sufﬁcient to derive the same asymptotic result in Theorem 1.
The proof of Theorem 1 uses the same technique as was used in [67].
The technical result hereis nontrivial, because f tm ( m = 1 , . .
, M ) in each round t are not jointly minimized with ˆ w t andˆ η t in (7), and thus their linear combination may not be the most greedy solution of minimizing
L ( F t − 1 + f ) within f ∈ span ( F 1 , . .
. , F M ) .
Proof of Theorem 1 :
Let f ∗ ∈ span ( F 1 , . .
. , F M ) be an arbitrary ﬁxed function.
It is introduced for technical convenienceand can be treated as the function that (approximately) attains the inﬁmum of L ( f ) .
For every f ∈ span ( F 1 , . . .
, F M ) , we deﬁne the following norm with respect to the basis functions,
[Equation]
where (cid:107) µ (cid:107) 1 denotes the abstract sum of its entries, namely (cid:80) m
=1 ,...,M,j =1 ,...,K m | µ m,j | .
For each t , let S t
⊂
∪ Mm =1 F m denote the ﬁnite set of functions such that
1) f τm ∈ S t for all 0 < τ < t , and
2) f ∗ = (cid:80) g ∈ S t µ g f ∗ g ( µ g ∈ R ), with (cid:107) µ f ∗ (cid:107) 1 ≤ (cid:107) f ∗ (cid:107) 1 + ε .
Note that S t exists due to the deﬁnition of (cid:107)·(cid:107) 1 and the construction of each f τm .
Suppose that F t − 1admits the representation F t − 1 = (cid:80) g ∈ S t µ gF t − 1 g .
From (7), we have
[Equation]
(cid:1) ,
[Equation]
[Equation]
, M.
Meanwhile, it follows from (6) that for each m , and each g ∈ S t ∩ F m ,
[Equation]
[Equation] where s g ∆ = sign ( µ gf ∗ − µ gF t − 1 ) .
Combining (9) and (10), we obtain
[Equation]
[Equation]
Applying Taylor expansion to f (cid:55)→ L ( f ) at f = F t − 1 , and invoking (11) and Assumption (A1), wehave
[Equation]
for all sufﬁciently small a t > 0 .
Let (cid:107) µ f ∗ − µ F
[Equation]
[Equation] =
[Equation] − µ gF [Equation] − 1 | .
Multiplying bothsides by | µ gf ∗ − µ gF t − 1 | , and add up all the g ∈ S t , we have
[Equation]
where the last inequality is due to the convexity of L .
If (cid:107) µ f ∗ − µ F t − 1 (cid:107) 1 = 0 , F t − 1 already convergesto f ∗ .
Otherwise, we rearrange (13) to obtain
[Equation]
where the last inequality is due to the triangle inequality, the way F t − 1 is constructed, and the factthat ε can be arbitrarily chosen.
Here, we deﬁned a 0 ∆ = 0 .
Let a 1: t = (cid:80) tτ
=1 a τ for each t ≥ 1 .
Applying (15) and the Lemma 4.2 in [67], we have
[Equation]
Since f ∗ is arbitrarily chosen, it can be seen from Inequality (16) and Assumption (A2)
thatlim t →∞ L ( F t ) = inf f ∗ ∈ span ( [Equation]
[Equation]
[Equation] M ) L ( f ∗ ) , and the rate of convergence is at the order of O ( (cid:80) tτ =1 ( a 1: τ /a 1: t ) a 2 τ ) as t → ∞ .
C Experimental Setup
C.1 Dataset
In Table [Table] we illustrate the statistics of datasets used in our experiments.
In Figure
[Table]
we show how
MNIST and CIFAR10 images are split into 2 , 4 , and 8 image patches.
The left upper image patch(labeled as [1] ) of MNIST image is less informative which demonstrates that an organization withlittle informative data can leverage other organizations’ local data and models.
The central imagepatches (labeled [2 , 3 , 6 , 7] ) of MNIST and CIFAR10 images are more informative than others, whichleads to larger corresponding gradient assistance weights.
[Table]
M = M =
M =
ServiceReceiver Collaborator
M = M =
M =
ServiceReceiver Collaborator    2            2   48 6(a) (b)
The patch
MNIST images is less informative in general.
In contrast, the central image patches (labeled [2 , 3 , 6 , 7] ) of MNIST and CIFAR are more informative.
C.2 Model and hyperparameters
Table 6 summarizes the deep neural network architecture used for the MNIST, CIFAR10, ModelNet40,and ShapeNet55 datasets.
Table 7 shows the hyperparameters used in our experiments.
[Table] [Table]
D Further Experimental ResultsD.1 Model Autonomy
In Tables 8 and
[Table]
we demonstrate the results of our experiments related to model autonomy for
M = 2 and 4 respectively.
Our method signiﬁcantly outperforms the baselines ‘Alone’ and ‘AL.’
Theresults also demonstrate that with GAL, an organization with little informative data and free choiceof its local model (model autonomy) can leverage other organizations’ local data and models andeven achieve near-oracle performance.
[Table] [Table]
D.2 Deep Model Sharing
In Tables 10 and
[Table]
we demonstrate the results of our experiments related to deep model sharingfor M = 2 and 4 respectively.
The results show that sharing the feature extractor across multipleassistance rounds can still outperform the ‘Alone’ case.
Thus, DMS can provide a trade-off betweenpredictive performance and computation space.
represents the results with Deep Model Sharing
[Table] [Table]
D.3 Comparison with AL
In Table
[Table]
we demonstrate the comparison of computation and communication complexity between
GAL and AL.
We compare the computation and communication complexity between AL and GALunder the constraint of the same communication cost as demonstrated.
Because AL sequentiallytrains each organization while GAL allows organizations to train locally in parallel, the computationtime and communication round of AL is M × those of GAL.
The GAL with Deep Model Sharing(GAL DMS ) saves T × computation space by sharing the feature extractor of deep models.
In summary,
GAL generalizes the problem scope, reduces the computation and communication complexity
[Table]
andachieves signiﬁcantly better results.
[Table]
.4 Ablation StudiesD.4.1 Privacy enhancement
Our learning framework does not require the organizations to share local data, model, and objectivefunction.
One potential limitation of our approach is that assisting organizations may infer Alice’sinformation based on shared the pseudo-residuals.
Therefore, we suggest to further enhance theprivacy by adopting the framework of Differential Privacy (DP) [68] or Interval Privacy (IP) [69].
Weuse the Laplace mechanism with α = 1 for DP and set the number of intervals of IP to be 1 .
We adda moderate amount of noise to the pseudo-residuals in hindsight.
In Tables 13-15, we demonstratethat privacy-enhanced GAL can still outperform the ‘Alone’ case.
[Table] [Table]
[Table]
D.4.2 Noisy training with gradient assistance weights
To optimize gradient assistance weights, we use the Adam optimizer and enforce the parameters tosum to one by using the softmax function.
The cost to optimize the gradient assistance weights w andgradient assisted learning rate η is often negligible compared with the cost to ﬁt the pseudo-residualssince the number of parameters involved in w ∈ R M and η ∈ R 1 is small.
In Tables [Table] wedemonstrate the results of our ablation studies of gradient assistance weights by adding noise to thepredicted pseudo-residuals (namely the outputs) from half of the organizations.
[Table]
by adding noises to the predicted outputs from half
[Table] [Table]
D.5 Additional Results
[Figure]
In Figure 7-
[Figure]
we illustrate results of all datasets (maximal M ).
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure] [Figure]