Distributed Distributionally Robust Optimization with Non-Convex Objectives
Anonymous Author(s)
AffiliationAddress email
Abstract
Distributionally Robust Optimization (DRO), which aims to find an optimal decision that minimizes the worst case cost over the ambiguity set of probabilitydistribution, has been widely applied in diverse applications, e.g. , network behavioranalysis, risk management, etc.
However, existing DRO techniques face three keychallenges: 1) how to deal with the asynchronous updating in a distributed environment; 2) how to leverage the prior distribution effectively; 3) how to properlyadjust the degree of robustness according to different scenarios.
To this end, wepropose an asynchronous distributed algorithm, named A synchronous S ingle-loo Palternat I ve g R adient
proj E ction (ASPIRE) algorithm with the it E rative A ctive
SE t method (EASE) to tackle the distributed distributionally robust optimization(DDRO) problem.
Furthermore, a new uncertainty set, i.e. , constrained D -normuncertainty set, is developed to effectively leverage the prior distribution and flexibly control the degree of robustness.
Finally, our theoretical analysis elucidatesthat the proposed algorithm is guaranteed to converge and the iteration complexityis also analyzed.
Extensive empirical studies on real-world datasets demonstratethat the proposed method can not only achieve fast convergence, and remain robustagainst data heterogeneity as well as malicious attacks, but also tradeoff robustnesswith performance.
Introduction
The past decade has witnessed the proliferation of smartphones and Internet of Things (IoT) devices,which generate a plethora of data everyday.
Centralized machine learning requires gathering thedata to a particular server to train models which incurs high communication overhead [49] andsuffers privacy risks [46].
As a remedy, distributed machine learning methods have been proposed.
Considering a distributed system composed of N devices (workers), we denote the dataset ofthese workers as { D 1 , · · · , D N } .
For the j th ( 1 ≤ j ≤ N ) worker, the labeled dataset is given as
D j = { x ij , y ij } , where x ij ∈ R d and y ij ∈ { 1 , · · · , c } denote the i th data sample and the correspondinglabel, respectively.
The distributed learning tasks can be formulated as the following optimizationproblem,
[Equation]
where w ∈ R p is the model parameter to be learned and W ⊆
R p is a nonempty closed convex set,f j ( · ) is the empirical risk over the j th worker involving only the local data:
[Equation]
where L j is the local objective function over the j th worker.
Problem in Eq.
(1) arises in numerousareas, such as distributed signal processing [21], multi-agent optimization [38], etc .
However, such problem does not consider the data heterogeneity [59, 43, 42, 32] among different workers ( i.e. ,data distribution of workers could be substantially different from each other [47]).
Indeed, it hasbeen shown that traditional federated approaches, such as FedAvg [35], built for independent andidentically distributed (IID) data may perform poorly when applied to Non-IID data [29].
This issuecan be mitigated via learning a robust model that aims to achieve uniformly good performance over allworkers by solving the following distributionally robust optimization (DRO) problem in a distributedmanner:
[Equation]
where p =
[ p 1 , · · · , p N ] ∈ R N is the adversarial distribution in N workers, the j th entry in this vector,i.e. , p j represents the adversarial distribution value for the j th worker.
∆ N = { p ∈ R N + : 1 ⊤ p = 1 }and Ω is a subset of ∆ N .
Agnostic federated learning (AFL) [37] firstly introduces the distributionallyrobust (agnostic) loss in federated learning and provides the convergence rate for (strongly) convexfunctions.
However, AFL does not discuss the setting of Ω .
DRFA-Prox [18] considers Ω = ∆ Nand imposes a regularizer on adversarial distribution to leverage the prior distribution.
Nevertheless,three key challenges have not yet been addressed by prior works.
First, whether it is possible toconstruct an uncertainty framework that can not only flexibly maintain the trade-off between themodel robustness and performance but also effectively leverage the prior distribution?
Second, how todesign asynchronous algorithms with guaranteed convergence?
Compared to synchronous algorithms,the master in asynchronous algorithms can update its parameters after receiving updates from onlya small subset of workers [60, 12].
Asynchronous algorithms are particularly desirable in practicesince they can relax strict data dependencies and ensure convergence even in the presence of devicefailures [60].
Finally, whether it is possible to flexibly adjust the degree of robustness?
Moreover, itis necessary to provide convergence guarantee when the objectives ( i.e. , f j ( w j ) , ∀ j ) are non-convex.
To this end, we propose ASPIRE-EASE to effectively address the aforementioned challenges.
Firstly,different from existing works, the prior distribution is incorporated within the constraint in ourformulation, which can not only leverage the prior distribution more effectively but also achieveguaranteed feasibility for any adversarial distribution within the uncertainty set.
The prior distributioncan be obtained from side information or uniform distribution [44], which is necessary to constructthe uncertainty (ambiguity) set and obtain a more robust model [18].
Specifically, we formulate theprior distribution informed distributionally robust optimization (PD-DRO) problem as:
[Equation]
where z ∈ R p is the global consensus variable, w j ∈ R p is the local variable (local model parameter)of
j th worker and Z ⊆
R p is a nonempty closed convex set.
P ⊆ R N + is the uncertainty (ambiguity)set of adversarial distribution p , which is set based on the prior distribution.
To solve the PDDRO problem in an asynchronous manner, we first propose A synchronous S ingle-loo P alternat I veg R adient
proj E ction (ASPIRE), which employs simple gradient projection steps for the update ofprimal and dual variables at every iteration, thus is computationally efficient .
Next, the it E rative
A ctive SE t method (EASE) is employed to replace the traditional cutting plane method to improvethe computational efficiency and speed up the convergence.
We further provide the convergenceguarantee for the proposed algorithm.
Furthermore, a new uncertainty set, i.e. , constrained D -norm( CD -norm), is proposed in this paper and its advantages include: 1) it can flexibly
control the degree
of robustness; 2) the resulting subproblem is computationally simple; 3) it can effectively leveragethe prior distribution and
flexibly set the bounds for every p j .
Contributions.
Our contributions can be summarized as follows:
We formulate a PD-DRO problem with CD -norm uncertainty set.
PD-DRO incorporates the priordistribution as constraints which can leverage prior distribution more effectively and guarantee robustness.
In addition, CD -norm is developed to model the ambiguity set around the prior distributionand it provides a flexible way to control the trade-off between model robustness and performance.2.
We develop a single-loop asynchronous algorithm, namely ASPIRE-EASE, to optimize PD-DROin an asynchronous manner.
ASPIRE employs simple gradient projection steps to update the variablesat every iteration, which is computationally efficient.
And EASE is proposed to replace cutting plane
method to enhance the computational efficiency and speed up the convergence.
We demonstrate thateven if the objectives
f j ( w j ) , ∀ j are non-convex, the proposed algorithm is guaranteed to converge.
We also theoretically derive the iteration complexity of ASPIRE-EASE.3.
Extensive empirical studies on four different real world datasets demonstrate the superior performance of the proposed algorithm.
It is seen that ASPIRE-EASE can not only ensure the model’srobustness against data heterogeneity but also mitigate malicious attacks.
Preliminaries
2.1 Distributionally Robust Optimization
Optimization problems often contain uncertain parameters.
A small perturbation of the parameterscould render the optimal solution of the original optimization problem infeasible or completelymeaningless [6].
Distributionally robust optimization (DRO) [30, 19, 8] assumes that the probabilitydistributions of uncertain parameters are unknown but remain in an ambiguity (uncertainty) set andaims to find a decision that minimizes the worst case expected cost over the ambiguity set, whosegeneral form can be expressed as,
[Equation]
(5) where x ∈ X represents the decision variable, P is the ambiguity set of probability distributions
P of uncertain parameters ξ .
Existing methods for solving DRO can be broadly grouped into twowidely-used categories [45]: 1) Dual methods [17, 53, 20] reformulate the primal DRO problemsas deterministic optimization problems through duality theory.
Ben-Tal et al.
[2] reformulate therobust linear optimization (RLO) problem with an ellipsoidal uncertainty set as a second-order coneoptimization problem (SOCP).
Cutting plane methods [36, 7] (also called adversarial approaches[23]) continuously solve an approximate problem with a finite number of constraints of the primal
DRO problem, and subsequently check whether new constraints are needed to refine the feasible set.
Recently, several new methods [44, 31, 25] have been developed to solve DRO, which need to solvethe inner maximization problem at every iteration.
2.2 Cutting Plane Method for PD-DRO
In this section, we introduce the cutting plane method for PD-DRO in Eq.
We first reformulate
PD-DRO by introducing an additional variable h ∈ H ( H ⊆ R 1 is a nonempty closed convex set)
andprotection function g ( { w j } )
Introducing additional variable h is an epigraph reformulation[4, 58].
In this case, Eq.
(4) can be reformulated as the form with uncertainty in the constraints:
[Equation]
where p is the nominal value of the adversarial distribution for every worker and g ( { w j } )
=
max p ∈ P (cid:80) j ( p j − p )
f j ( w j ) is the protection function.
Eq.
(6) is a semi-infinite program (SIP) whichcontains infinite constraints and cannot be solved directly [45].
Denoting the set of cutting planeparameters in ( t +1) th iteration as A t ⊆ R N , the following function is used to approximate g ( { w j } ) :
[Equation]
[Equation]
where a l = [ a l, 1 , · · · , a l,N ] ∈ R N denotes the parameters of l th cutting plane in A t and f ( w ) =[ f 1 ( w 1 ) , · · · , f N ( w N )]
∈ R N .
Substituting the protection function g ( { w j } ) with g ( { w j } ) , we canobtain the following approximate problem:
[Equation]
variables z
[Equation]
ASPIRE
Distributed optimization is an attractive approach for large-scale learning tasks [9, 10] since it doesnot require data aggregation, which protects data privacy while also reducing bandwidth requirements[48].
When the neural network models ( i.e. , f j ( w j ) , ∀ j are non-convex functions) are used, solving
Eq.
(8) in a distributed manner facing two challenges.
1) Computing the optimal solution to anon-convex subproblem requires a large number of iterations and therefore is highly computationallyintensive if not impossible.
Thus, the traditional Alternating Direction Method of Multipliers(ADMM) is ineffective.
2) the communication delays of workers may differ significantly [13], thus,asynchronous algorithms are strongly preferred.
To this end, we propose the A synchronous S ingle-loo P alternat I ve g R adient
proj
E ction (ASPIRE).
The advantages of the proposed algorithm include: 1) ASPIRE uses simple gradient projection stepsto update primal variables in each iteration and therefore it is computationally more efficient thanthe traditional ADMM method, which seeks to find the optimal solution in non-convex (for w j , ∀ j )and convex (for z and h ) optimization subproblems every iteration, 2) the proposed asynchronousalgorithm does not need strict synchronization among different workers.
Therefore, ASPIRE remainsresilient against communication delays and potential hardware failures from workers.
Details of thealgorithm are given below.
Firstly, we define the node as master which is responsible for updatingthe global variable z , and we define the node which is responsible for updating the local variablew j as worker j .
In each iteration, the master updates its variables once it receives updates fromat least S workers, e.g. , active workers, satisfying 1 ≤ S ≤ N .
Q t +1 denotes the index subset ofworkers from which the master receives updates during ( t +1) th iteration.
We also assume the masterwill receive updated variables from every worker at least once for each τ iterations.
Then a batchof instances with size m are randomly sampled from each worker during each iteration.
The lossfunction of these instances from j th worker is given by ˆ f j ( w j )
= m (cid:80) i = 1 m L j ( x ij , y ij ; w j ) .
The use ofmini-batch loss avoids enumerating the whole dataset, which is more efficient [44].
It is evident that
E [ ˆ f j ( w j )]=
f j ( w j ) and E [ ∇ ˆ f j ( w j )]= ∇ f j ( w j ) .
Thus, the augmented Lagrangian function of Eq.(8) can be written as:
[Equation]
where L p =
L p ( { w j } , z , h, { λ l } , { ϕ j } ) , λ l ∈ Λ , ∀ l and ϕ j ∈ Φ , ∀ j represent the dual variables ofinequality and equality constraints in Eq.
(8), respectively.
Λ ⊆ R 1 and Φ ⊆
R p are nonemptyclosed convex sets, constant κ 1 > 0 is a penalty parameter.
Note that Eq.
(9) does not consider thesecond-order penalty term for inequality constraint since it will invalidate the distributed optimization.
Following [55], the regularized version of Eq.
(9) is employed to update all variables as follows,
[Equation]
[Equation]
where c t 1 and c t 2 denote the regularization terms in ( t +1) th iteration.
In ( t +1) th iteration, theproposed algorithm proceeds as follows.
1) Each worker updates the local variables w j as follows,
[Equation]
11) where η w represents the step-size and P W represents the projection onto the closed convex set Wand we set W = { w j |
||
w j || ∞ ≤ α 1 } , α 1 is a positive constant.
And (cid:101) t j is the last iteration duringwhich worker j was active.
It is seen that w tj = w (cid:101) t j j and ϕ tj = ϕ (cid:101) t
j j , ∀ j ∈ Q t +1 .
Then, the activeworkers transmit their local model parameters w t +1 j and loss ˆ f j ( w j ) to the master.2) After receiving the updates from active workers, the master updates the global consensus variablez , additional variable h and dual variables
λ l as follows,
[Equation]
[Equation] [Equation]
[Equation]
where η z , η h and ρ 1 represent the step-size.
P Z , P H and P Λ respectively represent the projectiononto the closed convex sets Z , H and Φ .
We set Z =
{ z | ||
z || ∞ ≤ α 1 } , H = { h | 0 ≤ h ≤ α 2 }and Λ = { λ l | 0 ≤ λ l ≤ α 3 } , where α 2 and α 3 are positive constants.
Then, master broadcasts z t +1 ,h t +1 , λ t +1 l , ∀ 1 ≤ l ≤|
A t | to the active workers.3)
Each worker updates the dual variables ϕ j as follows,
[Equation]
where ρ 2 represents the step-size and P Φ represents the projection onto the closed convex set Φ andwe set Φ = { ϕ j | || ϕ j || ∞ ≤ α 4 } , α 4 is a positive constant.
It is seen that the projection operation ineach step is computationally simple since the closed convex sets have simple structures [5].
Iterative Active Set Method
Cutting plane methods may give rise to numerous linear constraints and lots of extra message passing[57].
Moreover, more iterations are required to obtain the ε -stationary point when the size of a setcontaining cutting planes increases (which corresponds to a larger M ), which can be seen in Theorem1.
To improve the computational efficiency and speed up the convergence, we consider removingthe inactive cutting planes.
The proposed it E rative
A ctive SE t method (EASE) can be divided intothe two steps: 1) solving the cutting plane generation subproblem to generate cutting plane, and 2)removing the inactive cutting plane every k iterations, where k> 0 is a pre-set constant.
The settingof k can be controlled flexibly and we can set k as a relatively large number.
The cutting planes are generated according to the uncertainty set.
For example, if we employ ellipsoiduncertainty set, the cutting plane is generated via solving a SOCP.
In this paper, we propose CD -normuncertainty set, which can be expressed as follows,
[Equation]
where Γ ∈ R 1 can flexibly control the level of robustness, q =
[ q 1 , . . . , q N ] ∈ R N represents the priordistribution, − (cid:101) p j and (cid:101)
p j ( (cid:101) p
j ≥ 0 ) represent the lower and upper bounds for p
j − q j , respectively.
The setting of q and (cid:101) p j , ∀ j are based on the prior knowledge.
D -norm is a classical uncertainty set(which is also called as budget uncertainty set)
We call Eq.
(16) CD -norm uncertainty set sincep is a probability vector so all the entries of this vector are non-negative and add up to exactly one,i.e. , 1 ⊤ p = 1 .
Due to the special structure of CD -norm, the cutting plane generation subproblemis easy to solve and level of robustness in terms of the outage probability, i.e. , probabilistic boundsof the violations of constraints can be flexibly adjusted via a single parameter Γ .
We claim that
L 1 -norm (or twice total variation distance) uncertainty set is closely related to CD -norm uncertaintyset.
Nevertheless, there are two differences: 1) CD -norm uncertainty set could be regarded as aweighted L 1 -norm with additional constraints.
2) CD -norm uncertainty set can flexibly set the lowerand upper bounds for every p j ( i.e. , q j − (cid:101)
p j ≤ p j ≤ p j + (cid:101) p j ), while 0 ≤ p j ≤ 1 , ∀ j in L 1 -norm uncertaintyset.
Based on the CD -norm uncertainty set, the cutting plane can be derived as follows,1)
Solve the following problem,
[Equation]
17) variable
[Equation]
where p t +1 =
[ p t +11 , · · · , p t +1 N ] ∈ R N .
We denote (cid:101) a t +1 =
p t +1 − p , where p =
[ p, · · · , p ] ∈ R N .
This first step aims to obtain the distribution (cid:101) a t +1 by solving problem in Eq.
This problemcan be effectively solved through combining merge sort [15] (for sorting (cid:101) p
j ˆ f j ( w j ) , j = 1 , . .
. , N )with few basic arithmetic operations (for obtaining p t +1 j , j =1 , . .
. , N ).
Since N is relatively large in
Algorithm
Initialization: iteration t = 0 , variables { w 0 j } , z 0 , h 0 , λ 01 , { ϕ 0 j } and set A 0 .
repeatfor each worker do updates w t +1 j according to Eq.
(11); end for active workers transmit variables and loss to master ; master receives updates from active workers do updates z t +1 , h t +1 and λ t +1 l according to Eq.
(12), (13) and (14); master broadcasts variables to active workers ; for each worker do updates ϕ t +1 j according to Eq.
(15); end for if t mod k == 0 then master updates A t
+1 according to Eq.
(19) and (20), then broadcasting A t +1 to all workers; end if t = t + 1 ; until convergence distributed system, the arithmetic complexity of solving problem in Eq.
(17) is dominated by mergesort, which can be regarded as O ( N log( N )) .2)
Let ˆ f ( w )=[
ˆ f 1 ( w 1 ) , · · · , ˆ f N ( w N )] ∈ R N , check the feasibility of the following constraints:
[Equation]
If Eq.
(18) is violated, (cid:101) a t +1 will be added into A t
[Equation] [Equation]
18) is
violated
[Equation]
when a new cutting plane is added, its corresponding dual variable λ t +1 |
A t | +1 will be generated.
Afterthe cutting plane subproblem is solved, the inactive cutting plane will be removed, that is:
[Equation]
where ∁ A t { a l } is the complement of { a l } in A t .
Then the master broadcasts A t +1 to all workers.
Details of the algorithm are summarized in Algorithm 1.
Convergence Analysis
Definition 1 (Stationarity gap)
Following [55, 34, 56], the stationarity gap of our problem at t thiteration is defined as:
[Equation]
where ∇ G t is the simplified form of ∇ G ( { w tj } , z t , h t , { λ tl } , { ϕ tj } ) .
Definition 2 ( ε -stationary point) ( { w t j } , z t , h t , { λ tl } , { ϕ tj } ) is an ε -stationary point ( ε ≥ 0 ) of adifferentiable function L p , if ||∇ G t || ≤ ε .
T ( ε ) is the first iteration index such that ||∇ G t ||≤ ε , i.e.,
T ( ε )=
min { t | ||∇ G t ||≤ ε } .
Assumption 1 (Smoothness/Gradient Lipschitz)
L p has Lipschitz continuous gradients.
We assumethat there exists L > 0 satisfying
[Equation]
where θ ∈ {{ w j } , z , h, { λ l } , { ϕ j }} and [; ] represents the concatenation.
w cat − ˆ w cat =
[ w 1 −ˆ
[Equation]
Assumption 2 (Bounded change of variables)
The change of the variables in master is upperbounded within τ iterations:
[Equation]
where k 1 > 0 is a constant and k z
= ||
z
[Equation]
and
[Equation]
− λ
[Equation]
|
[Equation]
Setting 1 (Bounded | A t | )
| A t | ≤ M, ∀ t , i.e., an upper bound is set for the number of cutting planes.
Setting 2 (Setting of c t 1 , c t 2 )
c t 1 = ρ 1 ( t +1) 16 ≥ c 1 and c t 2 =
[Equation]
( t +1) 16 ≥ c 2 are nonnegative nonincreasing sequences, where c 1 and c 2 are positive constants and meet Mc 12 + Nc 22 ≤ ε 2 4 .
Theorem 1 (Iteration complexity) Suppose the Assumption 1 and 2 hold, if we set the step-sizesof primal variables as η w = η z = η h =
[Equation]
L + ρ 1 | A t | L 2 + ρ 2 NL 2 +8( | A
[Equation]
| γL 2 ρ 1( ct
[Equation]
2 + NγL 2 ρ
[Equation]
And we set thestep-sizes of dual variables as ρ 1 ≤ min { 2 L +2 c 01 , 1 5 τk 1 NL 2 } and ρ 2 ≤ 2 L +2 c 02 , respectively.
For agiven ε , we have:
[Equation]
where σ 1 , σ 2 , γ , k d , − d and d 5 are constants.
The detailed proof is given in Appendix A.
There exists a wide array of works regarding the convergence analysis of various algorithms fornonconvex/convex optimization problems involved in machine learning [27, 56].
Our analysis,however, differs from existing works in two aspects.
First, we solve the non-convex PD-DRO in anasynchronous distributed manner .
To our best knowledge, there are few works focusing on solvingthe DRO in a distributed manner.
Compared to solving the non-convex PD-DRO in a centralizedmanner, solving it in an asynchronous distributed manner poses significant challenges in algorithmdesign and convergence analysis.
Secondly, we do not assume the inner problem can be solved nearlyoptimally for each outer iteration, which is numerically difficult to achieve in practice [5].
Instead,
ASPIRE-EASE is single loop and involves a simple gradient projection operation at each step.
Experiment
In this section, we conduct experiments on four real-world datasets to assess the performance of theproposed method.
Specifically, we evaluate the robustness against data heterogeneity, robustnessagainst malicious attacks and efficiency of the proposed method.
Ablation study is also carried out todemonstrate the excellent performance of ASPIRE-EASE.
6.1 Datasets and Baseline Methods
We compare our proposed ASPIRE-EASE with baseline methods based on SHL [22], Person Activity [28], Single Chest-Mounted Accelerometer (SM-AC)
[11] and Fashion MNIST [54] datasets.
The baseline methods include Ind j (learning the model from an individual worker j ), Mix Even(learning the model from all workers with even weights using ASPIRE), FedAvg [35], AFL [37] and
DRFA-Prox [18].
The detailed descriptions of datasets and baselines are given in Appendix C.
In our empirical studies, since the downstream tasks are multi-class classification, the cross entropyloss is used on each worker ( i.e. , L j ( · ) , ∀ j ).
For SHL, Person Activity, and SM-AC, we adopt thedeep multilayer perceptron [52] as the base model.
And we use the same logistic regression modelas in [37, 18] for Fashion MNIST dataset.
The base models are trained with SGD.
More details aregiven in Appendix C.
Following related works in this direction [44, 37, 18], worst case performanceare reported for the comparison of robustness.
Specifically, we use Acc w and Loss w to represent theworst case test accuracy and training loss ( i.e. , the test accuracy and training loss on the worker withworst performance), respectively.
We also report the standard deviation Std of [Acc 1 , · · · , Acc N ](the test accuracy on every worker).
In the experiment, S is set as 1, that means the master will make
[Table]
an update once it receives a message.
Each experiment is repeated 10 times, both mean and standarddeviations are reported.
We implement our model with PyTorch and conduct all the experiments on aserver with two TITAN V GPUs.
6.2 Results
Robustness against Data Heterogeneity .
We first assess the robustness of the proposed ASPIREEASE by comparing it with baseline methods when data are heterogeneously distributed acrossdifferent workers.
Specifically, we compare the Acc w , Loss w and Std of different methods on alldatasets.
The performance comparison results are shown in Table 1.
In this table, we can observe thatmax { Ind j } , which represents the best performance of individual training over all workers, exhibitsthe worst robustness on SHL, Person Activity, and SC-MA.
This is because individual training( max { Ind j } ) only learns from the data in its local worker and cannot generalize to other workers dueto different data distributions.
Note that max { Ind j } is unavailable for Fashion MNIST since eachworker only contains one class of data and cross-entropy loss cannot be used in this case.
max { Ind j }also does not have Loss w , since Ind j is trained only on individual worker j .
The FedAvg and
Mix Even exhibit better performance than max { Ind j } since they consider the data from all workers.
Nevertheless, FedAvg and Mix Even only assign the fixed weight for each worker.
AFL is more robustthan FedAvg and Mix Even since it not only utilizes the data from all workers but also considersoptimizing the weight of each worker.
DRFA-Prox outperforms AFL since it also considers the priordistribution and regards it as a regularizer in the objective function.
Finally, we can observe thatthe proposed ASPIRE-EASE shows excellent robustness, which can be attributed to two factors:
1) ASPIRE-EASE considers data from all workers and can optimize the weight of each worker; 2)
compared with DRFA-Prox which uses prior distribution as a regularizer, the prior distribution is
incorporated within the constraint in our formulation (Eq.
, which can be leveraged more effectively.
And it is seen that ASPIRE-EASE can perform periodic communication since ASPIRE-EASE per ,which represents ASPIRE-EASE with periodic communication, also has excellent performance.
Within ASPIRE-EASE, the level of robustness can be controlled by adjusting Γ .
Specially, whenΓ = 0 , we obtain a nominal optimization problem in which no adversarial distribution is considered.
The size of the uncertainty set will increase with Γ (when Γ ≤ N ), which enhances the adversarialrobustness of the model.
As shown in Figure 1, the robustness of ASPIRE-EASE can be graduallyenhanced when Γ increases.
More results are available in Figure C2 of Appendix C.
Robustness against Malicious Attacks .
To assess the model robustness against malicious attacks,malicious workers with backdoor attacks [1, 51], which attempt to mislead the model training process,are added to the distributed system.
Following [16], we report the success attack rate of backdoorattacks for comparison.
It can be calculated by checking how many instances in the backdoordataset can be misled and categorized into the target labels.
Lower success attack rates indicatemore robustness against backdoor attacks.
The comparison results are summarized in Table 2 andmore detailed settings of backdoor attacks are available in Appendix C.
In Table 2, we observe that
AFL can be attacked easily since it could assign higher weights to malicious workers.
Comparedto AFL, FedAvg and Mix Even achieve relatively lower success attack rates since they assign equalweights to the malicious workers and other workers.
DRFA-Prox can achieve even lower successattack rates since it can leverage the prior distribution to assign lower weights for malicious workers.
The proposed ASPIRE-EASE achieves the lowest success attack rates since it can leverage the priordistribution more effectively.
Specifically, it will assign lower weights to malicious workers withtight theoretical guarantees.
[Figure] [Figure]
[Figure]
tt
[Figure]
g
[Figure]
l
[Figure]
[Table]
Efficiency .
In Figure 2, we compare the convergence speed of the proposed ASPIRE-EASEwith AFL and DRFA-Prox by considering different communication and computation delays foreach worker.
The proposed ASPIRE-EASE has two variants, ASPIRE-CP (ASPIRE with cuttingplane method), ASPIRE-EASE(-)(ASPIRE-EASE without asynchronous setting).
More results areavailable in Figure C3 of Appendix C. Based on the comparison, we can observe that the proposed
ASPIRE-EASE generally converges faster than baseline methods and its two variants.
This is because1) compared with AFL, DRFA-Prox, and ASPIRE-EASE(-), ASPIRE-EASE is an asynchronousalgorithm in which the master updates its parameters only after receiving the updates from activeworkers instead of all workers; 2) unlike DRFA-Prox, the master in ASPIRE-EASE only needs tocommunicate with active workers once per iteration; 3) compared with ASPIRE-CP, ASPIRE-EASEutilizes active set method instead of cutting plane method, which is more efficient.
It is seen from
Figure 2 that, the convergence speed of ASPIRE-EASE mainly benefits from the asynchronoussetting.
Ablation Study .
For ASPIRE, compared with cutting plane method, EASE is more efficient since itconsiders removing the inactive cutting planes.
To demonstrate the efficiency of EASE, we firstlycompare ASPIRE-EASE with ASPIRE-CP concerning the number of cutting planes used during thetraining.
In Figure 3, we can observe that ASPIRE-EASE uses fewer cutting planes than ASPIRE-CP,thus is more efficient.
The convergence speed of ASPIRE-EASE and ASPIRE-CP in Figure 2 alsosuggests that ASPIRE-EASE converges much faster than ASPIRE-CP.
More results are available in
Figure C3 and C4, Appendix C.
Conclusion
In this paper, we present ASPIRE-EASE method to effectively solve the distributed distributionallyrobust optimization problem with non-convex objectives.
In addition, CD -norm uncertainty set hasbeen proposed to effectively incorporate the prior distribution into the problem formulation, whichallows for flexible adjustment of the degree of robustness of DRO.
Theoretical analysis has also beenconducted to analyze the convergence properties and the iteration complexity of ASPIRE-EASE.
ASPIRE-EASE exhibits strong empirical performance on multiple real-world datasets and is effectivein tackling DRO problems in a fully distributed and asynchronous manner.
In the future work, moreuncertainty sets could be designed for our framework and more update rule for variables in ASPIREcould be considered.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] There isno potential negative societal impact of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 5.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A andB.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The references
of the data used in this paper
are added in Section 6.1 and the code is proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Section C.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 6.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 6.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]