A Stochastic Linearized Augmented Lagrangian Method for Decentralized Bilevel Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
Bilevel optimization has been shown to be a powerful framework for formulatingmulti-task machine learning problems, e.g., reinforcement learning (RL) and metalearning, where the decision variables are coupled in both levels of the minimizationproblems.
In practice, the learning tasks would be located at different computingresource environments, and thus there is a need for deploying a decentralizedtraining framework to implement multi-agent and multi-task learning.
We developa stochastic linearized augmented Lagrangian method (SLAM) for solving generalnonconvex bilevel optimization problems over a graph, where both upper and loweroptimization variables are able to achieve a consensus.
We also establish that thetheoretical convergence rate of the proposed SLAM to the Karush-Kuhn-Tucker(KKT) points of this class of problems is on the same order as the one achieved bythe classical distributed stochastic gradient descent for only single-level nonconvexminimization problems.
Numerical results tested on multi-agent RL problemsshowcase the superiority of SLAM compared with the benchmarks.
Introduction
In this paper we consider the following general decentralized bilevel optimization (DBO) frameworkwith applications to machine learning problems.
Suppose that there are n nodes over a connectedgraph G = {E
V} where E and V represent the edges and vertices.
Let N i denote the set ofneighboring nodes for node i .
Then the goal of DBO is to have these nodes jointly minimize twolevels of optimization problems.
More formally, DBO is expressed as
[Equation]
1a)
[Equation]
1b)
[Equation]
n (cid:88) i
= g i,k ( x i , y
[Equation]
,k
[Equation]
s.t.
y
[Equation] [Equation]
= y
[Equation] [Equation]
[Equation]
∈ N
[Equation]
, ∀ k ∈ [ m ] , (1c) where vector x i is the upper level (UL) optimization variable at each node i , vector y i,k denotesthe lower level (LL) decision variable for the k th learning task at node i , f
i (; ) is a (smooth) ULloss function and possibly nonconvex with respect to (w.r.t.)
both the UL and LL variables, g i,k ( , )denotes the LL objective function of the k th task at node i , m represents the total number of LLoptimization problems, the consensus constraints x i =
y j , y i,k = y j,k , j ∈ N i , ∀ i ∈ [ n ] , ∀ k ∈ [ m ] ,enforce the model agreements at each level of the problems and for each LL learning task, and y ∗ k =[
y ∗ 1 ,k , . .
, y ∗ n,k ] T is the optimal solutions of the k th LL problem under the consensus constraints.
Applications of Bilevel Optimization.
Many machine learning problems can be formulated mathematically as a form of bilevel optimization or, more precisely, a special case of problem (1), e.g., meta-learning or meta reinforcement learning (RL), actor-critic (AC) schemes in RL, hyperparameteroptimization (HPO), and so on.
Classical bilevel optimization is referred to as the case where there is no consensus constraint butwith only two levels of the minimization subproblems, i.e., min x f ( x , y ∗ ( x )) , s.t.
y ∗ ( x ) =
arg min y g ( x , y ) , which is also known as Stackelberg games [1] with the UL decision variable asthe leader and the LL decision variable as the follower.
It turns out that this class of optimizationproblems is useful in formulating a wide range of hierarchical or nested structured machine learningproblems.
For example, one of the most popular domain adaption learning models, model-agnosticmeta-learning (MAML) [2, 3], can be written as a special case of bilevel programming [4], wherethe UL model provides a good initialization for accelerating learning procedures by implementingthe LL algorithms.
The idea behind the model design is that the UL model is considered as the metalearner that searches for a permutation-invariant subspace over multiple task-specific learners at the
LL so that the performance of the MAML model can be generalized well for unseen or testing datasamples.
The theoretical analysis of the generalization performance of this class of bilevel problemshas shown that MAML can indeed decrease the generalization error as the number of tasks increases,at least for strongly convex loss functions [5].
Subsequently, a thorough ablation study from thelatent representation perspective shows that feature reuse is the actual dominant factor in improvingthe generalization performance of MAML [6], and the authors propose a neural network-orientedalgorithm with almost no inner loop (ANIL) that splits the neural network parameters into twoparts corresponding to the UL and LL optimization problems, respectively.
Extensive numericalexperiments illustrate that ANIL achieves almost the same accuracy as the classical MAML butwith significant computational savings.
This example further strengthens the necessity of variablesplitting in the learning structure by optimizing two levels of objective functions to enhance thegeneralization performance.
Beyond the traditional supervised meta-learning scenarios, MAML hasalso been applied to increasing the generalization ability of agents in RL problems by replacing the(stochastic) gradient with the (natural) policy gradient (PG) [3] under the same two-level structure.
Besides meta-learning problems, AC structure in RL is another class of common learning frameworksthat can be formulated by a bilevel optimization problem in nature [7, 8, 9], where the actor step atthe UL aims at optimizing the policy while the critic step at the LL is responsible for value functionevaluation.
In addition, as the expressiveness of neural networks increased sharply over the pastdecades, the reuse of large models with adaptation to multi-task learning problems presents promisingsolutions by leveraging the pre-train and fine-tune strategy, such as in applications of HPO [10, 11]where the hyperparameters are trained at the UL problem so that the downstream learning tasks arelearned with low costs including the expense of both computation and memory.
Applications of Multi-agent Settings.
When multiple computational resources are available andconnected, it is well motivated that exploring them solves distributed large-scale problems witha reduced amount of training time or performs multi-task learning.
The bilevel structure of themeta-learning (ML) is a good fit in this scenario as either UL/LL or both levels may need to accessthe networked data samples rather than local ones.
For example, a federated learning setting of
MAML [12] and bilevel optimization [13] have been built up over multiple nodes recently, where themeta/UL learner finds an initial shared model while the local/LL learners leverage it for adapting datadistributions of individual users.
In such a way, the federated MAML model can realize personalizedlearning without sharing heterogeneous data over numerous clients.
Once there is no central controllerfor coordinating the model aggregation, a diffusion-based MAML (Dif-MAML) [14] is proposedby spreading the model parameters over a network, where the UL parameter is updated by one stepof stochastic gradient descent (SGD) based on a combination of the parameters of neighbors as theinitialization for local model updates.
Decentralized hierarchical structured learning is even more stringent in the multi-agent RL (MARL)setting [15] as the learning tasks are essentially located at scattered sensors and/or controllers.
Under this setting, MARL problem becomes a multi-objective optimization problem under provided(approximate) value functions, where the policy of each agent needs to be learned locally by certainefficient iterative methods, such as multi-agent deep deterministic policy gradient (MADDPG)
[16],trust region methods [17], optimal baseline based variance reduced policy gradient [18], and/orimproved by more advanced techniques, e.g., constrained policy optimization [19] and large sequencemodels [20].
In such a way, the total reward can be maximized over the distributed agents throughoptimizing the networked policy.
In a fully collaborative setting, the team-based value functionis even required to be shared over all the agents such that each agent is able to improve its policy based on the estimated total reward.
For example, the decentralized AC (DAC) scheme has beeninvestigated widely [15, 21, 22], where each agent uses the actor step to optimize its policy while thecritic step performs one step [23] or multiple steps of temporal difference learning with mini-batchsampling (MDAC)
[22, 24] and communications so that the team-based reward over the network isobtained by each agent.
It turns out that DAC can be formulated as a special case of problem (1)as there is no consensus at the UL.
Recently, it has been revealed that if there exists homogeneityof the state and action spaces, decentralized policy consensus (or a partial policy parameter sharingstrategy) provides significant merits to the centralized training and decentralized execution paradigmin terms of learning scalability and efficiency[23, 25], which motivates the consensus process at both
UL and LL DBO problems.
Related Theoretical Works.
Given the fruitful results across these many applications, the corresponding theoretical analysis has been developing very fast as well for variants of bilevel optimizationproblems.
For example, the convergence behaviors of classical inexact MAML (iMAML) methodshave been quantified for both convex [26, 27] and nonconvex [28] cases of the UL loss function,where the LL algorithm only performs one step of stochastic gradient descent (SGD) based on the LLobjective functions as the adaptation step.
Moreover, the iteration complexity of ANIL with multipleiterations for minimizing the LL problems have been studied in [29], which justifies the significantcomputational advantages of ANIL compared with MAML in theory.
Furthermore, the finite-timeanalysis of AC algorithms has shown [30] that, once the learning rates at both the actor and critic sidesare chosen properly, a two timescale AC algorithm can achieve an O ( ϵ − 2 . 5 ) iteration complexity forfinding the first-order stationary points (FOSPs) of general nonconvex reward functions.
Besides these theoretical analyses in a specific learning setting, the algorithm design and corresponding convergence analysis for general bilevel optimization solvers have been recently advancing at arapid pace under certain assumptions that the UL objective function is general nonconvex while the LLobjective functions are strongly convex, which covers the existing convergence results shown for ACalgorithms.
The typical algorithms include those with double-loop structure, those with two timescaleor single timescale but single-loop, and those with error-correction or accelerated/variance-reduction.
To be more specific, double-loop algorithms, such as bilevel stochastic approximation (BSA) methods[31] and stochastic bilevel optimizers (stoBiO) [32], mainly request an inner loop to solve the LLproblem up to a certain error tolerance or with a certain number of iterations and then switch backto optimize the UL problem, which can achieve an O ( ϵ − 2 ) convergence rate to the ϵ -FOSPs.
Inpractice, single-loop algorithms are implemented more efficiently in terms of computational complexity and hyperparameter tuning compared to double-loop algorithms.
A two-timescale stochasticapproximation (TTSA) was analyzed in [33], but it is shown that TTSA needs O ( ϵ − 2 . 5 ) number ofiterations to achieve the ϵ -FOSPs.
Later, an error correction method, named the Single-TimescalestochAstic BiLevEl optimization (STABLE) method [34], improves the convergence rate of thesingle-loop algorithm to O
( ϵ − 2 ) and a tighter analysis for ALternating Stochastic gradient dEscenT(ALSET) shows that the single-loop algorithm can also achieve a convergence of O ( ϵ − 2 ) without theerror correction technique.
When more advanced momentum-assisted or variance reduction methodsare adopted in the algorithm design, the subsequent works, such as the momentum-based recursivebilevel optimizer (MRBO) [35] and the single-timescale double-momentum stochastic approximation(SUSTAIN) [36] and the variance reduced BiAdam (VR-BiAdam) [37], can sharpen the convergencerate of bilevel algorithms to O
( ϵ − 1 . 5 ) .
For the theoretical works on MAML/MARL, it is shown in [22, 24] that when the critic side isallowed the consensus step at each agent to approximate the networked rewards, MDAC algorithmscan achieve an O ( ϵ − 2 ) convergence rate to FOSPs, but both of them require an inner loop procedurefor the LL problem which makes the algorithms double loop.
Dif-MAML [14] is able to performthe UL consensus-based meta learning, but iMAML considered in Dif-MAML is only a very specialcase of bilevel.
Thus, the applicability of Dif-MAML is restrictive.
One of the closest works to oursis coordinated AC (CAC) [23], which can realize the consensus on both UL and LL problems with
O ( ϵ − 2 . 5 ) number of iterations and is only for DAC problems.
A theoretical comparison between ourwork and closely related previous works on bilevel programming is shown in Table 1.
There is aline of independent work on decentralized optimization.
But the existing works are only suitable forsingle-level minimization of only nonconvex problems, such as distributed SGD [38, 39], stochasticgradient tracking [40, 41] and stochastic primal dual algorithm [42], which can achieve an O (1 / ( nϵ 2 ))convergence rate to FOSPs for general nonconvex objective function optimization problems.
[Table] [Table]
Main Contributions of This Work.
In this work
[Table]
we consider a very general DBO setting, whereboth UL and LL problems can include a consensus constraint for model parameter sharing and therewould be multiple LL problems coupled with the UL problem.
To solve this problem efficiently in afully decentralized way, we propose a S tochastic L inearized A ugmented Lagrangian M ethod (SLAM)for dealing with both of the two levels of the optimization processes and the consensus constraints ateach level.
Leveraging the linearized augmented Lagrangian function as a surrogate, the design of
SLAM is simple and easily implemented as it is a single-loop algorithm with only step sizes to betuned for convergence.
We make the standard assumptions on Lipschitz continuity and convexityfor both the UL and LL optimization problems as shown in the existing literature.
We establish theconditions of SLAM w.r.t.
convergence to ϵ -Karush-Kuhn-Tucker (KKT) points of problem (1) at arate of O (1 / ( nϵ 2 )) , matching the standard convergence rate achieved by decentralized SGD type ofalgorithm to FOSPs for only single-level nonconvex minimization problems.
Remarkably, throughnumerical experiments on MARL problems, it is observed that SLAM can converge faster than theexisting MARL methods and even achieve higher rewards in some cases.
To summarize, the main contributions of this work are highlighted as follows:
▶ Our proposed SLAM algorithm is generic, and thus generalizes the single agent-based bilevel
algorithms to the multi-agent setting and is amnable to be specialized to solve multiple consensusbased DBO problems.
160 ▶ SLAM is a single-timescale and single-loop algorithm that can find the ϵ -KKT points at a rateof O (1 / ( nϵ 2 )) , which shows a linear speedup w.r
t. the number of nodes.
To the best of ourknowledge, this is the first work that shows a decentralized stochastic algorithm can achievethis rate under the constraints where any level or both levels of the DBO problem requires the
consensus process.
165 ▶ Numerical results that illustrate the proposed SLAM outperforms the state-of-the-art MARLalgorithms over heterogeneous networks in terms of both convergence speed and achievablerewards.
Due to space limitations, all technical proofs are deferred to the supplement.
Decentralized Bilevel Optimization Framework
Problem formulation of DBO.
One of the main motivations for performing decentralized jointlearning is dealing with large-scale dataset or scattered data samples.
At each node, the loss function can be written as f i ( x i , y ∗
i, 1 ( x i ) , . . .
, y ∗ i,m ( x i ))
≜
E ξ ∈D U i [ F i ( x i , y ∗
i, 1 ( x i ) , . . .
, y ∗ i,m ( x i ); ξ )] ,where D U i denotes the local data distributions at the UL optimization problem, and
F i ( x i , y ∗ i, 1 ( x i ) , . . .
, y ∗ i,m ( x i ); ξ ) represents the estimation error of the UL learning model
ondata ξ ∈ D U i .
Similarly, the LL learning tasks also include randomly sampled data from alocal distribution D L i,k for task k , so the LL cost function at each node can be expressed asg i,k ( x i ,
y i,k ) ≜
E ζ ∈D L i,k
[ G i,k ( x i , y i ; ζ )] , ∀ k , where G i,k denotes the estimation error of the LLlearning model over y k,i on data ζ ∈ D L i,k .
It is well known that SGD is one of the most efficientalgorithms for tackling large amounts of data samples.
Before showing the algorithm design, wefirst reformulate problem (1) in a concise and compact way from a global view of the variables.
Let x ≜ [ x 1 , . . .
, x n ] T and y k ≜
[ y 1 ,k , . . .
, y n,k ] T .
Then, problem (1) can be rewritten by concatenatedvariables as
[Equation]
where g k ( x , y k ) denotes the k th LL loss function, A ∈ R |E|× n represents the incidence matrix 1 andf i ( x i , y ∗ i,k ( x i ))
abbreviates f i ( x i , y ∗ i, 1 ( x i ) , . .
, y ∗ i,m ( x i )) for notational brevity.
Algorithm Design.
Towards this end, it is straightforward to construct a variant of the classicalaugmented Lagrangian function for the UL optimization problem as
[Equation]
where λ denotes the dual variable (Lagrangian multiplier) for the consensus constraint, ρ > 0 , and γis a scaling factor (which will be determined later).
Motivated by the primal-dual optimization framework [44], one step of gradient descent based on thelinearized objective function with a following gradient ascent step is sufficient for the minimizationof the general nonconvex loss function under the linear constraints, which means that there is no needto solve an inner optimization problem before updating the Lagrangian multiplier as is done in theclassical augmented Lagrangian method.
Note that given y ∗ i,k , when both the UL and LL objective functions are differentiable and the inverseof the Hessian matrix at the LL problem exists, i.e., ∇
y
i,k y i,k g i,k ( x i , y ∗ i,k ( x i )) is invertible, thenthere exists a closed form for ∇ f i ( x i , y ∗ i,k ( x i ))
due to the finite sum structure of this class of DBOproblems (please see Lemma 1 in the supplement for details).
Following the existing works on bilevelalgorithm designs, replacing y ∗ i,k ( x i ) by y i,k in ∇ f i ( x i , y ∗ i,k ( x i )) can serve as an efficient surrogatefor the stochastic gradient estimate in the sense that the dissimilarity between ∇ f i ( x i , y ∗ i,k
( x i ))
and∇
f i ( x i , y i,k ) shrinks exponentially as the number of i.i.d.
samples data increases.
Let h rg,k and h rfrespectively denote the stochastic unbiased gradient estimate of the LL and UL objective functionsg k ( x r , y rk ) , f ( x r , y r +1 k ) , ∀ k , w.r.t.
y k and x , where r represents the index of iterations.
Thus, ourproposed SLAM can be expressed as
[Equation]
where ω k is the dual variable for ensuring the LL consensus process for each learning task, α and βare the parameters of the quadratic penalization terms, and ρ/γ here is the step-size for the updatesof the dual variables.
Implementation of SLAM.
Noting that the objective functions in each subproblem, i.e., (4a) and(4c), are quadratic, we can easily have the updates of both UL and LL optimization variables as
[Equation]
(5b) where 1 /α and 1 /β serve as the step-sizes of updating both UL and LL learning models.
Subtracting the equality with the same one from the previous iteration for both (5a) and (5b) ends upwith efficient model updates of both the UL and LL learning problems as follows:
[Equation]
where the mixing matrices, with τ g = β/γ and τ f = α/γ , are defined as
[Equation]
214 According to (6a) and (6b), it can be readily observed that SLAM is amenable to a fully decentralizedimplementation.
The detailed algorithm description is provided in Algorithm 1 from a local view ofthe model update, where [ W ] ij denotes the ij th entry of matrix W , [ h rg ] i,k is the gradient estimateof ∇ g
i,k ( x ri , y ri,k ) (i.e., h rg,k
=
[[ h rg ] 1 ,k , . .
, [ h rg ] n,k ] T ), and similarly [ h rf ] i is the local gradientestimate of ∇ f i ( x ri , y r
+1 i,k ) (i.e., h rf =
[[ h rf ] 1 , . . .
, [ h rf ] n ] T ).
Initialization: α, β, γ , x 1 i , y
i,k , ∀ i, k , and set λ
1 = ω 1 k = 0 , ∀ k ; 1: for r = 1 , 2 , · · · , T do 2: for i = 1 , 2 , · · · , n in parallel over the network do 3:
Estimate gradient ∇ g
i,k ( x ri , y ri,k ) for each task and ∇ f i ( x ri , y r
+
i,k ) locally 4:
[Equation] [Equation]
▷ UL model 6: end for 7: end for
Besides, if there is a consensus requirement at only one level of the optimization problem, then theproblem at the other level becomes one with multiple objective functions.
Our proposed SLAMcan also be applied for solving any of these problems by a minor revision of the generic SLAMformulation.
To be more specific, we provide the following discussion.
A Special Case of DBO (1) (with only consensus in the LL problems).
If there is only a need forconsensus of LL model parameters, then problem (2) reduces to the following DBO problem.
Forexample, in solving multi-agent actor-critic RL problems, the UL optimization problem consists ofimproving the policy for each agent while the LL problem requires all the agents to jointly evaluatethe value function over the whole network.
The DBO problem is then expressed as
[Equation]
8a
[Equation]
The major difference between problem (2) and (8) is that the UL optimization problem includesmultiple objectives over the model parameters x
i , ∀ i ∈ [ n ] .
In this case, the updating rule ofvariable x in (6b) reduces to x r +
= x
r − h rf /α by forgoing the dual update w.r.t.
λ .
The detailedimplementation is summarized in Algorithm 2, where we name this special case of SLAM by
SLAM-L as the LL consensus process is the main feature in this setting.
A Special Case of DBO (1) (with only consensus in the UL problem).
The other special isanalogous to the first one with the difference being the absence of the LL consensus process incomparison to (2), which is written as follows:
Initialization: α, β, γ , x 1 i , y
i,k , ∀ i, k , and set ω 1 k = 0 , ∀ k ; 1: for r = 1 , 2 , · · · , T do 2: for i = 1 , 2 , · · · , n in parallel over the network do 3:
Estimate gradient ∇ g
i,k ( x ri , y ri,k ) for each task and ∇ f i ( x ri , y r
+
i,k ) locally 4: y r
+
i,k = 2 (cid:80) j ∈N i
[ W g ] ij y rj,k − [ W ′ g ] ij y r − 1 j,k − β − 1 (cid:0) [ h rg ] i,k − [ h r − 1 g ] i,k (cid:1)
▷ LL models 5: x r
+1 i =
[Equation]
, ∀ i ▷ UL models 6: end for 7: end for min x
[Equation]
9b
[Equation]
where there are multiple objectives in the LL optimization problems.
Problem (9) also covers a widerange of applications in machine learning, e.g., multi-task and/or personalized learning, and so on.
Inthis case, the update of variable y k shown in (5a) is changed to y r +1 k
=
y rk − h
rg /β as there is noconsensus constraint involved.
Analogous to the previous case, the implementation of this algorithmis presented in Algorithm 3 and termed as SLAM-U.
Initialization: α, β, γ , x 1 i , y
i,k
− β − k ▷ LL models 5: x r
+1 i = 2 (cid:80) j ∈N
i
[ W f ] ij x rj − [ W ′ f ] ij x r − 1 j − α − 1 (cid:16) [ h rf ] i − [ h r − 1 f ] i (cid:17)
▷ UL model 6:
Theoretical Convergence Results
Before showing the theoretical results about the convergence guarantees of SLAM, we first need fivemain classes of assumptions used in showing the descent of some quantifiable function so that SLAMcan reach the ϵ -KKT points of the DBO problems.
More detailed definitions and properties regardingthese assumptions are deferred to the supplement.
3.1 Assumptions
Our theoretical results are based on the following assumptions on the properties of the loss functionsin both the UL and LL optimization problems, which are mainly related to the continuity of theobjective function and stochasticity of the gradient estimates.
A1.
(Lipschitz continuity of both UL and LL objective functions) Assume that functionsf i ( · ) , ∇ f i ( · , ) , ∇ g i,k ( · ) , ∇ 2 g i,k ( · ) , ∀ i , are (block-wise)
Lipschitz continuous with constants
L f, 0 , L f, 1 , L g, 1 , L g, 2 for both x and y k , ∀ k .
A2.
(Connectivity of graph G )
The communication graph G is assumed to be well-connected,i.e., 1 T L = 0 where L =
A T A , and the second-smallest eigenvalue of L is assumed to bestrictly positive, i.e., (cid:101) σ min ( A T A ) > 0 .
A3.
(Quality of the stochastic gradient estimate)
The stochastic estimates of ∇ x i f i ( x i , y ri,k ) ,∇ y
i
g
i,k ( x i , y i,k ) , ∇ 2 y i
y
i g i,k ( x i,k , y i,k ) , ∀ i, k , are unbiased and their variances are boundedby σ 2 f , σ 2 g, 1 , σ 2 g, 2 .
Further assume ∥ E [ h rf | x r , y r +1 k ] − ∇ x f ( x r , y r +1 k ) ∥
≤ b r where b r is aquantifiable parameter in terms of the mini-batch size.
A4.
Assume that the UL objective functions
f i ( x i , y ∗ i,k
( x i )) , ∀ i, k are lower bounded.
A5.
(Strong convexity of g i,k ( · ) w.r.t.
y )
Function g i,k ( · ) is µ g -strongly convex w.r.t.
y i,k , ∀ i, k .
Note that these assumptions are commonly used in the convergence analysis for bilevel and decentralized optimization algorithms.
Given these assumptions, we are now in a position to provide thefollowing theoretical convergence guarantees.
3.2 Convergence Rates of SLAM
Theorem 1.
(Convergence rate of SLAM to ϵ -KKT points) Suppose that A1-A5 hold.
When step-sizesare chosen as 1 /α ∼ 1 /β ∼ O ( (cid:112) n/T ) , τ f , τ g ≥ ρσ max ( A T A ) , the mini-batch size of h rf is
O (log( T )) , and ρ ≥ O ( n ) , then the iterates { x ri , λ r , y rk , ω rk , ∀ k, r } generated by SLAM satisfy
[Equation]
where x = n − 1 1 T x , and T denotes the total number of iterations.
Remark 1 .
It is noted in Theorem 1 that the convergence rate achieved by SLAM to find the ϵapproximate KKT points of (1) (including both the first-order stationarity of the solutions and theviolation of constraints) is on the order of 1 / ( nϵ 2 ) .
Therefore, it follows that a linear speedup w.r.t.
thenumber of learners can be achieved by SLAM for DBO, matching the classical results of distributed
SGD for only single-level general nonconvex problems.
Remark 2 .
In comparison with existing bilevel algorithms, SLAM is a single timescale algorithm asthe learning rates satisfy 1 /α ∼ 1 /β , which is consistent with ALSET [43].
Remark 3 .
The major novelty of obtaining these theoretical results relies on the developed variantof the augmented Lagrangian function and subsequently derived recursion of the successive dualvariables, which quantify well the consensus errors resulting from both UL and LL optimizationprocesses in terms of primal variables.
Note that this is distinct from the existing theoretical analysisof stochastic algorithms [38, 40, 41], even including stochastic primal-dual algorithms [42, 45], andalso the classical deterministic primal-dual algorithms [46] for decentralized optimization.
Corollary 1.
(Convergence rate of SLAM-L to ϵ -KKT points)
Given the conditions shown in Theorem 1 and 1 /α ∼ O (1 / √ T ) , the iterates { x ri , y rk , ω rk , ∀ k, r } generated by SLAM-L satisfy
UL
[Equation]
and
LL: (10b) .
Remark 4 .
Different from Theorem 1, the stationarity of the UL model parameters requires theshrinkage of the gradient size over each individual UL problem as shown in Corollary 1, so there isno speedup on the convergence rate guarantee at UL.
Corollary 2.
(Convergence rate of SLAM-U to ϵ -KKT points)
Given the conditions shown in Theorem 1, the iterates { x ri , λ r , y rk , ∀ k, r } generated by SLAM-U satisfy
UL: (10a) and
LL
[Equation]
Numerical Results
In this section, we evaluate our proposed algorithm using two MARL environments: 1) the cooperativenavigation task [16], which is built on the OpenAI Gym platform [47]; and 2) the pursuit-evasiongame [48], which is built on the PettingZoo platform [49].
Detailed experimental settings andadditional numerical results are provided in the supplement.
[Figure] [Figure]
Cooperative Navigation Task.
In this game, we consider that the n agents are aiming to jointly reachn different landmarks as soon as possible, where the Erdos Renyi Graph is used.
We assume that eachagent can observe the global state and has 5 possible actions: stay, left, right, up, and down.
Thistask consists of a shared common goal of avoiding collision among the agents while they navigate tothe targeting landmarks.
In the simulations, each agent locally maintains two fully connected neuralnetworks as the actor network (at UL w.r.t.
x i ) and the critic network (at LL w.r.t.
y i ), respectively.
Moreover, each agent shares its critic network with its neighbors to cooperatively estimate the globalvalue function and independently train its actor network to complete its local task.
We compare the performance of our proposed SLAM with application to the DAC setting, named
SLAM-AC, with two benchmark algorithms: DAC [15] and mini-batch DAC (MDAC) [22].
Theoretically, MDAC needs an O ( ϵ − 1 ln ϵ − 1 ) batch size in its inner loop to update critic parameters beforeeach update in policy parameters, which is not practical.
Here, we set a small batch B = 10 in theinner loop for MDAC to achieve fast convergence.
The simulation results on this coordination gameare presented in Figure1, where the performance is averaged over 5 independent Monte Carlo (MC)trials for each algorithm.
Pursuit-Evasion Game.
In the pursuit-evasion game, there are two groups of nodes: pursuers(agents) and evaders.
The agents are connected through a ring graph.
Pursuers could observe theglobal state of the video game.
An evader is considered caught if two pursuers simultaneously arriveat the evader’s location.
As each pursuer should learn to cooperate with other pursuers to catch theevaders, the pursuers share certain similarities with each other since they need to follow similarstrategies to achieve their local tasks: simultaneously catching an evader with other pursuers.
We follow the experimental set up in [23], where all agents partially share their actor networks withneighbors for collaborations in their policy spaces and fully share their critic network to cooperativelylearn the global value function.
In Figure 2, we compare SLAM-AC with two benchmarks, CAC [23]and MDAC [22], with 5 MC trials again.
To ensure a fair comparison, all algorithms use the sameparameter sharing scheme mentioned above.
Note that CAC [23] is a variant of DAC [15] and theonly difference is that CAC can partially share its policy parameters while the policy parameters arenot shared in DAC.
In the experiment, each agent maintains two convolutional neural networks, onefor the actor and one for the critic (Please refer to the supplement for detailed structures).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to 451 answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You arestrongly encouraged to include a justification to your answer , either by referencing the appropriate section of 453 your paper or providing a brief inline description.
For example:
▶ Did you include the license to the code and datasets?
[Yes] See
▶ Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
▶
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist 458 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,
internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
informa-
tion or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation?
[N/A]