Decentralized Gossip-Based Stochastic Bilevel Optimization over Communication Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Bilevel optimization have gained growing interests, with numerous applicationsfound in meta learning, minimax games, reinforcement learning, and nested composition optimization.
This paper studies the problem of distributed bilevel optimization over a network where agents can only communicate with neighbors, includingexamples from multi-task, multi-agent learning and federated learning.
In thispaper, we propose a gossip-based distributed bilevel learning algorithm that allowsnetworked agents to solve both the inner and outer optimization problems in asingle timescale and share information via network propagation.
We show that ouralgorithm enjoys the O ( 1 K ✏ 2 ) per-agent sample complexity for general nonconvexbilevel optimization and O ( 1 K ✏ ) for strongly convex objective, achieving a speedupthat scales linearly with the network size.
The sample complexities are optimal inboth ✏ and K .
We test our algorithm on the examples of hyperparameter tuningand decentralized reinforcement learning.
Simulated experiments conﬁrmed thatour algorithm achieves the state-of-the-art training efﬁciency and test accuracy.
Introduction
In recent years stochastic bilevel optimization (SBO) has attracted increasing attention from themachine learning community.
It has been found to provide favorable solutions in a variety ofproblems, such as meta learning and hyperparameter optimization (Franceschi et al., 2018;
Snell et al.
,2017; Bertinetto et al., 2018), composition optimization (Wang and Liu, 2016b), two-player games(Von Stackelberg and Von, 1952), reinforcement learning and imitation learning (Arora et al., 2020;
Hong et al., 2020).
While the majority of the above mentioned work focuses on algorithm designs inthe classic centralized setting, such problems often arise in distributed/federated applications whereagents are not willing to share data but rather perform local updates and communicate with neighbors.
Theories and algorithms for distributed stochastic bilevel optimization are less developed.
Consider the decentralized learning setting where the data are distributed
We consider the following decentralized stochastic bilevel optimization problem,
[Equation]
[Figure]
where f k ( x, y ) =
E ⇣ k
[ f k ( x, y ; ⇣ k )] and g k ( x, y ) =
E ⇠ k [ g k ( x, y ; ⇠ k )] may vary across agents.
The expectations E ⇣ k
[ • ] and E ⇠ k [ • ] are taken with respect to the random variables ⇣ k and ⇠ k ,with heterogeneous distributions across agents.
We denote by f ( x, y )
= 1 K
P k 2 K f k
( x, y ) andg ( x, y )
= 1 K
P k 2 K g k
( x, y ) for notational convenience.
1.1 Example applications of SBO
SBO was ﬁrst employed to formulate the resource allocation problem (Bracken and McGill, 1973)and has since found its application in many classic operations research settings (Cramer et al., 1994;
Sobieszczanski-Sobieski and Haftka, 1997; Livne, 1999; Tu et al., 2020; Tu, 2021), and more recentlyin machine learning problems (Franceschi et al., 2018;
Snell et al., 2017; Bertinetto et al., 2018;
Wang and Liu, 2016b).
In particular, we introduce two applications that have received a lot of recentattention, namely hyperparameter optimization and compositional optimization.
Hyperparameter Optimization
The problem of hyper-parameter tuning (Okuno et al., 2021) oftentakes the following form:
[Equation]
where D train and D val are two datasets used for training and validation, respectively,
y 2 R d y is avector of unknown parameters to optimize, ` j ( y ) is a convex loss over data i , and x 2 R d is a vectorof hyper-parameters for a strongly convex regularizer R ( x, y ) .
For any hyper-parameter x 2 R d ,the inner-level problem solves for the best parameter y ⇤ ( x ) over the training set D train under theregularized training loss ` j ( y )
+ R ( x, y ) .
The goal is to ﬁnd the hyper-parameter x
⇤ 2 R d
whosecorresponding best response y ⇤ ( x ) yields the least loss over the validation set D val .
In practice,tuning continuous hyperparameters is often done by grid search which is exponentially expensive.
Efﬁcient algorithm of SBO should ﬁnd optimal parameters in time polynomial to dimension, ratherthan exponential.
When the training and validation set are distributed across nodes, the problembecomes a distributed SBO.
Compositional Optimization Let g ( x, ⇠ ) : R d !
R d y and f ( y, ⇣ ) :
R d y !
R be two stochasticmappings, stochastic compositional optimization (SCO) (Wang et al., 2017a) takes the form
[Equation]
SCO applies to risk management (Yang et al., 2019; Ruszczynski, 2021), machine learning (Chenet al., 2021b), and reinforcement learning (Wang et al., 2017b).
SCO was identiﬁed as a specialcase of SBO by (Chen et al., 2021c).
To see this, we take the inner optimization objective to beg ( x, y ) =
E ⇠ [( y  h ( x, ⇠ )) 2 ] .
Thus y ⇤ ( x ) = argmin y g ( x, y ) becomes y ⇤
( x ) =
E ⇠ [ h ( x ; ⇠ )] , arrivingat an instance of SBO.
1.2 Challenges with Distributed SBO
Despite the recent fast development of distributed single-level optimization, a proper method for
Decentralized Stochastic Bilevel Optimization (DSBO) remains illusive.
The major hurdle of solving
DSBO lies in the absence of explicit knowledge of y ⇤ ( x ) , so that an unbiased gradient of r f ( x, y ⇤ ( x ))is unavailable.
Recently, by applying the implicit function theorem, (Couellan and Wang, 2016;
Ghadimi and Wang, 2018) showed that the gradient of non-distributed SBO can be expressed as
[Equation]
providing a connection between SBO and classical stochastic optimization.
Since then, variousalgorithms have been proposed to obtain sharp estimators for y ⇤ ( x ) and reduce the bias of theconstructed gradients (Chen et al., 2021a;
Hong et al., 2020; Ji et al., 2021;
Yang et al., 2021).
Thesetechniques result in nearly tight convergence analysis and give rise to algorithms widely used inapplications.
However, none of the prior algorithms can be applied to the distributed setting.
Solving Problem (1) becomes challenging in the distributed setting in several aspects:
Even in single-agent SBO, lack of y ⇤ ( x ) makes outer optimization nontrivial and estimating
y ⇤ ( x ) requires additional stochastic approximation, weighted averaging, and calculation.2.
Calculating outer gradient is highly nontrivial, even we have an inner solution y .
Note that
[Equation]
In other words, even the inner problem is solved, the outer gradient takes additional mechasim
toestimate.3.
Now we move to SBO over distributed networks.
In network learning, communication amongagents can be limited by the network stucture and communication protocol, so taking a simpleaverage across agents may require multiple rounds of communication.
Because of the above hurdles, it remains unclear how to develop an algorithm that estimates theouter gradient sharply under a distributed network.
As an attempt to tackle SBO in distributednetworks, this paper study the convergence theory and sample complexity of gossip-based algorithm.
In particular, we ask two theoretical questions:
(i) How does the sample complexity of DSBO scale with the optimality gap and network size?(ii)
How is the efﬁciency of DSBO affected by the network structure?
In this paper, we develop a gossip-based stochastic approximation scheme where each agent solvesan optimization problem collaboratively by sampling stochastic ﬁrst- and second-order informationusing its data and making gossip communications with its neighbors.
In addition, we develop novelconvergence analysis techniques to characterize the convergence behavior of our algorithm.
To thebest of our knowledge, our work is the ﬁrst to formulate DSBO mathematically and propose analgorithm with theoretical convergence guarantees.
Moreover, we show that our algorithm enjoys an
O ( 1 K ✏ 2 ) sample complexity for ﬁnding ✏ -stationary points for nonconvex objectives and enjoys an O ( 1 K ✏ ) sammple complexity for µ -PL functions, subsuming strongly convex optimization.
Theseresults match the state-of-the-art results for non-federated stochastic bilevel optimization (Chen et al. ,2021c) and central-server regime (Tarzanagh et al., 2022), showing nearly no degradation inducedby network consensus.
Further, these results suggest that our algorithm exhibits the linear speed-upeffect for decentralized settings; that is, the required per-agent sample complexities decrease linearlyto the number of agents.
Related Works
Bilevel optimization was ﬁrst formulated by (Bracken and McGill, 1973) for solving resourceallocation problems.
Later, a class of constrained-based algorithms was proposed by (Hansenet al., 1992; Shi et al., 2005), which treats the inner-level optimality condition as constraints tothe out-level problem.
Recently, (Couellan and Wang, 2016) examined the ﬁnite-sum case forunconstrained strongly convex lower-lower problems and proposed a gradient-based algorithm thatexhibits asymptotic convergence under certain step-sizes.
For SBO, (Ghadimi and Wang, 2018)
Stochastic
Optimization
BSA stocBio
ALSET
FEDNEST
This Work
Network
Single-Agent
Central-Server
Decentralized
Samples in ⇣
O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 ) O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 + 1 (1 p ⇢ ) 2 ✏ )
Samples in ⇠
O ( ✏  3 )
O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 + 1 (1 p ⇢ ) 2 ✏ )
Stochastic Compositional Optimization
SCGD
NASA
ALSET
FEDNEST
This Work
Network
Single-Agent
Central-Server
Decentralized
Samples in ⇣
O ( ✏  4 )
O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 ) O ( ✏  2 )
O ( 1 K ✏ 2 + 1 (1 p ⇢ ) 2 ✏ )
Samples in ⇠
O ( ✏  4 )
O ( ✏  2 ) O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 )
O ( 1 K ✏ 2 + 1 (1 p ⇢ ) 2 ✏ ) developed a double-loop algorithm and established the ﬁrst known complexity results.
Subsequently,various methods have been employed to improve the sample complexity, including two-timescalestochastic approximation (Hong et al., 2020), acceleration (Chen et al., 2021a), momentum (Khanduriet al., 2021), and variance reduction (Guo et al., 2021;
Ji et al., 2021; Yang et al., 2021).
From theoptimization perspective, it is worth emphasizing that bilevel optimization can be treated as a specialcase of mathematical programs with equilibrium constraints (Luo et al., 1996).
Distributed optimization was developed to handle real-world large-scale datasets (Dekel et al., 2012;
Feyzmahdavian et al., 2016).
Centralized and decentralized systems are two important problems thathave drawn signiﬁcant attention.
A centralized system considers the network topology where thereexists a central agent that communicates with the remaining agents (Lan and Zhou, 2018), while ina decentralized system (Gao and Huang, 2020; Koloskova et al., 2019; Lan et al., 2020; McMahanet al., 2017), each agent can only communicate with its neighbors by using gossip (Lian et al., 2017)or gradient tracking (Pu and Nedi´c, 2021) communication strategies.
Variance reduction approaches(Xin et al., 2020, 2021) have also been applied to improve the convergence rate of decentralizedoptimization.
All of the above trials were made on vanilla stochastic optimization problems.
We arethe ﬁrst to study the decentralized bilevel optimization problem to our knowledge.
Table 1 comparesthis work with prior arts under different settings.
Problem Setup
Assumption 3.1 (Sampling Oracle SO )
Agent k may query the sampler andreceive an independent local sampled ﬁrst and second order informationr x f k ( x, y ; ⇣ k ) , r y f k ( x, y ; ⇠ k ) , r [-
-]yx g k
( x, y ; ⇠ k ) , and r 2 yy g ( x, y ; ⇠ k ) with bounded secondorder moments.
Assumption 3.2 (Gossip Protocol)
The network gossip protocol is speciﬁed by a K ⇥ K symmetricmatrix W with nonnegative entries.
Each agent k may receive information from its neighbors, e.g.,z j , j 2 N k , and aggregate them by a weighted sum P
[Equation]
w k,j z j .
Further, Matrix W satisﬁes
(i) W is doubly stochastic such that P
i w i,j = 1 and P
j w i,j = 1 for all i, j 2 [ K ] .
(ii) There exists a constant ⇢ 2 (0 , 1) such that max {|  2 ( W > W ) | , |  n ( W > W ) |}  ⇢ where  k ( A ) denotes the k -th largest eigenvalue value of A 2 R K ⇥ K .
These assumptions on the adjacency matrix are crucial to ensure the convergence of decentralizedalgorithms, and are commonly made in the decentralized optimization literature (Lian et al., 2017).
Assumption 3.3 (Smoothness) We impose the following smoothness assumptions: input: Step-sizes { ↵ t } , {  t } , {  t } , total iterations T , hessian coefﬁcient { b t } , adjacency matrix W , smoothness constant L g .
[-y g k ( x, y ) , r 2 yx g k ( x, y ) , and r 
 yy g k ( x, y ) are L g -Lipschitz continuous in ( x, y ) . input: Step-sizes { ↵ t } , {  t } , {  t } , total iterations T , adjacency matrix W , smoothness constant L g , hessian sampling parameter b-] x k 0 = 0 , y k
= 0 , u k 0 = 0 , v k 0 = 0 , s k 0 = 0 , h t 0 =1: for t = 0 , 1 , · · · , T  1 do 2: for k = 1 , · · · , K do 3:
Local sampling: Query SO at ( x kt , y kt ) to obtain r x f k ( x kt , y kt ; ⇣ kt ) , r y f k ( x kt , y kt ; ⇣ kt ) , r y g k ( x kt , y kt ; ⇠ kt ) , r yx g k
( x kt , y kt ; ⇠ kt ) , and { r[-
-] yy g k ( x kt , y kt ;
⇠ kt,i ) } b t i =1 .
Outer loop update: x kt +1 =
P j 2 N k w k,j x jt  ↵ t  s kt  u kt v kt h kt  .
Inner loop update: y kt
+1 =
P j 2 N k w kj y jt   t r y g k ( x kt , y kt , ⇠ kt ) .
Estimate r x f ( x t , y t ) : s kt
+
=
(1   t )
P j 2 N k w kj s jt +  t r x f k ( x kt , y kt ; ⇣ kt ) .
Estimate r y f ( x t , y t ) : h kt +
=
(1   t )
P j 2 N k w kj h jt +  t r y f k ( x kt , y kt ; ⇣ kt ) .
Estimate r yx g ( x t , y t ) : u {+kt +
+}
= (1   t ) P j 2 N k w kj u jt +  t r 2 yx g k ( x kt , y kt , ⇠ kt ) .
Estimate [ r yy g ( x t , y t )]  1 : Set q kt [-+1 , 
-]= b t L g
{+Q pi =
+}
( I  1 L g r yy g k ( x kt , y kt ; ⇠ kt )) , where p ⇠ Unifom [0 , · · · , b t  1] .
{+Update v kt +
+}
= (1   t )
P j 2 N k w kj v jt +  t q kt .
10: end for 11: end for r y f
[-Set q kt +1 = 
-], y ) L Lipschitz continuous x
y g {+x, y ) , r 2 yx g k x, y ) , and r 
+}
yy g k ( x, y ) are L g -Lipschitz continuous in ( x, y ) .
We defer the detailed assumptions to Section A.1 of the supplement.
Here we point out that the above assumptions allow the heterogeneity of functions f k ’s and g k ’s overthe agents, and the smoothness and boundedness conditions are commonly adopted in SBO (Honget al., 2020; Chen et al., 2021a).
Algorithm
As discussed in Section 1.2, the key challenge in solving DSBO is that each agent has only accessto its own data but is required to construct estimators for the gradients and Hessian averaged acrossall agents.
It is particularly challenging to construct such estimators when limited by the network’scommunication protocol.
To overcome this issue, we propose a gossip-based DSBO algorithm where each agent k 2 Kiteratively updates a solution pair ( x k t , y k t ) by using the combination of gossip communications andweighted-average stochastic approximation, where y kt serves as an estimator of the best responsey ⇤ t :=
y ⇤ (¯ x t ) with ¯ x t := 1 K
P k 2 K x kt .
The full algorithm is given in Alg. 1.
Here we brieﬂy illustrate the idea of our DSBO algorithm.
Suppose agent k would like to estimater x f ( x kt , y kt ) by s kt , under Alg.
1 Step 6, it would query the stochastic ﬁrst-order information usingits own data, make gossip communications with neighbors, and update its estimators by taking theweighted average among its previous estimate s kt  1 , neighbors j ’s estimate s jt  1 , and the newlysampled gradient r x f k ( x kt , y kt ; ⇣ kt ) .
Roughly speaking, this procedure can be viewed as takingweighted average among gradients sampled by all agents over the network, despite the effect ofconsensus should also been taken into account.
The updates for r y f k and r yx g k are conducted in a similar manner (Steps 7 & 8), but it takes extraefforts to evaluate [ r yy g k ]  1 , due to the absence of its unbiased estimator.
To solve this issue, agentk uses a stochastic power iteration (similar to the HIA algorithm of Ghadimi et al.
to constructa nearly unbiased estimator q kt using b t stochastic Hessian { r[-
-] yy g k ( x kt , y kt ; ⇠ ki ) } b t i =1 sampled fromits own data, whose bias decays to zero geometrically.
[-We then estimate r 
 yy g ( x-]With such q kt , agent k can estimates r x f k viaconsensus and stochastic approximation (Step 9).
Finally, by utilizing the estimators obtained in the above procedure, each agent would compute thegradient (3) and update the outer solution x kt by using the combination of gossip communicationand stochastic gradient descent (Step 3).
The inner loop solution y kt can be updated by (Step 4) in asimilar way.
We highlight the following key features of our DSBO algorithm: (1) each agent would only communicate their estimates instead of the raw data in the gossip-communication process, which preservesdata privacy.
(2) agent k would make O ( |N k | ) communications with its neighbors in each round, 173 which is much smaller than the total number of agents in an naive approach.
(3) the algorithm isrobust to contingencies in the network.
If some agent fail, the rest of the agents can still jointly learnprovided that the network is still connected.
In contrast, a single-center-multi-user network wouldfail completely in case of a center failure.
Theory
In this section, we analyze the performance of our DSBO algorithm for both nonconvex and µ -PLobjectives, and derive the convergence rates for both cases.
5.1 Nonconvex Objectives
We ﬁrst consider the scenario where overall objective function F ( x ) is nonconvex.
For nonconvexobjectives, given the total number of iterations T , we employ the step-sizes in a constant form that
[Equation]
and b t
[Equation]
, for all t = 0 ,
[Equation]
[Equation]
(5) where C 0 > 0 is a small constant and ˜ c > 1 .
Compounded effect of consensus and SBO: As discussed earlier, to derive the convergence rateof SBO under decentralized federated setting, the key step is to quantify the compounded effectbetween the consensus errors induced by the network structure and the biases induced by estimatinggradient within (3).
Distinct from the central-server or non-federated regimes, the consensus errorsinduced by the decentralized network structure has to be handled carefully.
To address this issue, wedecompose the averaged estimation error across all agents into a stochastic recursion, where the biasterms are induced by consensus errors and variance terms are incurred from the stochastic samplingenvironment.
We conduct careful analysis to derive the contraction of consensus errors, and furthershow that both bias and variance of the averaged estimator diminish to zero, establishing a nontrivialconvergence argument to the desired gradient and Hessian.
In particular, the estimators preserve aconcentration property so that their variances decrease proportionally to 1 /K , suggesting that thenetwork consensus effect would not degrade the concentration of the generated stochastic samples.
To achieve the best possible convergence rate, we carefully set the algorithm parameters, includingstep-sizes ↵ t ,  t and averaging weights  t , to control the above consensus errors and biases.
We provide the convergence rate of Alg. 1 in the following theorem and defer the detailed proof to
Section B.8 of the supplementary material.
Theorem 5.1 Suppose Assumptions 3.1, 3.2, and 3.3 hold.
Then
[Equation]
More details and proof are deferred to Section B.8 of the supplement.
Effect of consensus: In this result, the O ( 1 T (1 p ⇢ ) 2 ) term represents the errors induced by consensusof network.
Despite the dependency with the network structure, such term diminishes to zero in theorder of O (1 /T ) , becoming a small order term when T is large.
Consequently, given the networktopology, the asymptotic convergence behavior of DSBO is independent of the network structure,answering the question (ii) raised in Section 1.
Linear speedup: Because each agent queries O (1) stochastic samples per round, clearly therequired iteration and per-agent sample complexities for ﬁnding an ✏ -stationary point such that
[-P T  1 t =
-][Equation]
both O ( 1 K ✏ 2 ) .
Such result implies that our algorithm achieves a linear speed-up effect proportionate to the number of agents K .
In other words, in the presence ofmore agents, each agent needs to query fewer stochastic samples to achieve a speciﬁed accuracy.
Meanwhile, our rate also matches the best-known O (1 /K ✏ 2 ) iteration and per-agent
sample complexities under the decentralized vanilla stochastic gradient descent settings (Lian et al., 2018).
Toour knowledge, it is the ﬁrst time such a result has been established for DSBO problems.
Single-center-multi-user-federated SBO :
We point out that a simpliﬁed version of our algorithmsolves single-center-multi-user-federated SBO, where the central server directly collects informationfrom each agent and calculates the gradient by employing the weighted-average stochastic approximation scheme among the collected information.
In such scenario, the agents no longer make gossipcommunication with neighbors but synchronously receives a common solution from the central server,so that the consensus effect would disappear.
In Theorem D.1, we show that a variant of our algorithmcan indeed achieve the same O ( 1 p KT ) convergence rate in this setting.
5.2 µ -PL Objectives
Next we study the case where the objective function satisﬁes the following µ -PL condition.
Assumption 5.2 There exists a constant µ > 0 such that the objective satisﬁes the PL condition:
[Equation]
Note that the class of strongly convex functions is a special case of µ -PL functions.
To utilize theµ -PL property and achieve fast convergence, unlike the nonconvex case (5) where the step-sizes areset as constants depending on the total number of iterations T , we employ step-sizes in a diminishingform that
[Equation]
where C 1 > 0 is a large constant and ˜ c > 1 .
By following an analysis routine similar to that in thenonconvex scenario, in the next result, we derive the convergence rate of Alg. 1 for µ -PL objectives.
Theorem 5.3 Suppose Assumptions 3.1, 3.2, and 3.3 hold and the function satisﬁes the µ -PL Assumption 5.2.
Then
[Equation]
Both iteration and per-agent sample complexities for ﬁnding an ✏ -optimal point E [ F (¯ x T )]
F ⇤  ✏are O ( 1 K ✏ ) .
Details and proof are defered to Section C.3 of the supplement.
This result shows that our algorithmachieves a faster convergence rate for functions satisfying the µ -PL condition in terms of bothiteration and sample complexities.
First, similar as the nonconvex scenario, the consensus errordecays in the order of ˜ O
( 1 T 2 (1  ⇢ ) ) .
Dominated by O ( 1 KT ) , such consensus decaying order indicatesthat the network structure would not affect Alg.
1’s asymptotic convergence behavior under µ -PLobjectives.
Meanwhile, the above result implies that Alg.
1 speeds up linearly with the number ofagents and matches the optimal O (1 / ✏ ) sample complexity for single-sever vanilla strongly-convexstochastic optimization (Rakhlin et al., 2011).
As a result, our algorithm achieves the optimal samplecomplexities for decentralized stochastic bilevel optimization, establishing the benchmark.
Numerical Experiments
In this section, we validate the practical performance of our algorithm in two examples: hyperparameter optimization and policy evaluation in Markov Decision Processes (MDP), on artiﬁciallyconstructed decentralized ring networks.
We run the experiments on a single server desktop computer.
We provide the details of federate hyper-parameter optimization and federated policy evaluation.
Hyper-parameter Optimization We consider federated hyper-parameter optimization (2) for ahandwriting recognition problem over the Australia handwriting dataset (Chang and Lin, 2011)consisting of data points ( w i , z i ) , where w i 2 R 14 is the feature and z i 2 { 0 , 1 } represents whether (a)    8000 0. [Figure] this data point belongs to category “1” or not.
In our experiment, we consider the sigmoid lossfunction that l i ( z )
= 1 / (1 + exp
(  z )) and a strongly convex regularizer R ( x, y ) =
P d i =1 x i 2 k y i k 2 .
We consider a ring network of K agents where each agent i preserves two neighbors ( i  1) and ( i +1)and conduct gossip communication strategy with adjacency matrix w i,j = 13 for j 2 { i  1 , i, i + 1 } .
Before testing Alg.
1, we ﬁrst randomly split the dataset for training and validation, and then allocatesboth training and validation dataset over K agents.
For benchmark comparison, we implement a baseline algorithm Decentralized Bilevel Stochastic
Approximation (DBSA), a naive extension of the double-loop BSA algorithm
(Ghadimi and Wang,2018) in the decentralized setting, formally stated in Section E.1 of the supplementary materials.
We ﬁrst consider K = 5 , test Alg.
1 for 5 ⇥ 10 4 iterations, and compare its performance with DSBO.
We report the validation loss against total samples in Figure 2 and observe that DSBO exhibitssuperior performance to DBSA.
In particular, Further, we observe that our algorithm outperforms thebaseline algorithm DSGD that it requires fewer samples for DSBO to achieve a certain accuracy than DSGD.
Further, our DSBO algorithm is able to ﬁnd a solution whose validation loss is only 0.308while DSGD can only ﬁnd solutions that have validation loss around 0.326.
This suggests DSGDmay not converge to the optimal solution even if sufﬁciently large number of iterations are conducted,because the naive stochastic gradient employed by DSGD is not necessarily an unbiased estimator ofthe overall gradient, as illustrated in (4).
To investigate the efﬁciency of Alg. 1 to the network structure, we test Alg.
1 over K = 5 , 10 , 20 ,and report the details of training and validation loss in Figure 2.
Further, comparing the performancesof Alg. 1 over different agents K = 5 , 10 , 20 , we observe that Alg.
1 converges faster by using moreagents.
This observation suggests that Alg.
1 exhibits a speed-up effect by using more agents.
Weprovide additional numerical results regarding prediction accuracy in Section E.1 of the supplement.
Distributed policy evaluation for reinforcement learning We consider a multi-agent MDPproblem arising in reinforcement learning.
Let S be the state space, for any state s 2 S , we denote by 277 V ( s ) the value function.
we consider the scenario where the value function can be approximated by alinear function such that V ( s ) =
> s x ⇤ , where  s 2 R m is a feature and x ⇤
R m is an unknownparameter.
To obtain the optimal x
⇤ , we consider the following regularized Bellman minimization problem
[Equation]
where r ( s, s 0 ) is the random reward incurred from transition s to s 0 ,  2 (0 , 1) is the discount factor, is the coefﬁcient for the l 2 -regularizer, and the expectation is taken over all random transitions froms to s 0 .
In the federated learning setting, we consider a ring network of K agents.
Here each agent k hasaccess to its own data with heterogeneous random reward function r k and can only communicate withits two neighbors
{+k +
+}
and k  1 .
We denote by y ⇤ s ( x ) =
> s x  E s
[ 1 K
P
[Equation]
2.5 10 - (a)    9 - - - - - (b)   208.
(c) where r k ( s, s 0 ) is the random reward function for agent k , the above problem can be recast as abilevel optimization problem
[Equation]
As pointed out by (Wang and Liu, 2016a), the above problem is  -strongly convex.
In our experiments, we simulate an environment with state space |S| = 100 and set the regularizer 291 parameter  = 1 .
We test the performance of Alg.1 over three setups with K = 5 , 10 , 20 and conduct10 independent simulations for each K .
We implement a baseline double-loop algorithm DSGD thatﬁrst estimates y ⇤ s ( x t ) with t samples in iteration t and then optimizes the solution x t .
We defer theimplementation details of the environment and above algorithms to Section E.2 of the supplement.
We ﬁrst consider K = 5 , run Alg. 1 for 10 4 iterations and compare its performance with DSGD.
Weplot the empirical averaged mean square error
k ¯ x t  x ⇤ k 2 against total samples generated by all agentsin Figure 2.
This empirical result suggests that Alg.
1 outperforms the DSGD.
To investigate theconvergence rate of DSBO, we compare the performance of DSBO over all three setups K = 5 , 10 , 20and plot the trajectory of averaged log-error log( k ¯ x t  x
⇤ k 2 ) averaged, with one straight line of slope-1 provided for comparison.
We observe that over all four setups, the slopes of log( k ¯ x t  x ⇤ k 2 ) areclose to -1, matching our theoretical claim in Theorem 5.
that Alg.
1 converges at the rate of O (1 /t )for strongly convex objectives.
In the above experiment, we also note that Alg.
1 converges faster when using more agents.
Tofurther demonstrate the linear speedup effect, we compute the total generated samples for ﬁnding an✏ -optimal solution k ¯ x t  x
⇤ k 2  ✏ and plot the 75% conﬁdence region of log-sample against number 306 of agents K = 5 , 10 , 20 in Figure 3.
We observe that it takes a roughly same amount of samplesto ﬁnd a 10  6 -optimal solution despite different number of agents are involved.
This suggests thatthe per-node sample complexity decreases linearly to K and validates the linear speedup claim in
Theorem 5.3.
We provide additional numerical results for other optimality level ✏ in Section E.2 ofthe supplementary material to further demonstrate the linear speedup effect.
Conclusion
In this paper, we propose a novel formulation of decentralized stochastic bilevel optimization.
Wedevelop a gossip-based stochastic approximation scheme to solve this problem in various settings.
We show that our proposed algorithm ﬁnds a stationary point at a rate of O ( 1 p KT ) for nonconvexobjectives, and converges to the optimal solution at a rate of O ( 1 KT ) for strongly convex objectives.
Numerical experiments on hyper-parameter optimization and multi-agent federated MDP demonstratethe practical efﬁciency of our algorithm, exhibit the effect of speed-up in decentralized setting, andvalidate our theoretical claims.
In our future work, we wish to develop algorithms that achieve loweriteration complexities and enjoy lower communication cost.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is an algorithmic and theoretical work justiﬁed by numerical experiments.
We are not aware of any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Complete proofs are
provided in the supplement.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We will providethe code via github in the camera-ready version stage.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We provided averaged results over different random seeds
and the obtained results are stable.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
The dataset is publicly available and
no license is required.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We are not releasing new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
We cited the website link where data is presented.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No] The data does not contain personally identiﬁable
information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
We did not involve human subjects.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
on participant compensation?
[N/
[Table]