Decentralized Gossip-Based Stochastic Bilevel Optimization over Communication Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Bilevel optimization have gained growing interests, with numerous applicationsfound in meta learning, minimax games, reinforcement learning, and nested composition optimization.
This paper studies the problem of distributed bilevel optimization over a network where agents can only communicate with neighbors, includingexamples from multi-task, multi-agent learning and federated learning.
In thispaper, we propose a gossip-based distributed bilevel learning algorithm that allowsnetworked agents to solve both the inner and outer optimization problems in asingle timescale and share information via network propagation.
We show that ouralgorithm enjoys the O ( 1 K ✏ 2 ) per-agent sample complexity for general nonconvexbilevel optimization and O ( 1 K ✏ ) for strongly convex objective, achieving a speedupthat scales linearly with the network size.
The sample complexities are optimal inboth ✏ and K .
We test our algorithm on the examples of hyperparameter tuningand decentralized reinforcement learning.
Simulated experiments conﬁrmed thatour algorithm achieves the state-of-the-art training efﬁciency and test accuracy.
Introduction
In recent years stochastic bilevel optimization (SBO) has attracted increasing attention from themachine learning community.
It has been found to provide favorable solutions in a variety ofproblems, such as meta learning and hyperparameter optimization (Franceschi et al., 2018;
Snell et al.
,2017; Bertinetto et al., 2018), composition optimization (Wang and Liu, 2016b), two-player games(Von Stackelberg and Von, 1952), reinforcement learning and imitation learning (Arora et al., 2020;
Hong et al., 2020).
While the majority of the above mentioned work focuses on algorithm designs inthe classic centralized setting, such problems often arise in distributed/federated applications whereagents are not willing to share data but rather perform local updates and communicate with neighbors.
Theories and algorithms for distributed stochastic bilevel optimization are less developed.
Consider the decentralized learning setting where the data are distributed optimization
[Equation]
[Table]
where f k ( x, y ) =
E ⇣ k
[ f k ( x, y ; ⇣ k )] and g k ( x, y ) =
E ⇠ k [ g k ( x, y ; ⇠ k )] may vary across agents.
The expectations E ⇣ k
[ • ] and E ⇠ k [ • ] are taken with respect to the random variables ⇣ k and ⇠ k ,with heterogeneous distributions across agents.
We consider the scenario where each g k ( x, y ) isstrongly convex in y .
We denote by F ⇤ = min x 2 R dx F ( x ) , f ( x, y )
= 1 K
P k 2 K f k
( x, y ) , andg ( x, y )
= 1 K
P k 2 K g k
( x, y ) for notational convenience.
1.1 Example applications of SBO
SBO was ﬁrst employed to formulate the resource allocation problem (Bracken and McGill, 1973)and has since found its application in many classic operations research settings (Cramer et al., 1994;
Sobieszczanski-Sobieski and Haftka, 1997; Livne, 1999; Tu et al., 2020; Tu, 2021), and more recentlyin machine learning problems (Franceschi et al., 2018;
Snell et al., 2017; Bertinetto et al., 2018;
Wang and Liu, 2016b).
In particular, we introduce two applications that have received a lot of recentattention, namely hyperparameter optimization and compositional optimization.
Hyperparameter Optimization
The problem of hyper-parameter tuning (Okuno et al., 2021) oftentakes the following form:
[Equation]
where D train and D val are two datasets used for training and validation, respectively,
y 2 R d y is avector of unknown parameters to optimize, ` j ( y ) is a convex loss over data i , and x 2 R d is a vectorof hyper-parameters for a strongly convex regularizer R ( x, y ) .
For any hyper-parameter x 2 R d ,the inner-level problem solves for the best parameter y ⇤ ( x ) over the training set D train under theregularized training loss ` j ( y )
+ R ( x, y ) .
The goal is to ﬁnd the hyper-parameter x
⇤ 2 R d
whosecorresponding best response y ⇤ ( x ) yields the least loss over the validation set D val .
In practice,tuning continuous hyperparameters is often done by grid search which is exponentially expensive.
Efﬁcient algorithm of SBO should ﬁnd optimal parameters in time polynomial to dimension, ratherthan exponential.
When the training and validation set are distributed across nodes, the problembecomes a distributed SBO.
Compositional Optimization Let g ( x, ⇠ ) : R d !
R d y and f ( y, ⇣ ) :
R d y !
R be two stochasticmappings, stochastic compositional optimization (SCO) (Wang et al., 2017a) takes the form
[Equation]
[Equation]
[Equation] [Equation] ⇣ )] . SCO applies to risk management (Yang et al., 2019; Ruszczynski, 2021), machine learning (Chenet al., 2021b), and reinforcement learning (Wang et al., 2017b).
SCO was identiﬁed as a specialcase of SBO by (Chen et al., 2021c).
1.2 Challenges with Distributed SBO
Despite the recent fast development of distributed single-level optimization, a proper method for
Decentralized Stochastic Bilevel Optimization (DSBO) remains illusive.
The major hurdle of solving
DSBO lies in the absence of explicit knowledge of y ⇤ ( x ) , so that an unbiased gradient of r f ( x, y ⇤ ( x ))is unavailable.
Recently, by applying the implicit function theorem, (Couellan and Wang, 2016;
Ghadimi and Wang, 2018) showed that the gradient of non-distributed SBO can be expressed as
[Equation]
providing a connection between SBO and classical stochastic optimization.
Since then, variousalgorithms have been proposed to obtain sharp estimators for y ⇤ ( x ) and reduce the bias of theconstructed gradients (Chen et al., 2021a;
Hong et al., 2020; Ji et al., 2021;
Yang et al., 2021).
Thesetechniques result in nearly tight convergence analysis and give rise to algorithms widely used inapplications.
However, none of the prior algorithms can be applied to the distributed setting.
Solving Problem (1) becomes challenging in the distributed setting in several aspects:
Even in single-agent SBO, lack of y ⇤ ( x ) makes outer optimization nontrivial, and estimating y ⇤ ( x )
requires additional stochastic approximation, weighted averaging, and sophisticated calculation.2.
Calculating outer gradient is highly nontrivial, even when we have an inner solution y .
Note that
[Equation]
In other words, even the inner problem is solved, the outer gradient requires new
mechanism toestimate.3.
Now we move to SBO over distributed networks.
In network learning, communication amongagents can be limited by the network stucture and communication protocol, so taking a simpleaverage across agents may require multiple rounds of communication.
Because of the above hurdles, it remains unclear how to sharply estimate the outer gradient under adistributed network.
As an attempt to tackle this problem, this paper study the convergence theoryand sample complexity of gossip-based algorithm.
In particular, we ask two theoretical questions:
(i) How does the sample complexity of DSBO scale with the optimality gap and network size?
(ii) How is the efﬁciency of DSBO affected by the network structure?
In this paper, we develop a gossip-based stochastic approximation scheme where each agent solvesan optimization problem collaboratively by sampling stochastic ﬁrst- and second-order informationusing its data and making gossip communications with its neighbors.
In addition, we develop novelconvergence analysis techniques to characterize the convergence behavior of our algorithm.
To thebest of our knowledge, our work is the ﬁrst to formulate DSBO mathematically and propose analgorithm with theoretical convergence guarantees.
Moreover, we show that our algorithm enjoysan O
( 1 K ✏ 2 ) sample complexity for ﬁnding ✏ -stationary points for nonconvex objectives and enjoysan O ( 1 K ✏ ) sample complexity for Polyak- Ł ojasiewicz (PL) functions, subsuming strongly convexoptimization.
These results subsume the state-of-the-art results for non-federated stochastic bileveloptimization (Chen et al., 2021c) and central-server regime (Tarzanagh et al., 2022), showing thatnearly no degradation is induced by network consensus.
Further, the above results suggest that ouralgorithm exhibits the linear speed-up effect for decentralized settings; that is, the required per-agentsample complexities decrease linearly to the number of agents.
Related Works
Bilevel optimization was ﬁrst formulated by (Bracken and McGill, 1973) for solving resourceallocation problems.
Later, a class of constrained-based algorithms was proposed by (Hansenet al., 1992; Shi et al., 2005), which treats the inner-level optimality condition as constraints tothe out-level problem.
Recently, (Couellan and Wang, 2016) examined the ﬁnite-sum case forunconstrained strongly convex lower-lower problems and proposed a gradient-based algorithm thatexhibits asymptotic convergence under certain step-sizes.
For SBO, (Ghadimi and Wang, 2018)developed a double-loop algorithm and established the ﬁrst known complexity results. Subsequently,
Stochastic
Optimization
BSA stocBio
ALSET
FEDNEST
This Work
Network
Single-Agent
Central-Server
Decentralized
Samples in ⇣
O ( ✏  2 ) O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 + K (1  ⇢ ) 2 ✏ )
Samples in ⇠
O ( ✏  3 )
O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 + K (1  ⇢ ) 2 ✏ )
Stochastic Compositional Optimization
SCGD
NASA
ALSET
FEDNEST
This Work
Network
Single-Agent
Central-Server
Decentralized
Samples in ⇣
O ( ✏  4 )
O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 ) O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 + K (1  ⇢ ) 2 ✏ )
Samples in ⇠
O ( ✏  4 )
O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 )
O ( 1 K ✏ 2 ) O ( 1 K ✏ 2 +
K (1  ⇢ ) 2 ✏ ) various methods have been employed to improve the sample complexity, including two-timescalestochastic approximation (Hong et al., 2020), acceleration (Chen et al., 2021a), momentum (Khanduriet al., 2021), and variance reduction (Guo et al., 2021;
Ji et al., 2021; Yang et al., 2021).
Distributed optimization was developed to handle real-world large-scale datasets (Dekel et al., 2012;
Feyzmahdavian et al., 2016) and graph estimation (Wang et al., 2015).
Centralized and decentralizedsystems are two important problems that have drawn signiﬁcant attention.
A centralized systemconsiders the network topology where there exists a central agent that communicates with theremaining agents (Lan and Zhou, 2018), while in a decentralized system (Gao and Huang, 2020; Koloskova et al., 2019; Lan et al., 2020; McMahan et al., 2017), each agent can only communicatewith its neighbors by using gossip (Lian et al., 2017b) or gradient tracking (Pu and Nedi´c, 2021)communication strategies.
Variance reduction approaches (Xin et al., 2020, 2021; Lian et al. , 2017a)have also been applied to improve the convergence rate of decentralized optimization.
Randomprojection schemes have been studied to handle large sets of constraints (Wang and Bertsekas, 2016;
Liu et al., 2015).
All of the above trials were made on vanilla stochastic optimization problems.
We are the ﬁrst to study the decentralized bilevel optimization problem to our knowledge.
Table 1compares this work with prior arts under different settings.
Problem Setup
Assumption 3.1 (Sampling Oracle SO )
Agent k may query the sampler and receive an independent locally sampled unbiased ﬁrst- and second-order information r x f k ( x, y ; ⇣ k ) , r y f k ( x, y ; ⇠ k ) ,r y g k {+( x, y ; ⇠ k ) , r 
+}
yy g k ( x, y ; ⇠ k ) , and r 2 xy g k ( x, y ; ⇠ k ) .
Assumption 3.2 (Gossip Protocol)
The network gossip protocol is speciﬁed by a K ⇥ K symmetricmatrix W with nonnegative entries.
Each agent k may receive information from its neighbors, e.g.,z j , j 2 N k , and aggregate them by a weighted sum
[Equation]
w k j z j .
Further, matrix W satisﬁes
(i) W is doubly stochastic such that P
i w i,j = 1 and P
j w i,j = 1 for all i, j 2 [ K ] .
(ii) There exists a constant ⇢ 2 (0 , 1) such that k W  1 K 11 > k 22 = ⇢ , where k A k 2 denotes thespectral norm of A 2 R K ⇥ K .
These assumptions on the adjacency matrix are crucial to ensure the convergence of decentralizedalgorithms, and are commonly made in the decentralized optimization literature (Lian et al., 2017b).
Assumption 3.3 (Smoothness and Convexity) We impose the following smoothness assumptions: (i) r x f k ( x, y ) and r y f k ( • , y ) are L f -Lipschitz continuous in ( x, y ) . (ii) r
{+y g k ( x, y ) , r 2 yx g k ( x, y ) , and r 
 yy g k ( x, y ) are L g -Lipschitz continuous in ( x, y ) . input: Step-sizes { ↵ t } , {  t } , {  t } , total iterations T , adjacency matrix W , smoothness constant L g , hessian sampling parameter b+} x k 0 = 0 , y k
= 0 , u k 0 = 0 , s k 0 = 0 , h t 0 = 0 , v k 0 ,i = µ g I for i = 1 , 2 , · · · , b 1: for t = 0 , 1 , · · · , T  1 do 2: for k = 1 , · · · , K do 3:
Local sampling: Query SO at ( x kt , y kt ) to obtain r x f k ( x kt , y kt ; ⇣ kt ) , r y f k ( x kt , y kt ; ⇣ kt ) , r y g k ( x kt , y kt ; ⇠ kt ) , r 2 xy g k
( x kt , y kt ; ⇠ kt ) , and { r {+
+} yy g k
( x kt , y kt ; ⇠ kt,i ) } bi =1 .
Outer loop update: x kt +1 =
P j 2 N k w k,j x jt  ↵ t  s kt  u kt q kt h kt  .
Inner loop update: y kt
+1 =
P j 2 N k w k,j y jt   t r y g k ( x kt , y kt ; ⇠ kt ) .
Estimate r x f ( x t , y t ) : s kt
+
=
(1   t )
P j 2 N k w k,j s jt +  t r x f k ( x kt , y kt ; ⇣ kt ) .
Estimate r y f ( x t , y t ) : h kt +
=
(1   t )
P j 2 N k w k,j h jt +  t r y f k ( x kt , y kt ; ⇣ kt ) .
Estimate r 2 xy g ( x t , y t ) : u[-kt +
-]
[Equation]
{+Estimate [ r 2 yy g ( x t , y t )]  1 : Set Q kt +1 , 
+}
=
I 10: for i = 1 , · · · , b do 11: v
[Equation] +1 , [Equation]
[-Q pi =
-]= [Equation]
[-1] . Update v kt +
-]
[Equation]
Q [Equation]
13: end for 14:
{+Set q kt +1 = 
+}
L g [-k x, y ) , and r 
-]Q kt +1 ,b 15: end for 16: end for
We defer the detailed assumptions to Section A.1 of the supplement.
Here we point out that the above assumptions allow the heterogeneity between functions f k ’s andg k ’s over the agents, and the smoothness and boundedness conditions are commonly adopted in SBO(Hong et al., 2020; Chen et al., 2021a).
Algorithm
As discussed in Section 1.2, the key challenge in solving DSBO is that each agent has only accessto its own data but is required to construct estimators for the gradients and Hessian averaged acrossall agents.
It is particularly challenging to construct such estimators when limited by the network’scommunication protocol.
To overcome this issue, we propose a gossip-based DSBO algorithm where each agent k 2 Kiteratively updates a solution pair ( x kt , y kt ) by using the combination of gossip communications andweighted-average stochastic approximation, where y kt serves as an estimator of the best responsey ⇤ t :=
y ⇤ (¯ x t ) with ¯ x t := 1 K
P k 2 K x kt .
The full algorithm is given in Alg. 1.
Here we brieﬂy explain the idea of our DSBO algorithm.
Suppose agent k would like to estimater x f ( x kt , y kt ) by s kt , under Alg.
1 Step 6, it would query the stochastic ﬁrst-order information usingits own data, make gossip communications with neighbors, and update its estimators by taking theweighted average among its previous estimate s kt  1 , neighbors j ’s estimate s jt  1 , and the newlysampled gradient r x f k ( x kt , y kt ; ⇣ kt ) .
Roughly speaking, this procedure can be viewed as takingweighted average among gradients sampled by all agents over the network, except that the effect ofconsensus should also been taken into account.
The updates for r y f and r 2 xy g are conducted in a similar manner (Steps 7 & 8), but it takes extra efforts to evaluate [ r 2 yy g  1 , due to the absence of its unbiased estimator.
To be speciﬁc, we note that 1 K [Equation] 1 , making the unbiased estimator of the desired term unavailable even if each agent has an unbiased estimator for [ r 2 yy g k ]  1 . This is indeed a unique challenge for decentralized bilevel optimization.
To overcome this issue, we propose a novel approach that each agent k constructs b independent estimators { v kt,j } bj =1 for r {+
+} yy g ( x kt , y kt ) via consensus and stochastic approximation.
We then estimate r {+
 yy g (+} x kt , y kt ) by utilizing the following approximation. [Equation] We provide details in Steps 4 & 9.
Finally, by utilizing the estimators obtained in the above procedure, each agent would compute thegradient (3) and update the outer solution x kt by using the combination of gossip communicationand stochastic gradient descent (Step 3).
The inner loop solution y kt can be updated by (Step 4) in asimilar way.
We highlight the following key features of our DSBO algorithm: (1) each agent would only communicate their estimates instead of the raw data in the gossip-communication process, which preservesdata privacy.
(2) agent k would make O ( |N k | ) communications with its neighbors in each round, 168 which is much smaller than the total number of agents in a naive approach.
(3) the algorithm is robustto contingencies in the network.
If some communication channel fails, the agents can still jointlylearn provided that the network is still connected.
In contrast, a single-center-multi-user networkwould fail completely in case of a center failure.
Theory
In this section, we analyze the performance of our DSBO algorithm for both nonconvex and µ -PLobjectives, and derive the convergence rates for both cases.
5.1 Nonconvex Objectives
We ﬁrst consider the scenario where overall objective function F ( x ) is nonconvex.
For nonconvexobjectives, given the total number of iterations T , we employ the step-sizes in a constant form that
[Equation]
⇥
[Equation]
) for all t [Equation] [Equation]
[Equation]
T
[Equation]
where C 0 > 0 is a small constant.
Compounded effect of consensus and SBO: As discussed earlier, to derive the convergence rate of SBO under decentralized federated setting, the key step is to quantify the compounded effect betweenthe consensus errors induced by the network structure and the biases induced by estimating gradientwithin (3).
Distinct from the central-server or non-federated regimes, the consensus errors induced bythe decentralized network structure has to be handled carefully.
We conduct careful analysis to derivethe contraction of consensus errors, and further show that both bias and variance of the averagedestimator diminish to zero, establishing a nontrivial convergence argument to the desired gradientand Hessian.
In particular, the estimators preserve a concentration property so that their variancesdecrease proportionally to 1 /K , suggesting that the network consensus effect would not degrade theconcentration of the generated stochastic samples.
To achieve the best possible convergence rate,we carefully set the algorithm parameters, including step-sizes ↵ t ,  t and averaging weights  t , tocontrol the above consensus errors and biases.
We provide the convergence rate of Alg. 1 in the following theorem and defer the detailed proof to
Section B.7 of the supplementary material.
Theorem 5.1 Suppose Assumptions 3.1, 3.2, and 3.3 hold.
Letting ¯ x t = 1 K P k 2 K x kt , then
[Equation]
More details and proof are deferred to Section B.7 of the supplement.
Effect of consensus: In this result, the O ( K T (1  ⇢ ) 2 ) term represents the errors induced by consensusof network.
Despite the dependency with the network structure, such term diminishes to zero in the order of O (1 /T ) , becoming a small order term when T is large.
Consequently, given the networktopology, the asymptotic convergence behavior of DSBO is independent of the network structure,answering the question (ii) raised in Section 1.
Linear speedup: Because each agent queries O (1) stochastic samples per round, clearly therequired iteration and per-agent sample complexities for ﬁnding an ✏ -stationary point such that1 T
{+P T  1 t =
+}
E [ kr F (¯ x t ) k 2 ]  ✏ are both O ( 1 K ✏ 2 ) .
Such result implies that our algorithm achieves alinear speed-up effect proportionate to the number of agents K .
In other words, in the presence ofmore agents, each agent needs to query fewer stochastic samples to achieve a speciﬁed accuracy.
Meanwhile, our rate also matches the best-known O (1 /K ✏ 2 ) iteration and per-agent
sample complexities under the decentralized vanilla stochastic gradient descent settings (Lian et al., 2018).
Toour knowledge, it is the ﬁrst time such a result has been established for DSBO problems.
Single-center-multi-user-federated SBO :
We point out that a simpliﬁed version of our algorithmsolves single-center-multi-user-federated SBO, where the central server directly collects informationfrom each agent and calculates the gradient by employing the weighted-average stochastic approximation scheme among the collected information.
In such scenario, the agents no longer make gossipcommunication with neighbors but synchronously receives a common solution from the central server,so that the consensus effect would disappear.
In Theorem D.1, we show that a variant of our algorithmcan indeed achieve the same O ( 1 p KT ) convergence rate in this setting.
5.2 µ -PL Objectives
Next we study the case where the objective function satisﬁes the following µ -PL condition.
Assumption 5.2 There exists a constant µ > 0 such that the objective satisﬁes the PL condition:
[Equation]
Note that the class of strongly convex functions is a special case of µ -PL functions.
To utilize theµ -PL property and achieve fast convergence, unlike the nonconvex case (5) where the step-sizes areset as constants depending on the total number of iterations T , we employ step-sizes in a diminishingform that↵
[Equation]
t, and b = ⇥ (log( T )) for t  1 , (6) where C 1 > 0 is a large constant and b = .
By following an analysis routine similar to that in thenonconvex scenario, in the next result, we derive the convergence rate of Alg. 1 for µ -PL objectives.
Theorem 5.3 Suppose Assumptions 3.1, 3.2, and 3.3 hold and the function satisﬁes the µ -PL Assumption 5.2.
Letting ¯ x T = 1 K
[Equation]
kT , then
[Equation]
Both iteration and per-agent sample complexities for ﬁnding an ✏ -optimal point E [ F (¯ x T )]
F ⇤  ✏are O ( 1 K ✏ ) .
Details and proof are deferred to Section C.2 of the supplement.
This result shows that our algorithmachieves a faster convergence rate for functions satisfying the µ -PL condition in terms of bothiteration and sample complexities.
First, similar as the nonconvex scenario, the consensus errordecays in the order of ˜ O
( 1 T 2 (1  ⇢ ) 2 ) .
Dominated by O ( 1 KT ) , such consensus decaying order indicatesthat the network structure would not affect Alg.
1’s asymptotic convergence behavior under µ -PLobjectives.
Meanwhile, the above result implies that Alg.
1 speeds up linearly with the number ofagents and matches the optimal O (1 / ✏ ) sample complexity for single-sever vanilla strongly-convexstochastic optimization (Rakhlin et al., 2011).
As a result, our algorithm achieves the optimal samplecomplexities for decentralized stochastic bilevel optimization, establishing the benchmark.
Numerical Experiments
In this section, we validate the practical performance of our algorithm in two examples: hyperparameter optimization and policy evaluation in Markov Decision Processes (MDP), on artiﬁcially [Figure] [Figure] constructed decentralized ring networks.
We run the experiments on a single server desktop computer.
We provide the details of federate hyper-parameter optimization and federated policy evaluation.
Hyper-parameter Optimization We consider federated hyper-parameter optimization (2) for ahandwriting recognition problem over the Australia handwriting dataset (Chang and Lin, 2011)consisting of data points ( w i , z i ) , where w i 2 R 14 is the feature and z i 2 { 0 , 1 } represents whether 245 this data point belongs to category “1” or not.
In our experiment, we consider the sigmoid lossfunction that l i ( z )
= 1 / (1 + exp
(  z )) and a strongly convex regularizer R ( x, y ) =
P di =1 x i 2 k y i k 2 .
247 We consider a ring network of K agents where each agent i preserves two neighbors ( i  1) and ( i +1)and conduct gossip communication strategy with adjacency matrix w i,j = 13 for j 2 { i  1 , i, i + 1 } .
Before testing Alg.
1, we ﬁrst randomly split the dataset for training and validation, and then allocatesboth training and validation dataset over K agents.
We then run Algorithm 1 for T = 20000 iterations,with b = 200 , ↵ t = 0 .p
K/T , and  t =
t =p
K/T .
For benchmark comparison, we implement a baseline algorithm Decentralized Bilevel Stochastic
Approximation (DBSA), a naive extension of the double-loop BSA algorithm
(Ghadimi and Wang,2018) in the decentralized setting, formally stated in Section E.1 of the supplementary materials.
We ﬁrst consider K = 5 , test Alg.
1 for 5 ⇥ 10 4 iterations, and compare its performance with DBSA.
We report the validation loss against total samples in Figure 2 and observe that DSBO exhibitssuperior performance to DBSA.
In particular, Further, we observe that our algorithm outperforms thebaseline algorithm DBSA that it requires fewer samples for DSBO to achieve a certain accuracy.
To investigate the efﬁciency of Alg. 1 to the network structure, we test Alg.
1 over K = 5 , 10 , 20 ,and report the details of training and validation loss in Figure 2.
Further, comparing the performancesof Alg. 1 over different agents K = 5 , 10 , 20 , we observe that Alg.
1 converges faster by usingmore agents.
This observation suggests that Alg.
1 exhibits a speed-up effect by using more agents.
We provide additional experiments on networks of larger size ( K = 100 ) and various topologies(fully-connected and randomly-connected) in Section E.1 of the supplement.
Distributed policy evaluation for reinforcement learning We consider a multi-agent MDPproblem arising in reinforcement learning.
Let S be the state space, for any state s 2 S , we denote by 267 V ( s ) the value function.
we consider the scenario where the value function can be approximated by alinear function such that V ( s ) =
> s x ⇤ , where  s 2 R m is a feature and x ⇤
R m is an unknownparameter.
To obtain the optimal x
⇤ , we consider the following regularized Bellman minimizationproblem
[Equation]
where r ( s, s 0 ) is the random reward incurred from transition s to s 0 ,  2 (0 , 1) is the discount factor, is the coefﬁcient for the l 2 -regularizer, and the expectation is taken over all random transitions froms to s 0 .
In the federated learning setting, we consider a ring network of K agents.
Here each agent k hasaccess to its own data with heterogeneous random reward function r k and can only communicate with
[-k +
-]2.5 10 - (a)    9 - - - - - (b)   208.
(c) its two neighbors k + 1 and k  1 .
We denote by y ⇤ s ( x ) =
> s  E s
[ 1 K
[Equation]
where r k ( s, s 0 ) is the random reward function for agent k , the above problem can be recast as abilevel optimization problem
[Equation]
As pointed out by (Wang and Liu, 2016a), the above problem is  -strongly convex.
In our experiments, we simulate an environment with state space |S| = 100 and set the regularizer 281 parameter  = 1 .
We test the performance of Alg.1 over three setups with K = 5 , 10 , 20 and conduct10 independent simulations for each K .
We implement a baseline double-loop algorithm DSGD thatﬁrst estimates y ⇤ s ( x t ) with t samples in iteration t and then optimizes the solution x t .
We defer theimplementation details of the environment and above algorithms to Section E.2 of the supplement.
We ﬁrst consider K = 5 , run Alg. 1 for 10 4 iterations and compare its performance with DSGD.
Weplot the empirical averaged mean square error
k ¯ x t  x ⇤ k 2 against total samples generated by all agentsin Figure 2.
This empirical result suggests that Alg.
1 outperforms the DSGD.
To investigate theconvergence rate of DSBO, we compare the performance of DSBO over all three setups K = 5 , 10 , 20and plot the trajectory of averaged log-error log( k ¯ x t  x
⇤ k 2 ) averaged, with one straight line of slope-1 provided for comparison.
We observe that over all four setups, the slopes of log( k ¯ x t  x ⇤ k 2 ) areclose to -1, matching our theoretical claim in Theorem 5.
that Alg.
1 converges at the rate of O (1 /t )for strongly convex objectives.
In the above experiment, we also note that Alg.
1 converges faster when using more agents.
Tofurther demonstrate the linear speedup effect, we compute the total generated samples for ﬁnding an✏ -optimal solution k ¯ x t  x
⇤ k 2  ✏ and plot the 75% conﬁdence region of log-sample against number 296 of agents K = 5 , 10 , 20 in Figure 3.
We observe that it takes a roughly same amount of samplesto ﬁnd a 10  6 -optimal solution despite different number of agents are involved.
This suggests thatthe per-node sample complexity decreases linearly to K and validates the linear speedup claim in
Theorem 5.3.
We provide additional numerical results for other optimality level ✏ in Section E.2 ofthe supplementary material to further demonstrate the linear speedup effect.
Conclusion
In this paper, we propose a novel formulation of decentralized stochastic bilevel optimization.
Wedevelop a gossip-based stochastic approximation scheme to solve this problem in various settings.
We show that our proposed algorithm ﬁnds a stationary point at a rate of O ( 1 p KT ) for nonconvexobjectives, and converges to the optimal solution at a rate of O ( 1 KT ) for strongly convex objectives.
Numerical experiments on hyper-parameter optimization and multi-agent federated MDP demonstratethe practical efﬁciency of our algorithm, exhibit the effect of speed-up in decentralized setting, andvalidate our theoretical claims.
In our future work, we wish to develop algorithms that achieve loweriteration complexities and enjoy lower communication cost.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
This is analgorithmic and theoretical work justiﬁed by numerical experiments.
We are not awareof any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Complete proofs are
provided in the supplement.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We will providethe code via github in the camera-ready version stage.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We provided averaged results over different random seeds
and the obtained results are stable.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
The dataset is publicly available and
no license is required.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We are not releasing new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
We cited the website link where data is presented.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No] The data does not contain personally identiﬁable
information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
We did not involve human subjects.
[Table]