Decentralized Gossip-Based Stochastic Bilevel Optimization over Communication Networks
Abstract
Bilevel optimization have gained growing interests, with numerous applications found in meta learning, minimax games, reinforcement learning, and nested composition optimization.
This paper studies the problem of distributed bilevel optimization over a network where agents can only communicate with neighbors, including examples from multi-task, multi-agent learning and federated learning.
In this paper, we propose a gossip-based distributed bilevel learning algorithm that allows networked agents to solve both the inner and outer optimization problems in a single timescale and share information via network propagation.
We show that our algorithm enjoys the O ( 1 K ✏ 2 ) per-agent sample complexity for general nonconvex bilevel optimization and O ( 1 K ✏ ) for strongly convex objective, achieving a speedup that scales linearly with the network size.
The sample complexities are optimal in both ✏ and K .
We test our algorithm on the examples of hyperparameter tuning and decentralized reinforcement learning.
Simulated experiments conﬁrmed that our algorithm achieves the state-of-the-art training efﬁciency and test accuracy.
Introduction
In recent years, stochastic bilevel optimization (SBO) has attracted increasing attention from the machine learning community.
It has been found to provide favorable solutions in a variety of problems, such as meta learning and hyperparameter optimization (Franceschi et al., 2018;
Snell et al., 2017; Bertinetto et al., 2018), composition optimization (Wang and Liu, 2016), two-player games (Von Stackelberg and Von, 1952), reinforcement learning and imitation learning (Arora et al., 2020;
Hong et al., 2020).
While the majority of the above mentioned work focuses on algorithm designs in the classic centralized setting, such problems often arise in distributed/federated applications, where agents are not willing to share data but rather perform local updates and communicate with neighbors.
Theories and algorithms for distributed stochastic bilevel optimization are less developed.
Consider the decentralized learning setting where the data are distributed over K agents K = { 1 , 2 , · · · , K } over a communication network.
Each agent can only communicate with its neighbors.
One example is federated learning which often concerns with a single-server-multi-user system, where agents communicate with a central server to cooperatively solve a task (Lan and Zhou, 2018; Ge et al., 2018). Another example is sensor network, where sensors are fully decentralized and can only communicate with nearby neighbors (Taj and Cavallaro, 2011) . We consider the following decentralized stochastic bilevel optimization problem,
[Equation]
where f k ( x, y ) =
E ⇣ k
[ f k ( x, y ; ⇣ k )] and g k ( x, y ) =
E ⇠ k [ g k ( x, y ; ⇠ k )] may vary across agents.
The expectations E ⇣ k
[ • ] and E ⇠ k [ • ] are taken with respect to the random variables ⇣ k and ⇠ k , [Figure] with heterogeneous distributions across agents.
We consider the scenario where each g k ( x, y ) is strongly convex in y .
We denote by F ⇤ =
min x 2 R dx F ( x ) , f ( x, y )
= 1 K
P k 2 K f k
( x, y ) , and g ( x, y )
= 1 K
P k 2 K g k
( x, y ) for notational convenience.
1.1 Example applications of SBO
SBO was ﬁrst employed to formulate the resource allocation problem (Bracken and McGill, 1973) and has since found its application in many classic operations research settings (Cramer et al., 1994;
Sobieszczanski-Sobieski and Haftka, 1997; Livne, 1999; Tu et al., 2020; Tu, 2021), and more recently in machine learning problems (Franceschi et al., 2018;
Snell et al., 2017; Bertinetto et al., 2018;
Wang and Liu, 2016).
In particular, we introduce two applications that have received a lot of recent attention, namely hyperparameter optimization and compositional optimization.
Hyperparameter Optimization
The problem of hyper-parameter tuning (Okuno et al., 2021) often takes the following form:
[Equation]
where D train and D val are two datasets used for training and validation, respectively,
y 2 R d y is a vector of unknown parameters to optimize, ` j ( y ) is a convex loss over data i , and x 2 R d is a vector of hyper-parameters for a strongly convex regularizer R ( x, y ) .
For any hyper-parameter x 2 R d , the inner-level problem solves for the best parameter y ⇤ ( x ) over the training set D train under the regularized training loss ` j ( y )
+ R ( x, y ) .
The goal is to ﬁnd the hyper-parameter x
⇤ 2 R d whose corresponding best response y ⇤ ( x ) yields the least loss over the validation set D val .
In practice, tuning continuous hyperparameters is often done by grid search which is exponentially expensive.
Efﬁcient algorithm of SBO should ﬁnd optimal parameters in time polynomial to dimension, rather than exponential.
When the training and validation set are distributed across nodes, the problem becomes a distributed SBO.
Compositional Optimization Let g ( x, ⇠ ) : R d !
R d y and f ( y, ⇣ ) :
R d y !
R be two stochastic mappings, stochastic compositional optimization (SCO) (Wang et al., 2017a) takes the form
[Equation]
SCO applies to risk management (Yang et al., 2019; Ruszczynski, 2021), machine learning (Chen et al., 2021b), and reinforcement learning (Wang et al., 2017b).
SCO was identiﬁed as a special case of SBO by (Chen et al., 2021c).
To see this, we take the inner optimization objective to be g ( x, y ) =
E ⇠ [( y  h ( x, ⇠ )) 2 ] .
Thus y ⇤ ( x ) = argmin y g ( x, y ) becomes y ⇤
( x ) =
E ⇠ [ h ( x ; ⇠ )] , arriving at an instance of SBO.
1.2 Challenges with Distributed SBO
Despite the recent fast development of distributed single-level optimization, a proper method for
Decentralized Stochastic Bilevel Optimization (DSBO) remains illusive.
The major hurdle of solving
DSBO lies in the absence of explicit knowledge of y ⇤ ( x ) , so that an unbiased gradient of r f ( x, y ⇤ ( x )) is unavailable.
Recently, by applying the implicit function theorem, (Couellan and Wang, 2016;
Ghadimi and Wang, 2018) showed that the gradient of non-distributed SBO can be expressed as
[Equation]
providing a connection between SBO and classical stochastic optimization.
Since then, various algorithms have been proposed to obtain sharp estimators for y ⇤ ( x ) and reduce the bias of the constructed gradients (Chen et al., 2021a;
Hong et al., 2020; Ji et al., 2021;
Yang et al., 2021).
These techniques result in nearly tight convergence analysis and give rise to algorithms widely used in applications.
However, none of the prior algorithms can be applied to the distributed setting.
Solving Problem (1) becomes challenging in the distributed setting in several aspects:
Even in single-agent SBO, lack of y ⇤ ( x ) makes outer optimization nontrivial, and estimating y ⇤ ( x ) requires additional stochastic approximation, weighted averaging, and sophisticated calculation.
Calculating outer gradient is highly nontrivial, even when we have an inner solution y .
Note that
[Equation]
In other words, even the inner problem is solved, the outer gradient requires new mechanism to estimate.
Now we move to SBO over distributed networks.
In network learning, communication among agents can be limited by the network stucture and communication protocol, so taking a simple average across agents may require multiple rounds of communication.
Because of the above hurdles, it remains unclear how to sharply estimate the outer gradient under a distributed network.
As an attempt to tackle this problem, this paper study the convergence theory and sample complexity of gossip-based algorithm.
In particular, we ask two theoretical questions:
(i) How does the sample complexity of DSBO scale with the optimality gap and network size?
(ii) How is the efﬁciency of DSBO affected by the network structure?
In this paper, we develop a gossip-based stochastic approximation scheme where each agent solves an optimization problem collaboratively by sampling stochastic ﬁrst- and second-order information using its data and making gossip communications with its neighbors.
In addition, we develop novel convergence analysis techniques to characterize the convergence behavior of our algorithm.
To the best of our knowledge, our work is the ﬁrst to formulate DSBO mathematically and propose an algorithm with theoretical convergence guarantees.
Moreover, we show that our algorithm enjoys an O ( 1 K ✏ 2 ) sample complexity for ﬁnding ✏ -stationary points for nonconvex objectives and enjoys an O ( 1 K ✏ ) sample complexity for Polyak- Ł ojasiewicz (PL) functions, subsuming strongly convex optimization.
These results subsume the state-of-the-art results for non-federated stochastic bilevel optimization (Chen et al., 2021c) and central-server regime (Tarzanagh et al., 2022), showing that nearly no degradation is induced by network consensus.
Further, the above results suggest that our algorithm exhibits the linear speed-up effect for decentralized settings; that is, the required per-agent sample complexities decrease linearly to the number of agents.
Related Works
Bilevel optimization was ﬁrst formulated by (Bracken and McGill, 1973) for solving resource allocation problems.
Later, a class of constrained-based algorithms was proposed by (Hansen et al., 1992; Shi et al., 2005), which treats the inner-level optimality condition as constraints to the out-level problem.
Recently, (Couellan and Wang, 2016) examined the ﬁnite-sum case for unconstrained strongly convex lower-lower problems and proposed a gradient-based algorithm that exhibits asymptotic convergence under certain step-sizes.
For SBO, (Ghadimi and Wang, 2018) developed a double-loop algorithm and established the ﬁrst known complexity results.
Subsequently,
Stochastic
Optimization
BSA stocBio
ALSET
FEDNEST
This Work
Network
Single-Agent
Central-Server
Decentralized
Samples in ⇣
O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 ) O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 +
K (1  ⇢ ) 2 ✏ )
Samples in ⇠
O ( ✏  3 )
O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 +
K (1  ⇢ ) 2 ✏ )
Stochastic Compositional Optimization
SCGD
NASA
ALSET
FEDNEST
This Work
Network
Single-Agent
Central-Server
Decentralized
Samples in ⇣
O ( ✏  4 )
O ( ✏  2 )
O ( ✏  2 ) O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 +
K (1  ⇢ ) 2 ✏ )
Samples in ⇠
O ( ✏  4 )
O ( ✏  2 )
O ( ✏  2 ) O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 +
K (1  ⇢ ) 2 ✏ ) various methods have been employed to improve the sample complexity, including two-timescale stochastic approximation (Hong et al., 2020), acceleration (Chen et al., 2021a), momentum (Khanduri et al., 2021), and variance reduction (Guo et al., 2021; Ji et al., 2021;
Yang et al., 2021).
Distributed optimization was developed to handle real-world large-scale datasets (Dekel et al., 2012;
Feyzmahdavian et al., 2016) and graph estimation (Wang et al., 2015).
Centralized and decentralized systems are two important problems that have drawn signiﬁcant attention.
A centralized system considers the network topology where there exists a central agent that communicates with the remaining agents (Lan and Zhou, 2018), while in a decentralized system (Gao and Huang, 2020;
Koloskova et al., 2019; Lan et al., 2020; McMahan et al., 2017), each agent can only communicate with its neighbors by using gossip (Lian et al., 2017b) or gradient tracking (Pu and Nedi´c, 2021) communication strategies, with applications in multi-agent reinforcement learning (Xu et al., 2020).
Variance reduction approaches (Xin et al., 2020, 2021; Lian et al., 2017a) have also been applied to improve the convergence rate of decentralized optimization.
Random projection schemes have been studied to handle large sets of constraints (Wang and Bertsekas, 2015, 2016;
Liu et al., 2015).
All of the above trials were made on vanilla stochastic optimization problems.
We are the ﬁrst to study the decentralized bilevel optimization problem to our knowledge.
Table 1 compares this work with prior arts under different settings.
Problem Setup
Assumption 3.1 (Sampling Oracle SO )
Agent k may query the sampler and receive an independent locally sampled unbiased ﬁrst- and second-order information r x f k ( x, y ; ⇣ k ) , r y f k ( x, y ; ⇠ k ) , r y g k ( x, y ; ⇠ k ) , r
yy g k ( x, y ; ⇠ k ) , and r 2 xy g k
( x, y ; ⇠ k ) .
Assumption 3.2 (Gossip Protocol)
The network gossip protocol is speciﬁed by a K ⇥ K symmetric matrix W with nonnegative entries.
Each agent k may receive information from its neighbors, e.g., z j , j 2 N k , and aggregate them by a weighted sum
[Equation]
j z j .
Further, matrix W satisﬁes
(i) W is doubly stochastic such that P
i w i,j = 1 and P
j w i,j = 1 for all i, j 2 [ K ] .
(ii) There exists a constant ⇢ 2 (0 , 1) such that k W  1 K 11 > k 22 = ⇢ , where k A k 2 denotes the spectral norm of A 2 R K ⇥ K .
These assumptions on the adjacency matrix are crucial to ensure the convergence of decentralized algorithms, and are commonly made in the decentralized optimization literature (Lian et al., 2017b).
Assumption 3.3 (Smoothness and Convexity)
We impose the following smoothness assumptions:
(i) r x f k ( x, y ) and r y f k ( • , y ) are L f -Lipschitz continuous in ( x, y ) .
(ii) r
y g k ( x, y ) , r 2 yx g k
( x, y ) , and r
yy g k ( x, y ) are L g -Lipschitz continuous in ( x, y ) .
input: Step-sizes { ↵ t } , {  t } , {  t } , total iterations T , adjacency matrix W , smoothness constant L g , hessian sampling parameter b x k 0 = 0 , y k
= 0 , u k 0 = 0 , s k 0 = 0 , h t 0 = 0 , v k 0 ,i = µ g I for i
= 1 , 2 , · · · , b 1: for t = 0 , 1 , · · · , T  1 do 2: for k = 1 , · · · , K do 3:
Local sampling: Query SO at ( x kt , y kt ) to obtain r x f k ( x kt , y kt ; ⇣ kt ) , r y f k ( x kt , y kt ; ⇣ kt ) , r y g k ( x kt , y kt ; ⇠ kt ) , r 2 xy g k
( x kt , y kt ; ⇠ kt ) , and { r
yy g k
( x kt , y kt ; ⇠ kt,i ) } bi =1 .
Outer loop update: x kt +1 =
P j 2 N k w k,j x jt  ↵ t  s kt  u kt q kt h kt  .
Inner loop update: y kt
+1 =
P j 2 N k w k,j y jt   t r y g k ( x kt , y kt ; ⇠ kt ) .
Estimate r x f ( x t , y t ) : s kt
+
=
(1   t )
P j 2 N k w k,j s jt +  t r x f k ( x kt , y kt ; ⇣ kt ) .
Estimate r y f ( x t , y t ) : h kt +
=
(1   t )
P j 2 N k w k,j h jt +  t r y f k ( x kt , y kt ; ⇣ kt ) .
Estimate r 2 xy g ( x t , y t ) : u
[Equation]
Estimate [ r 2 yy g ( x t , y t )]  1 : Set Q kt +1 ,
=
I 10: for i = 1 , · · · , b do 11: v
[Equation]
+1 ,
[Equation]
[Equation]
Q
[Equation]
13: end for 14:
Set q kt {++
+} =[-
-]
g Q kt +1 , 15: end for 16: end for
We defer the detailed assumptions to Section A.1 of the supplement.
Here we point out that the above assumptions allow the heterogeneity between functions f k ’s and g k ’s over the agents, and the smoothness and boundedness conditions are commonly adopted in SBO (Hong et al., 2020; Chen et al., 2021a).
Algorithm
As discussed in Section 1.2, the key challenge in solving DSBO is that each agent has only access to its own data but is required to construct estimators for the gradients and Hessian averaged across all agents.
It is particularly challenging to construct such estimators when limited by the network’s communication protocol.
To overcome this issue, we propose a gossip-based DSBO algorithm where each agent k 2 K iteratively updates a solution pair ( x kt , y kt ) by using the combination of gossip communications and weighted-average stochastic approximation, where y kt serves as an estimator of the best response y ⇤ t :=
y ⇤ (¯ x t ) with ¯ x t := 1 K
P [Equation]
The full algorithm is given in Alg. 1.
Here we brieﬂy explain the idea of our DSBO algorithm.
Suppose agent k would like to estimate r x f ( x kt , y kt ) by s kt , under Alg.
1 Step 6, it would query the stochastic ﬁrst-order information using its own data, make gossip communications with neighbors, and update its estimators by taking the weighted average among its previous estimate s kt  1 , neighbors j ’s estimate s jt  1 , and the newly sampled gradient r x f k ( x kt , y kt ; ⇣ kt ) .
Roughly speaking, this procedure can be viewed as taking weighted average among gradients sampled by all agents over the network, except that the effect of consensus should also been taken into account.
The updates for r y f and r 2 xy g are conducted in a similar manner (Steps 7 & 8), but it takes extra efforts to evaluate [ r 2 yy g
1 , due to the absence of its unbiased estimator.
To be speciﬁc, we note that 1 K
[Equation]
[Equation] 1 , making the unbiased estimator of the desired term unavailable even if each agent has an unbiased estimator for [ r 2 yy g k ]  1 .
This is indeed a unique challenge for decentralized bilevel optimization.
To overcome this issue, we propose a novel approach that each agent k constructs b independent estimators { v kt,j } bj =1 for r
yy g ( x kt , y kt ) via consensus and stochastic approximation.
We then estimate r
yy g ( x kt , y kt ) by utilizing the following approximation.
[Equation]
We provide details in Steps 4 & 9.
Finally, by utilizing the estimators obtained in the above procedure, each agent would compute the gradient (3) and update the outer solution x kt by using the combination of gossip communication and stochastic gradient descent (Step 3).
The inner loop solution y kt can be updated by (Step 4) in a similar way.
We highlight the following key features of our DSBO algorithm: (1) each agent would only communicate their estimates instead of the raw data in the gossip-communication process, which preserves data privacy.
(2) agent k would make O ( |N k | ) communications with its neighbors in each round, which is much smaller than the total number of agents in a naive approach.
(3) the algorithm is robust to contingencies in the network.
If some communication channel fails, the agents can still jointly learn provided that the network is still connected.
In contrast, a single-center-multi-user network would fail completely in case of a center failure.
Theory
In this section, we analyze the performance of our DSBO algorithm for both nonconvex and µ -PL objectives, and derive the convergence rates for both cases.
5.1 Nonconvex Objectives
We ﬁrst consider the scenario where overall objective function F ( x ) is nonconvex.
For nonconvex objectives, given the total number of iterations T , we employ the step-sizes in a constant form that
[Equation]
where C 0 > 0 is a small constant.
Compounded effect of consensus and SBO: As discussed earlier, to derive the convergence rate of
SBO under decentralized federated setting, the key step is to quantify the compounded effect between the consensus errors induced by the network structure and the biases induced by estimating gradient within (3).
Distinct from the central-server or non-federated regimes, the consensus errors induced by the decentralized network structure has to be handled carefully.
We conduct careful analysis to derive the contraction of consensus errors, and further show that both bias and variance of the averaged estimator diminish to zero, establishing a nontrivial convergence argument to the desired gradient and Hessian.
In particular, the estimators preserve a concentration property so that their variances decrease proportionally to 1 /K , suggesting that the network consensus effect would not degrade the concentration of the generated stochastic samples.
To achieve the best possible convergence rate, we carefully set the algorithm parameters, including step-sizes ↵ t ,  t and averaging weights  t , to control the above consensus errors and biases.
We provide the convergence rate of Alg. 1 in the following theorem and defer the detailed proof to
Section B.7 of the supplementary material.
Theorem 5.1 Suppose Assumptions 3.1, 3.2, and 3.3 hold.
Letting ¯ x t = 1 K
P k 2 K x kt , then
[Equation]
More details and proof are deferred to Section B.7 of the supplement.
Effect of consensus: In this result, the O (
K T (1  ⇢ ) 2 ) term represents the errors induced by consensus of network.
Despite the dependency with the network structure, such term diminishes to zero in the order of O (1 /T ) , becoming a small order term when T is large.
Consequently, given the network topology, the asymptotic convergence behavior of DSBO is independent of the network structure, answering the question (ii) raised in Section 1.
Linear speedup: Because each agent queries O (1) stochastic samples per round, clearly the required iteration and per-agent sample complexities for ﬁnding an ✏ -stationary point such that T
P
T  1 t =
E [ kr F (¯ x t ) k 2 ]  ✏ are both O ( 1 K ✏ 2 ) .
Such result implies that our algorithm achieves a linear speed-up effect proportionate to the number of agents K .
In other words, in the presence of more agents, each agent needs to query fewer stochastic samples to achieve a speciﬁed accuracy.
Meanwhile, our rate also matches the best-known O (1 /K ✏ 2 ) iteration and per-agent
sample complexities under the decentralized vanilla stochastic gradient descent settings (Lian et al., 2018).
To our knowledge, it is the ﬁrst time such a result has been established for DSBO problems.
Single-center-multi-user-federated SBO :
We point out that a simpliﬁed version of our algorithm solves single-center-multi-user-federated SBO, where the central server directly collects information from each agent and calculates the gradient by employing the weighted-average stochastic approximation scheme among the collected information.
In such scenario, the agents no longer make gossip communication with neighbors but synchronously receives a common solution from the central server, so that the consensus effect would disappear.
In Theorem D.1, we show that a variant of our algorithm can indeed achieve the same O ( 1 p KT ) convergence rate in this setting.
5.2 µ -PL Objectives
Next we study the case where the objective function satisﬁes the following µ -PL condition.
Assumption 5.2 There exists a constant µ > 0 such that the objective satisﬁes the PL condition:
[Equation]
Note that the class of strongly convex functions is a special case of µ -PL functions.
To utilize the µ -PL property and achieve fast convergence, unlike the nonconvex case (5) where the step-sizes are set as constants depending on the total number of iterations T , we employ step-sizes in a diminishing form that
[Equation]
where C 1 > 0 is a large constant and b = .
By following an analysis routine similar to that in the nonconvex scenario, in the next result, we derive the convergence rate of Alg. 1 for µ -PL objectives.
Theorem 5.3 Suppose Assumptions 3.1, 3.2, and 3.3 hold and the function satisﬁes the µ -PL Assumption 5.2.
Letting ¯ x T = 1 K
P k 2 K x kT , then
[Equation]
Both iteration and per-agent sample complexities for ﬁnding an ✏ -optimal point E [ F (¯ x T )]
F ⇤  ✏ are O ( 1 K ✏ ) .
Details and proof are deferred to Section C.2 of the supplement.
This result shows that our algorithm achieves a faster convergence rate for functions satisfying the µ -PL condition in terms of both iteration and sample complexities.
First, similar as the nonconvex scenario, the consensus error decays in the order of ˜ O
( 1 T 2 (1  ⇢ ) 2 ) .
Dominated by O ( 1 KT ) , such consensus decaying order indicates that the network structure would not affect Alg.
1’s asymptotic convergence behavior under µ -PL objectives.
Meanwhile, the above result implies that Alg.
1 speeds up linearly with the number of agents and matches the optimal O (1 / ✏ ) sample complexity for single-sever vanilla strongly-convex stochastic optimization (Rakhlin et al., 2011).
As a result, our algorithm achieves the optimal sample complexities for decentralized stochastic bilevel optimization, establishing the benchmark.
Numerical Experiments
In this section, we validate the practical performance of our algorithm in two examples: hyperparameter optimization and policy evaluation in Markov Decision Processes (MDP), on artiﬁcially
[Figure]
[Figure]
constructed decentralized ring networks.
We run the experiments on a single server desktop computer.
We provide the details of federate hyper-parameter optimization and federated policy evaluation.
Hyper-parameter Optimization We consider federated hyper-parameter optimization (2) for a handwriting recognition problem over the Australia handwriting dataset (Chang and Lin, 2011) consisting of data points ( w i , z i ) , where w i 2 R 14 is the feature and z i 2 { 0 , 1 } represents whether this data point belongs to category “1” or not.
In our experiment, we consider the sigmoid loss function that l i ( z ) = 1 / (1 + exp
(  z )) and a strongly convex regularizer R ( x, y ) =
P [Equation]
y i k 2 .
We consider a ring network of K agents where each agent i preserves two neighbors ( i  1) and ( i +1) and conduct gossip communication strategy with adjacency matrix w i,j = 13 for j 2 { i  1 , i, i + 1 } .
Before testing Alg.
1, we ﬁrst randomly split the dataset for training and validation, and then allocates both training and validation dataset over K agents.
We then run Algorithm 1 for T = 20000 iterations, with b = 200 , ↵ t = 0 .
[Equation]
For benchmark comparison, we implement a baseline algorithm Decentralized Bilevel Stochastic
Approximation (DBSA), a naive extension of the double-loop BSA algorithm
(Ghadimi and Wang, 2018) in the decentralized setting, formally stated in Section E.1 of the supplementary materials.
We ﬁrst consider K = 5 , test Alg.
1 for 5 ⇥ 10 4 iterations, and compare its performance with DBSA.
We report the validation loss against total samples in Figure 2 and observe that DSBO exhibits superior performance to DBSA.
In particular, Further, we observe that our algorithm outperforms the baseline algorithm DBSA that it requires fewer samples for DSBO to achieve a certain accuracy.
To investigate the efﬁciency of Alg. 1 to the network structure, we test Alg.
1 over K = 5 , 10 , 20 , and report the details of training and validation loss in Figure 2.
Further, comparing the performances of Alg. 1 over different agents K = 5 , 10 , 20 , we observe that Alg.
1 converges faster by using more agents.
This observation suggests that Alg.
1 exhibits a speed-up effect by using more agents.
We provide additional experiments on networks of larger size ( K = 100 ) and various topologies (fully-connected and randomly-connected) in Section E.1 of the supplement.
Distributed policy evaluation for reinforcement learning We consider a multi-agent MDP problem arising in reinforcement learning.
Let S be the state space, for any state s 2 S , we denote by V ( s ) the value function.
we consider the scenario where the value function can be approximated by a linear function such that V ( s ) =
> s x ⇤ , where  s 2 R m is a feature and x ⇤
R m is an unknown parameter.
To obtain the optimal x
⇤ , we consider the following regularized Bellman minimization problem
[Equation]
where r ( s, s 0 ) is the random reward incurred from transition s to s 0 ,  2 (0 , 1) is the discount factor,  is the coefﬁcient for the l 2 -regularizer, and the expectation is taken over all random transitions from s to s 0 .
In the federated learning setting, we consider a ring network of K agents.
Here each agent k has access to its own data with heterogeneous random reward function r k and can only communicate with
[Figure] )    9 - - - - - (b)   208.
(c) its two neighbors k + 1 and k  1 .
We denote by y ⇤ s ( x ) =
> s  E s
[ 1 K
[Equation]
where r k ( s, s 0 ) is the random reward function for agent k , the above problem can be recast as a bilevel optimization problem
[Equation]
As pointed out by (Wang et al., 2016), the above problem is  -strongly convex.
In our experiments, we simulate an environment with state space |S| = 100 and set the regularizer parameter  = 1 .
We test the performance of Alg.1 over three setups with K = 5 , 10 , 20 and conduct 10 independent simulations for each K .
We implement a baseline double-loop algorithm DSGD that ﬁrst estimates y ⇤ s ( x t ) with t samples in iteration t and then optimizes the solution x t .
We defer the implementation details of the environment and above algorithms to Section E.2 of the supplement.
We ﬁrst consider K = 5 , run Alg. 1 for 10 4 iterations and compare its performance with DSGD.
We plot the empirical averaged mean square error
k ¯ x t  x ⇤ k 2 against total samples generated by all agents in Figure 2.
This empirical result suggests that Alg.
1 outperforms the DSGD.
To investigate the convergence rate of DSBO, we compare the performance of DSBO over all three setups K = 5 , 10 ,and plot the trajectory of averaged log-error log( k ¯ x t  x
⇤ k 2 ) averaged, with one straight line of slope -1 provided for comparison.
We observe that over all four setups, the slopes of log( k ¯ x t  x ⇤ k 2 ) are close to -1, matching our theoretical claim in Theorem [-5.
-]5.3 that Alg.
1 converges at the rate of O (1 /t ) for strongly convex objectives.
In the above experiment, we also note that Alg.
1 converges faster when using more agents.
To further demonstrate the linear speedup effect, we compute the total generated samples for ﬁnding an ✏ -optimal solution k ¯ x t  x ⇤ k 2  ✏ and plot the 75% conﬁdence region of log-sample against number of agents K = 5 , 10 , 20 in Figure 3.
We observe that it takes a roughly same amount of samples to ﬁnd a 10  6 -optimal solution despite different number of agents are involved.
This suggests that the per-node sample complexity decreases linearly to K and validates the linear speedup claim in
Theorem 5.3.
We provide additional numerical results for other optimality level ✏ in Section E.2 of the supplementary material to further demonstrate the linear speedup effect.
Conclusion
In this paper, we propose a novel formulation of decentralized stochastic bilevel optimization.
We develop a gossip-based stochastic approximation scheme to solve this problem in various settings.
We show that our proposed algorithm ﬁnds a stationary point at a rate of O ( 1 p KT ) for nonconvex objectives, and converges to the optimal solution at a rate of O ( 1 KT ) for strongly convex objectives.
Numerical experiments on hyper-parameter optimization and multi-agent federated MDP demonstrate the practical efﬁciency of our algorithm, exhibit the effect of speed-up in decentralized setting, and validate our theoretical claims.
In our future work, we wish to develop algorithms that achieve lower iteration complexities and enjoy lower communication cost.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is an algorithmic and theoretical work justiﬁed by numerical experiments.
We are not aware of any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Complete proofs are provided in the supplement.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We will provide the code via github in the camera-ready version stage.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We provided averaged results over different random seeds and the obtained results are stable.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A]
The dataset is publicly available and no license is required.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] We are not releasing new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
We cited the website link where data is presented.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] The data does not contain personally identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not involve human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]