Decentralized Gossip-Based Stochastic Bilevel Optimization over Communication Networks
Abstract
Bilevel optimization have gained growing interests, with numerous applications being found in meta learning, minimax games, reinforcement learning, and nested composition optimization.
This paper studies the problem of decentralized distributed stochastic bilevel optimization over a network where each agent can only communicate with its neighbors, and gives examples from multi-task, multi-agent learning and federated learning.
In this paper, we propose a gossip-based decentralized bilevel learning algorithm that allows networked agents to solve both the inner and outer optimization problems in a single timescale and share information through network propagation.
We show that our algorithm enjoys the e
O ( 1 K ✏ 2 ) peragent sample complexity for general nonconvex bilevel optimization and e
O ( 1 K ✏ ) for Polyak- Ł ojasiewicz objectives, achieving a speedup that scales linearly with the network size K .
The sample complexities are optimal in both ✏ and K .
We test our algorithm on the examples of hyperparameter tuning and decentralized reinforcement learning.
Simulated experiments conﬁrmed that our algorithm achieves the state-of-the-art training efﬁciency and test accuracy.
Introduction
In recent years stochastic bilevel optimization (SBO) has attracted increasing attention from the machine learning community.
It has been found to provide favorable solutions to a variety of problems, such as meta learning and hyperparameter optimization (Franceschi et al., 2018;
Snell et al., 2017; Bertinetto et al., 2018), composition optimization (Wang and Liu, 2016), two-player games (Von Stackelberg and Von, 1952), reinforcement learning and imitation learning (Arora et al., 2020;
Hong et al., 2020).
While the majority of the above mentioned work focuses on algorithm designs in the classic centralized setting, such problems often arise in distributed/federated applications, where agents are unwilling to share data but rather perform local updates and communicate with neighbors.
Theories and algorithms for distributed stochastic bilevel optimization are less developed.
Consider the decentralized learning setting where the data are distributed over K agents K = { 1 , 2 , · · · , K } over a communication network.
Each agent can only communicate with its neighbors over the network.
One example is federated learning which is often concerned with a single-servermulti-user system, where agents communicate with a central server to solve a task cooperatively (Lan and Zhou, 2018;
Ge et al., 2018).
Another example is the sensor network, where sensors are fully decentralized and can only communicate with nearby neighbors (Taj and Cavallaro, 2011).
We consider the following decentralized stochastic bilevel optimization (DSBO) problem,
[Equation]
[Table]
[Figure]
where f k ( x, y ) =
E ⇣ k
[ f k ( x, y ; ⇣ k )] and g k ( x, y ) =
E ⇠ k [ g k ( x, y ; ⇠ k )] may vary between agents.
The expectations E ⇣ k
[ • ] and E ⇠ k [ • ] are taken with respect to the random variables ⇣ k and ⇠ k , with heterogeneous distributions across agents.
We consider the scenario where each g k ( x, y ) is strongly convex in y .
We use the notation F ⇤ =
min x 2 R dx F ( x ) , f ( x, y )
= 1 K
P k 2 K f k
( x, y ) , and g ( x, y )
= 1 K
P k 2 K g k
( x, y ) for convenience.
1.1 Example applications of SBO
SBO was ﬁrst employed to formulate the resource allocation problem (Bracken and McGill, 1973) and has since found its applications in many classic operations research settings (Cramer et al., 1994;
Sobieszczanski-Sobieski and Haftka, 1997; Livne, 1999; Tu et al., 2020; Tu, 2021), and more recently in machine learning problems (Franceschi et al., 2018;
Snell et al., 2017; Bertinetto et al., 2018;
Wang and Liu, 2016).
In particular, we introduce two applications that have recently attracted lot of attention, namely hyperparameter optimization and compositional optimization.
Hyperparameter Optimization
The problem of hyper-parameter tuning (Okuno et al., 2021) often takes the following form:
[Equation]
(2) where D train and D val are two datasets used for training and validation, respectively,
y 2 R d y is a vector of unknown parameters to optimize, ` j ( y ) is a convex loss over data i , and x 2 R d is a vector of hyper-parameters for a strongly convex regularizer R ( x, y ) .
For any hyper-parameter x 2 R d , the inner-level problem solves for the best parameter y ?
( x ) over the training set D train under the regularized training loss ` j ( y )
+ R ( x, y ) .
The goal is to ﬁnd the hyper-parameter x
⇤ 2 R d
whose corresponding best response y ?
( x ) yields the least loss over the validation set D val .
In practice, continuous hyperparameters are often tuned by a grid search which is exponentially expensive.
An efﬁcient SBO algorithm should ﬁnd the optimal parameters in time increasingly polynomially with the dimension, rather than exponentially.
When the training and validation sets are distributed across nodes, the problem becomes a distributed SBO.
Compositional Optimization Let g ( x, ⇠ ) : R d !
R d y and f ( y, ⇣ ) :
R d y !
R be two stochastic mappings.
Stochastic compositional optimization (SCO) (Wang et al., 2017a) takes the form
[Equation]
SCO applies to risk management (Yang et al., 2019; Ruszczynski, 2021), machine learning (Chen et al., 2021b), and reinforcement learning (Wang et al., 2017b).
SCO was identiﬁed as a special case of SBO by (Chen et al., 2021c).
To see this, we take the inner optimization objective to be g ( x, y ) =
E ⇠ [( y  h ( x, ⇠ )) 2 ] .
Thus y ?
( x ) = argmin y g
( x, y ) becomes y ?
( x ) =
E ⇠ [ h ( x ; ⇠ )] , arriving at an instance of SBO.
1.2 Challenges with Distributed SBO
Despite the recent rapid development of distributed single-level optimization, a method appropriate to
DSBO remains elusive.
The major hurdle to solving DSBO lies in the absence of explicit knowledge of y ?
( x ) , so that an unbiased gradient for r f ( x, y ?
( x )) is not available.
Recently, by applying the implicit function theorem, (Couellan and Wang, 2016; Ghadimi and Wang, 2018) showed that the gradient of a non-distributed SBO can be expressed as
[Equation]
providing a connection between SBO and classical stochastic optimization.
Since then, various algorithms have been proposed to obtain sharp estimators for y ?
( x ) and reduce the bias of the constructed gradients (Chen et al., 2021a;
Hong et al., 2020; Ji et al., 2021;
Yang et al., 2021).
These techniques result in tight convergence analysis and give rise to algorithms widely used in applications.
However, no prior algorithms can be applied to the distributed setting.
Solving Problem (1) becomes challenging in the distributed setting in several aspects:
Even in single-agent SBO, the lack of y ?
( x ) makes outer optimization nontrivial, and estimating y ?
( x ) requires additional stochastic approximation, weighted averaging
and sophisticated
calculation
Calculating the outer gradient is highly nontrivial, even when we have an inner solution y .
Note that
[Equation]
In other words, even if the inner problem is solved, the outer gradient requires a new estimation mechanism.
Now we move to SBO over distributed networks.
In network learning
, communication between agents can be limited by the network structure and communication protocol, so taking a simple average across agents may require multiple communication rounds.
Because of the above difﬁculties, it remains unclear how to estimate the outer gradient sharply under a distributed network.
In an attempt to tackle this problem, this paper studies the convergence theory and sample complexity of gossip-based algorithms.
In particular, we ask two theoretical questions:
(i) How does the sample complexity of DSBO scale with the optimality gap and network size?
(ii) How is the efﬁciency of DSBO affected by the network structure?
In this paper, we develop a gossip-based stochastic approximation scheme where each agent solves an optimization problem collaboratively by sampling stochastic ﬁrst- and second-order information using its data and making gossip communications with its neighbors.
In addition, we develop novel techniques for convergence analysis to characterize the convergence behavior of our algorithm.
To the best of our knowledge, our work is the ﬁrst to formulate DSBO mathematically and propose an algorithm with theoretical convergence guarantees.
Moreover, we show that our algorithm enjoys an e
O ( 1 K ✏ 2 ) sample complexity for ﬁnding ✏ -stationary points for nonconvex objectives, where e
O ( · ) hides logarithmic factors, and enjoys an e
O ( 1 K ✏ ) sample complexity for Polyak- Ł ojasiewicz (PL) functions, subsuming strongly convex optimization.
These results subsume the state-of-the-art results for nonfederated stochastic bilevel optimization (Chen et al., 2021c) and central-server regimes (Tarzanagh et al., 2022), showing that almost no degradation is induced by network consensus.
Further, the above results suggest that our algorithm exhibits a linear speed-up effect for decentralized settings; that is, the required per-agent sample complexities decrease linearly with the number of agents.
Related Works
Bilevel optimization was ﬁrst formulated by (Bracken and McGill, 1973) for solving resource allocation problems.
Later, a class of constrained-based algorithms was proposed by (Hansen et al., 1992; Shi et al., 2005), which treats the inner-level optimality condition as constraints to the out-level problem.
Recently, (Couellan and Wang, 2016) examined the ﬁnite-sum case for unconstrained strongly convex lower-lower problems and proposed a gradient-based algorithm that
Stochastic Bilevel Optimization
BSA stocBio
ALSET
FEDNEST
This Work
Network
Single-Agent
Central-Server
Decentralized
Samples in ⇣
O ( ✏  2 ) O ( ✏  2 )
O ( ✏  2 ) O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 +
K (1  ⇢ ) 2 ✏ )
Samples in ⇠ e
O ( ✏  3 ) e
O ( ✏  2 ) e O ( ✏  2 ) e
O ( ✏  2 ) e
O ( 1 K ✏ 2 ) e
O ( 1 K ✏ 2 +
K (1  ⇢ ) 2 ✏ )
Stochastic Compositional Optimization
SCGD
NASA
ALSET
FEDNEST
This Work
Network
Single-Agent
Central-Server
Decentralized
Samples in ⇣
O ( ✏  4 )
O ( ✏  2 )
O ( ✏  2 ) O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 +
K (1  ⇢ ) 2 ✏ )
Samples in ⇠
O ( ✏  4 )
O ( ✏  2 )
O ( ✏  2 ) O ( ✏  2 )
O ( 1 K ✏ 2 )
O ( 1 K ✏ 2 +
K (1  ⇢ ) 2 ✏ ) exhibits asymptotic convergence
SBO developed a double-loop algorithm and established the ﬁrst known complexity results.
Subsequently, various methods have been employed to improve the sample complexity, including two-timescale stochastic approximation (Hong et al., 2020), acceleration (Chen et al., 2021a), momentum (Khanduri et al., 2021), and variance reduction (Guo et al., 2021; Ji et al., 2021; Yang et al., 2021).
Distributed optimization was developed to handle real-world large-scale datasets (Dekel et al., 2012;
Feyzmahdavian et al., 2016) and graph estimation (Wang et al., 2015).
Centralized and decentralized systems are both important problems that have drawn signiﬁcant attention.
A centralized system considers the network topology where there is a central agent that communicates with the remaining agents (Lan and Zhou, 2018), while in a decentralized system (Gao and Huang, 2020; Koloskova et al., 2019; Lan et al., 2020; McMahan et al., 2017), each agent can only communicate with its neighbors by using gossip (Lian et al., 2017b) or gradient tracking (Pu and Nedi´c, 2021) communication strategies, with applications in multi-agent reinforcement learning (Xu et al., 2020).
Variance reduction approaches (Xin et al., 2020, 2021; Lian et al., 2017a) have also been applied to improve the convergence rate of decentralized optimization.
Random projection schemes have been studied to handle large sets of constraints (Wang and Bertsekas, 2015, 2016;
Liu et al., 2015).
All of the above trials were made on vanilla stochastic optimization problems.
We are the ﬁrst to study the decentralized bilevel optimization problem to our knowledge.
Table 1 compares this work with prior arts under different settings.
Problem Setup
Assumption 3.1 (Sampling Oracle SO )
Agent k may query the sampler and receive an independent locally sampled unbiased ﬁrst- and second-order information r x f k ( x, y ; ⇣ k ) , r y f k ( x, y ; ⇠ k ) , r y g k ( x, y ; ⇠ k ) , r
yy g k ( x, y ; ⇠ k ) , and r 2 xy g k
( x, y ; ⇠ k ) .
Assumption 3.2 (Gossip Protocol)
The network gossip protocol is speciﬁed by a K ⇥ K symmetric weight matrix W with nonnegative entries.
Each agent k may receive information from its neighbors, e.g., z j , j 2 N k , and aggregate them by a weighted sum
P j 2 N k w k,j z j .
Further, matrix W satisﬁes
(i) W is doubly stochastic such that P
i w i,j = 1 and P
j w i,j = 1 for all i, j 2 [ K ] .
(ii) There exists a constant ⇢ 2 (0 , 1) such that k W  1 K 11 > k 22 = ⇢ , where k A k 2 denotes the spectral norm of A 2 R K ⇥ K .
These assumptions on the adjacency matrix are crucial to ensure the convergence of decentralized algorithms and are commonly made in the literature of decentralized optimization (Lian et al., 2017b).
Assumption 3.3 Let C f , L f be positive scalars.
The outer level functions { f k } k 2 K satisfy the followings.
(i)
There exists at least one optimal solution to Prob.
(ii)
Both r x f k ( x, y ) and r y f k ( x, y ) are L f -Lipschitz continuous in ( x, y ) such that for all x 1 , x 2 2 R d x and y 1 , y 2 2 R d y ,
[Equation]
and kr [Equation] [Equation]
(iii)
For all x 2 R d x and y 2 R d y , E [ [Equation] k ( x, [Equation]
[Equation] k ) [Equation] 2 ]  C 2 f and E [ kr x f k ( x, y ; ⇣ k ) k 2 ]  C 2 f .
Assumption 3.4 Let C g , L g , e
L g , µ g ,  g be positive scalars.
The inner level functions { g k } k 2 K satisfy the followings.
(i)
For all x 2 R d x , g ( x, y ) is µ g -strongly convex in y .
(ii)
For all x 2 R d x and y 2 R d y , g k
( x, y ) is twice continuously differentiable in ( x, y ) .
(iii) r
y g k
( x, y ) , r 2 xy g k
( x, y ) , and r
yy g k ( x, y ) are Lipschitz continuous in ( x, y ) such that for all x 1 , x 2 2 R d x and y 1 , y 2 2 R d y ,
[Equation]
x 1  x 2 k + k y 1  y 2 k ) , and kr
yy g k ( x 1 , y 1 )  r
yy g k ( x 2 , y 2 ) k F  e L g ( k x 1  x 2 k
+ k y 1  y 2 k ) .
(iv) For all x 2 R d x , y 2 R d y , r y g k ( x, y ; ⇠ k ) , r
yy g k
( x, y ; ⇠ k ) , and r 2 xy g k
( x, y ; ⇠ k ) have bounded second-order moments such that E [ kr y g k ( x, y ; ⇠ k ) k 2 ]  C 2 g , E
[ kr 2 yy g k ( x, y ; ⇠ k ) k 22 ]  L 2 g , and E [ kr 2 xy g k
( x, y ; ⇠ k ) k 22 ]  L 2 g .
(v) For all x 2 R d x , y 2 R d y , I  1 L g r
yy g k ( x, y ; ⇠ k ) has bounded second moment such that E [ k I  1 L g r
yy g k ( x, y ; ⇠ k ) k 22 ]  (1   g ) 2 , where 0 <  g  µ g L g  1 .
We defer the detailed assumptions to Section A.1 of the supplement.
Note that we denote by k
A k
=  max ( A ) the induced 2-norm for any matrix A .
Here we point out that the above assumptions allow heterogeneity between functions f k ’s and g k ’s over the agents, and the smoothness and boundedness conditions are commonly adopted in SBO (Hong et al., 2020; Chen et al., 2021a).
Algorithm
As discussed in Section 1.2, the key challenge to solving DSBO is that each agent only has access to its own data but is required to construct estimators for the gradients and Hessian averaged across all agents.
It is particularly challenging to construct such estimators when limited by the network’s communication protocol.
To overcome this issue, we propose a gossip-based DSBO algorithm where each agent k 2 K iteratively updates a solution pair ( x kt , y kt ) by using the combination of gossip communications and weighted-average stochastic approximation, where y kt serves as an estimator of the best response y ?
t :=
y ?
(¯ x t ) with ¯ x t := 1 K [Equation] k [Equation]
K [Equation] kt .
The full algorithm is given in Alg. 1.
Here we brieﬂy explain the idea of our DSBO algorithm.
Suppose agent k would like to estimate r x f ( x kt , y kt ) by s kt , under Alg.
1 Step 6, it would query the stochastic ﬁrst-order information using its own data, make gossip communications with neighbors, and update its estimators by taking the weighted average of its previous estimate s kt  1 , neighbors j ’s estimate s jt  1 , and the newly sampled gradient r x f k ( x kt , y kt ; ⇣ kt ) . Roughly speaking, this procedure can be viewed as taking a weighted average of the gradients sampled by all agents over the network, except that the effect of consensus should also been taken into account. input: Step-sizes { ↵ t } , {  t } , {  t } , total iterations T , sampling oracle SO , weight matrix W , smoothness constant L g , hessian sampling parameter b x k 0 = 0 , y k 0 = 0 , u k 0 = 0 , s k {+
+}
= 0 , h t 0 = 0 , v k 0 , i = µ g I for i = 1 , 2 , · · · , b 1: for t = 0 , 1 , · · · , T  1 do 2: for k = 1 , · · · , K do 3:
Local sampling: Query SO at ( x kt , y kt ) to obtain r x f k ( x kt , y kt ; ⇣ kt ) , r y f k ( x kt , y kt ; ⇣ kt ) , r y g k ( x kt , y kt ; ⇠ kt ) , r 2 xy g k
( x kt , y kt ; ⇠ kt ) , and { r
yy g k
( x kt , y kt ; ⇠ kt,i ) } bi =1 .
Outer loop update: x kt +1 =
P j 2 N k w k,j x jt  ↵ t  s kt  u kt q kt h kt  .
Inner loop update: y kt
+1 =
P j 2 N k w k,j y jt   t r y g k ( x kt , y kt ; ⇠ kt ) .
Estimate r x f ( x t , y t ) : s kt
+
=
(1   t )
P j 2 N k w k,j s jt +  t r x f k ( x kt , y kt ; ⇣ kt ) .
Estimate r y f ( x t , y t ) : h kt +
=
(1   t )
P j 2 N k w k,j h jt +  t r y f k ( x kt , y kt ; ⇣ kt ) .
Estimate r 2 xy g ( x t , y t ) :[-u kt +
 = (1   t ) P-]
[Equation]
Estimate [ r 2 yy g ( x t , y t )]  1 : Set Q kt +1 ,
=
I 10: for i = 1 , · · · , b do 11:
[Equation]
+ [Equation]
[Equation]
13: end for 14:
Set q kt +1 =
L g Q [Equation] +1 ,b 15: end for 16: end for output: ¯ x t = 1 K
[Equation]
The updates for r y f and r 2 xy g are conducted in a similar manner (Steps 7 & 8), but it requires extra efforts to evaluate [ r
yy g ]  1 , because it has no unbiased estimator.
To be speciﬁc, we note that 1 K
[Equation]
yy g k ]  1 , making the unbiased estimator of the desired term unavailable even if each agent has an unbiased estimator for [ r 2 yy g k ]  1 .
This is indeed a unique challenge for decentralized bilevel optimization.
To overcome this issue, we propose a novel approach that each agent k constructs b independent estimators { v kt,j } bj =1 for r
yy g ( x kt , y kt ) using consensus and stochastic approximation.
We then estimate r
yy g ( x kt , y kt ) by utilizing the following approximation.
[Equation]
We provide details in Steps 9 - 14.
Finally, each agent will compute the gradient (3) using the estimators obtained in the above procedure and update the outer solution x kt by using the combination of gossip communication and stochastic gradient descent (Step 4).
The inner loop solution y kt can be updated similarly by Step 5.
We highlight the following key features of our DSBO algorithm: (1) each agent only communicates its estimates instead of the raw data in the gossip-communication process, preserving data privacy.
(2) agent k makes O ( |N k | ) communications with its neighbors in each round, which is much less than the total number of agents in a naive approach.
(3) the algorithm is robust to contingencies in the network.
If a communication channel fails, the agents can still jointly learn provided that the network is still connected.
By contrast, a single-center-multi-user network would fail completely in case of a center failure.
Theory
In this section, we analyze the performance of our DSBO algorithm for both nonconvex and PL objectives and derive the convergence rates for both cases.
5.1 Nonconvex Objectives
We ﬁrst consider the scenario where overall the objective function F ( x ) is nonconvex.
For nonconvex objectives, given the total number of iterations T , we employ the step-sizes in a constant form such that
[Equation]
where C 0 > 0 is a small constant.
Compounded effect of consensus and SBO: As discussed earlier, to derive the convergence rate of
SBO under decentralized federated setting, the key step is to quantify the compounded effect between the consensus errors induced by the network structure and the biases induced by estimating gradient within (3).
Unlike the central-server or non-federated regimes, the consensus errors induced by the decentralized network structure must be handled carefully.
We conduct a thorough analysis to derive the contraction of consensus errors, and further show that both the bias and variance of the averaged estimator diminish to zero, establishing a nontrivial convergence argument for the desired gradient and Hessian.
In particular, the estimators preserve a concentration property so that their variances decrease proportionally to 1 /K , suggesting that the network consensus effect does not degrade the concentration of the generated stochastic samples.
To achieve the best possible convergence rate, we carefully set the algorithm parameters, including the step-sizes ↵ t ,  t and averaging weights  t , to control the above consensus errors and biases.
We provide the convergence rate of Alg. 1 in the following theorem and defer the detailed proof to
Section B.7 of the supplementary material.
Theorem 5.1 Suppose Assumptions 3.1, 3.2, 3.3, and 3.4 hold.
Letting ¯ x t = 1 K
P k 2 K x kt , then
[Equation]
More details and proof are deferred to Section B.7 of the supplement.
Effect of consensus: In this result, the O (
K T (1  ⇢ ) 2 ) term represents the errors induced by the consensus of the network.
Despite the depending on the network structure, this term diminishes to zero in the order of O (1 /T ) , becoming a small order term when T is large.
Consequently, given the network topology, the asymptotic convergence behavior of DSBO is independent of the network structure, answering question (ii) raised in Section 1.
Linear speedup: Because each agent queries O ( b ) stochastic samples per round, clearly the required iteration and per-agent sample complexities for ﬁnding an ✏ -stationary point such that
T
P
T  1 t =
E [ kr F (¯ x t ) k 2 ]  ✏ are O ( 1 K ✏ 2 ) and e
O ( 1 K ✏ 2 ) , respectively.
This result implies that our algorithm achieves a linear speed-up effect proportionate to the number of agents K .
In other words, in the presence of more agents, each agent needs to obtain fewer stochastic samples to achieve a speciﬁed accuracy.
Meanwhile, our rate also matches the best-known O (1 /K ✏ 2 ) iteration and per-agent
sample complexities under the decentralized vanilla stochastic gradient descent settings (Lian et al., 2018).
This is the ﬁrst time such a result has been established for DSBO problems.
Single-center-multi-user-federated SBO :
We point out that a simpliﬁed version of our algorithm solves single-center-multi-user-federated SBO, where the central server collects information directly from each agent and calculates the gradient by employing the weighted-average stochastic approximation scheme for the collected information.
In such a scenario, the agents no longer communicate by gossip with neighbors but synchronously receive a common solution from the central server, so that the consensus effect disappears.
In Theorem D.1, we show that a variant of our algorithm can indeed achieve the same O ( 1 p KT ) convergence rate in this setting.
5.2 PL Objectives
Next we study the case where the objective function satisﬁes the following µ -PL condition.
Assumption 5.2 There exists a constant µ > 0 such that the objective satisﬁes the PL condition:
[Equation]
Note that the class of strongly convex functions is a special case of PL functions.
To utilize the µ -PL property and achieve fast convergence, unlike the nonconvex case (5) where the step-sizes are set as constants depending on the total number of iterations T , we employ step-sizes in a diminishing form that
[Equation]
where C 1 > 0 is a large constant.
By following an analytical process similar to that of the nonconvex scenario, in the next result, we derive the convergence rate of Alg. 1 for µ -PL objectives.
Theorem 5.3 Suppose Assumptions 3.1, 3.2, 3.3, and 3.4 hold and the function satisﬁes the µ -PL
Assumption 5.2.
Letting ¯ x T = 1 K
[Equation]
, then
[Equation]
The iteration and per-agent sample complexities for ﬁnding an ✏ -optimal point E [ F (¯ x T )]  F ⇤  ✏ are O ( 1 K ✏ ) and
[Equation]
Details and proof are deferred to Section C.2 of the supplement.
This result shows that our algorithm achieves a faster convergence rate for functions satisfying the µ -PL condition in terms of both iteration and sample complexities.
First, as in the nonconvex scenario, the consensus error decays in the order of e
O ( 1 T 2 (1  ⇢ ) 2 ) .
Dominated by O ( 1 KT ) , such on order of consensus decaying order indicates that the network structure will not affect Alg.
1’s asymptotic convergence behavior under µ -PL objectives.
Meanwhile, the above result implies that Alg.
1 speeds up linearly with the number of agents and matches the optimal O (1 / ✏ ) sample complexity for single-server vanilla strongly-convex stochastic optimization (Rakhlin et al., 2011).
As a result, our algorithm achieves the optimal sample complexities for decentralized stochastic bilevel optimization, establishing the benchmark.
Numerical Experiments
In this section, we validate the practical performance of our algorithm in two examples: hyperparameter optimization and policy evaluation in Markov Decision Processes (MDP), on artiﬁcially constructed decentralized ring networks.
We run the experiments on a single server desktop computer.
We provide the details of federate hyper-parameter optimization and federated policy evaluation.
Hyper-parameter Optimization We consider federated hyper-parameter optimization (2) for a handwriting recognition problem over the Australia handwriting dataset (Chang and Lin, 2011) consisting of data points ( w i , z i ) , where w i 2 R 14 is the feature and z i 2 { 0 , 1 } indicates whether this data point belongs to category “1” or not.
In our experiment, we consider the sigmoid loss function that l i ( z ) = 1 / (1 + exp
(  z )) and a strongly convex regularizer R ( x, y ) =
P di =1 x i 2 k
y i k 2 .
We consider a ring network of K agents where each agent i preserves two neighbors ( i  1) and ( i +1) and conducts a gossip communication strategy with adjacency matrix w i,j = 13 for j 2 { i  1 , i, i +1 } .
Before testing Alg.
1, we ﬁrst randomly split the dataset for training and validation, and then allocates both training and validation dataset over K agents.
We then run Algorithm 1 for T = 20000 iterations, with b = 200 , ↵ t = 0 .p
K/T , and  t =
t =p
K/T .
To provide a benchmark for comparison, we implement a baseline algorithm Decentralized Bilevel
Stochastic Approximation (DBSA) algorithm, a naive extension of the double-loop BSA algorithm
(Ghadimi and Wang, 2018) in the decentralized setting, formally stated in Section E.1 of the supplementary materials.
We ﬁrst consider K = 5 , test Alg.
1 for 5 ⇥ 10 4 iterations, and compare its performance with DBSA.
We report the validation loss against total samples in Figure 2 and observe that DSBO exhibits better performance than DBSA.
In particular, Further, we observe that our algorithm outperforms the baseline algorithm DBSA in that it requires fewer samples for DSBO to achieve a certain accuracy.
To investigate the efﬁciency of Alg. 1 to the network structure, we test Alg.
1 over K = 5 , 10 , 20 , and report the details of training and validation loss in Figure 2.
Further, comparing the performances of Alg. 1 over different agents K = 5 , 10 , 20 , we observe that Alg.
1 converges faster when using more agents.
This observation suggests that Alg.
1 exhibits a speed-up effect when using more agents.
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] We provide additional experiments on networks of larger size ( K = 100 ) and various topologies (fully-connected and randomly-connected) in Section E.1 of the supplement.
Distributed Policy Evaluation for Reinforcement Learning We consider a multi-agent MDP problem that arises in reinforcement learning.
Let S be the state space.
For any state s 2 S , we denote the value function by V ( s ) .
We consider the scenario where the value function can be approximated by a linear function such that V ( s ) =
> s x ⇤ , where  s 2 R m is a feature and x ⇤
R m is an unknown parameter.
To obtain the optimal x
⇤ , we consider the following regularized Bellman minimization problem
[Equation]
where r ( s, s 0 ) is the random reward incurred by a transition s to s 0 ,  2 (0 , 1) is the discount factor,  is the coefﬁcient for the l 2 -regularizer, and the expectation is taken over all random transitions from s to s 0 .
In the federated learning setting, we consider a ring network of K agents.
Here each agent k has access to its own data with a heterogeneous random reward function r k and can only communicate with its two neighbors k +
and k  1 .
We denote by y ⇤ s ( x ) =
> s x  E s {+
+}
[ 1 K
P
[Equation]
where r k ( s, s 0 ) is the random reward function for agent k .
The above problem can then be recast as a bilevel optimization problem
[Equation]
As pointed out by (Wang et al., 2016), the above problem is  -strongly convex.
In our experiments, we simulate an environment with state space |S| = 100 and set the regularizer parameter  = 1 .
We test the performance of Alg.1 over three cases with K = 5 , 10 , 20 and conduct 10 independent simulations for each K .
We implement a baseline double-loop algorithm DSGD that ﬁrst estimates y ⇤ s ( x t ) with t samples in iteration t and then optimizes the solution x t .
We defer the implementation details of the environment and above algorithms to Section E.2 of the supplement.
We ﬁrst consider K = 5 , run Alg. 1 for 10 4 iterations and compare its performance with DSGD.
We plot the empirical averaged mean square error
k ¯ x t  x ⇤ k 2 against total samples generated by all agents in Figure 2.
This empirical result suggests that Alg.
1 outperforms the DSGD.
To investigate the convergence rate of DSBO, we compare the performance of DSBO over all three scenarios with
K = 5 , 10 , 20 and plot the trajectory of the averaged log-error log( k ¯ x t  x ⇤ k 2 ) averaged, with one a straight line of slope -1 provided for comparison.
We observe that for all four cases, the slopes of log( k ¯ x t  x ⇤ k 2 ) are close to -1, matching our theoretical claim in Theorem 5.3 that Alg.
1 converges at a rate of O (1 /t ) for strongly convex objectives.
In the above experiment, we also note that Alg.
1 converges faster when using more agents.
To further demonstrate the linear speedup effect, we compute the total generated samples for ﬁnding an ✏ -optimal solution k ¯ x t  x ⇤ k 2  ✏ and plot the 75% conﬁdence region of the log-sample against the number of agents K = 5 , 10 , 20 in Figure 3.
We observe that it takes a roughly same number of samples to [Figure] 9 - - - - - (b)   208. (c) ﬁnd a 10  6 -optimal solution despite different number of agents being involved.
This suggests that the per-node sample complexity decreases linearly with K , validating the linear speedup claim in
Theorem 5.3.
We provide additional numerical results for other optimality level ✏ in Section E.2 of the supplementary material to further demonstrate the linear speedup effect.
Conclusion
In this paper, we propose a novel formulation for decentralized stochastic bilevel optimization.
We develop a gossip-based stochastic approximation scheme to solve this problem in various settings.
We show that our proposed algorithm ﬁnds a stationary point at a rate of O ( 1 p KT ) for nonconvex objectives, and converges to the optimal solution at a rate of O ( 1 KT ) for PL objectives.
Numerical experiments on hyper-parameter optimization and multi-agent federated MDP demonstrate the practical efﬁciency of our algorithm, exhibit the effect of speed-up in a decentralized setting, and validate our theoretical claims.
In our future work, we wish to develop algorithms that achieve lower iteration complexities and enjoy lower communication costs.
Acknowledgments and Disclosure of Funding
Mengdi Wang acknowledges support by NSF grants DMS-1953686, IIS-2107304, CMMI-1653435, and ONR grant 1006977.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is an algorithmic and theoretical work justiﬁed by numerical experiments.
We are not aware of any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Complete proofs are provided in the supplement.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We will provide the code via github in the camera-ready version stage.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We provided averaged results over different random seeds and the obtained results are stable.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A]
The dataset is publicly available and no license is required.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] We are not releasing new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
We cited the website link where data is presented.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] The data does not contain personally identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not involve human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]