A SYNCHRONOUS D ISTRIBUTED B ILEVEL O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Bilevel optimization plays an essential role in many machine learning tasks, ranging from hyperparameter optimization to meta-learning.
Existing studies on bilevel optimization, however, focus on either centralized or synchronous distributed setting.
The centralized bilevel optimization approaches require collecting massive amount of data to a single server, which inevitably incur significant communication expenses and may give rise to data privacy risks.
Synchronous distributed bilevel optimization algorithms, on the other hand, often face the straggler problem and will immediately stop working if a few workers fail to respond.
As a remedy, we propose A synchronous D istributed B ilevel O ptimization (ADBO) algorithm.
The proposed ADBO can tackle bilevel optimization problems with both nonconvex upper-level and lower-level objective functions, and its convergence is theoretically guaranteed.
Furthermore, it is revealed through theoretic analysis that the iteration complexity of ADBO to obtain the ϵ -stationary point is upper bounded by O ( 1 ϵ 2 ) .
Thorough empirical studies on public datasets have been conducted to elucidate the effectiveness and efficiency of the proposed ADBO.
I NTRODUCTION
Recently, bilevel optimization has emerged due to its popularity in various machine learning applications, i.e. , hyperparameter optimization (Khanduri et al., 2021; Liu et al., 2021a), metalearning (Likhosherstov et al., 2021; Ji et al., 2020), reinforcement learning (Hong et al., 2020;
Zhou & Liu, 2022), and neural architecture search (Jiang et al., 2020).
In bilevel optimization, one optimization problem is embedded or nested with another.
Specifically, the outer optimization problem is called the upper-level optimization problem and the inner optimization problem is called the lower-level optimization problem.
A general form of the bilevel optimization problem can be written as,
[Equation]
where F and f denote the upper-level and lower-level objective functions, respectively.
x ∈ R n and y ∈ R m represent the upper-level and lower-level variables, respectively.
Bilevel optimization can be treated as a special case of constrained optimization since the lower-level optimization problem can be viewed as a constraint for the upper-level optimization problem (Sinha et al., 2017).
The proliferation of smartphones and Internet of Things (IoT) devices has generated a plethora of data in various real-world applications.
Centralized bilevel optimization approaches require collecting a massive amount of data from distributed edge devices and passing them to a centralized server for model training.
These methods, however, may give rise to data privacy risks (Subramanya &
Riggio, 2021) and encounter communication bottlenecks (Subramanya & Riggio, 2021).
To tackle these challenges, recently, distributed algorithms have been developed to solve the decentralized bilevel optimization problems (Yang et al., 2022;
Chen et al., 2022b; Lu et al., 2022).
Tarzanagh et al.
(2022) and Li et al.
(2022) study the bilevel optimization problems under a federated setting.
Specifically, the distributed bilevel optimization problem can be given by
[Equation]
where N is the number of workers (devices), G i and g i denote the local upper-level and lower-level objective functions, respectively.
Although existing approaches have shown their success in resolving distributed bilevel optimization problems, they only focus on the synchronous distributed setting.
Synchronous distributed methods may encounter the straggler problem (Jiang et al., 2021) and its speed is limited by the worker with maximum delay (Chang et al., 2016).
Moreover, synchronous distributed method will immediately stop working if a few workers fail to respond (Zhang & Kwok, 2014) (which is common in large-scale distributed systems).
The aforementioned issues give rise to the following question:
Can we design an asynchronous distributed algorithm for bilevel optimization?
To this end, we develop an A synchronous D istributed B ilevel O ptimization (ADBO) algorithm which is a single-loop algorithm and computationally efficient.
The proposed ADBO regards the lower-level optimization problem as a constraint to the upper-level optimization problem, and utilizes cutting planes to approximate this constraint.
Then, the approximate problem is solved in an asynchronous distributed manner by the proposed ADBO.
We prove that even if both the upperlevel and lower-level objectives are non-convex , the proposed ADBO is guaranteed to converge.
The iteration complexity of ADBO is also theoretically derived.
To facilitate the comparison, we not only present a centralized bilevel optimization algorithm in Appendix A, but also compare the convergence results of ADBO to state-of-the-art bilevel optimization algorithms with both centralized and distributed settings in Table 1.
Contributions.
Our contributions can be summarized as:
We propose a novel algorithm, ADBO, to solve the bilevel optimization problem in an
asynchronous distributed manner .
ADBO is a single-loop algorithm and is computationally efficient.
To the best of our knowledge, it is the first work in tackling asynchronous distributed bilevel optimization problem.
We demonstrate that the proposed ADBO can be applied to bilevel optimization with non-convex upper-level and lower-level objectives with constraints .
We also theoretically derive that the iteration complexity for the proposed ADBO to obtain the ϵ -stationary point is upper bounded by O ( 1 ϵ 2 ) .
Our thorough empirical studies justify the superiority of the proposed ADBO over the existing state of the art methods.
R ELATED W ORK
Bilevel optimization: The bilevel optimization problem was firstly introduced by Bracken & McGill (1973).
In recent years, many approaches have been developed to solve this problem and they can be divided into three categories (Gould et al., 2016).
The first type of approaches assume there is an analytic solution to the lower-level optimization problem ( i.e. , ϕ ( x ) =
arg min
y ′
f ( x , y ′ ) )
(Zhang et al., 2021).
In this case, the bilevel optimization problem can be simplified to a single-level optimization problem ( i.e. , min x F ( x , ϕ ( x )) .
Nevertheless, finding the analytical solution for the lower-level optimization problem is often very difficult, if not impossible.
The second type of approaches replace the lower-level optimization problem with the sufficient conditions for optimality ( e.g. , KKT conditions) (Biswas & Hoyle, 2019; Sinha et al., 2017).
Then, the bilevel program can be reformulated as a single-level constrained optimization problem.
However, the resulting problem could be hard to solve since it often involves a large number of constraints (Ji et al., 2021;
Gould et al., 2016).
The third type of approaches are gradient-based methods (Ghadimi & Wang, 2018;
Hong et al., 2020; Liao et al., 2018) that compute the hypergradient (or the estimation of hypergradient), i.e. , ∂F ( x , y ) ∂ x
[Equation]
( x , y ) ∂ y ∂ y ∂ x , and use gradient descent to solve the bilevel optimization problems.
Most of the existing bilevel optimziation methods focus on centralized settings and require collecting a massive amount of data from distributed edge devices (workers).
This may give rise to data privacy risks (Subramanya & Riggio, 2021) and encounter communication bottlenecks (Subramanya & Riggio, 2021).
Asynchronous distributed optimization: To alleviate the aforementioned issues in the centralized setting, various distributed optimization methods can be employed.
Distributed optimization methods can be generally divided into synchronous distributed methods and asynchronous distributed methods (Assran et al., 2020).
For synchronous distributed methods (Boyd et al., 2011), the master needs to wait for the updates from all workers before it proceeds to the next iteration.
Therefore, it may suffer from the straggler problem and the speed is limited by the worker with maximum delay (Chang et al., 2016).
There are several advanced techniques have been proposed to make the synchronous algorithm more efficient, such as large batch size, warmup and so on (Goyal et al., 2017;
You et al., 2019; Huo et al., 2021; Liu & Mozafari, 2022; Wang et al., 2020).
For asynchronous distributed methods (Chen et al., 2020; Matamoros, 2017), the master can update its variables once it receives updates from S workers, i.e. , active workers ( 1 ≤ S ≤ N , where N is the number of all workers).
The asynchronous distributed algorithm is strongly preferred for large scale distributed systems in practice since it does not suffer from the straggler problem (Jiang et al., 2021).
Asynchronous distributed methods (Wu et al., 2017; Liu et al., 2017) have been employed for many real-world applications, such as Google’s DistBelief system (Dean et al., 2012), the training ofmillion YouTube videos (Le, 2013), federated learning for edge computing (Lu et al., 2019; Liu et al., 2021c).
Since the action orders of each worker are different in the asynchronous distributed setting, which will result in complex interaction dynamics (Jiang et al., 2021), the theoretical analysis for asynchronous distributed algorithms is usually more challenging than that of the synchronous distributed algorithms.
In summary, the synchronous and asynchronous algorithm have different application scenarios.
When the delay of each worker is not much different, the synchronous algorithm suits better.
While there are stragglers in the distributed system, the asynchronous algorithm is more preferred.
So far, existing works for distributed bilevel optimization only focus on the synchronous setting (Tarzanagh et al., 2022; Li et al., 2022; Chen et al., 2022b), how to design an asynchronous algorithm for distributed bilevel optimization remains under-explored.
To the best of our knowledge, this is the first work that designs an asynchronous algorithm for distributed bilevel optimization.
A SYNCHRONOUS D ISTRIBUTED B ILEVEL O PTIMIZATION
In this section, we propose A synchronous D istributed B ilevel O ptimization (ADBO) to solve the distributed bilevel optimization problem in an asynchronous distributed manner.
First, we reformulate problem in Eq.
(2) as a consensus problem (Matamoros, 2017; Chang et al., 2016),
[Equation]
3) where x i ∈ R n and y i ∈ R m are local variables in the i th worker, v ∈ R n and z ∈ R m are the consensus variables in the master node.
The reformulation given in Eq.
(3) is a consensus problem which allows to develop distributed training algorithms for bilevel optimization based on the parameter server architecture (Assran et al., 2020).
As shown in Figure 13, in parameter server architecture, the communication is centralized around the master, and workers pull the consensus variables v , z from and send their local variables x i , y i to the master.
Parameter server training is a well-known data-parallel approach for scaling up machine learning model training on a multitude of machines (Verbraeken et al., 2020).
Most of the existing bilevel optimization works in machine learning only consider the bilevel programs without upper-level and lower-level constraints (Franceschi et al., 2018;
Yang et al., 2021; Chen et al., 2022a) or bilevel programs with only upper-level (or lower-level) constraint (Zhang et al., 2022; Mehra & Hamm, 2021).
On the contrary, we focus on the bilevel programs ( i.e. , Eq.
(3)) with both lower-level and upper-level constraints, which is more challenging.
By defining ϕ ( v ) =
arg min { y ′
[Equation]
and h ( v , { y i } , z ) =
|| (cid:20) {
y i } z (cid:21) − ϕ ( v )
|| 1 , we can reformulate problem in Eq.
(3) as:
[Equation]
To better clarify how ADBO works, we sketch the procedure of ADBO.
Firstly, ADBO computes the estimate of the solution to lower-level optimization problem.
Then, inspired by cutting plane method, a set of cutting planes is utilized to approximate the feasible region of the upper-level bilevel optimization problem.
Finally, the asynchronous algorithm for solving the resulting problem and how to update cutting planes are proposed.
The remaining contents are divided into four parts, i.e. , estimate of solution to lower-level optimization problem, polyhedral approximation, asynchronous algorithm, updating cutting planes.
3.1 Estimate of Solution to Lower-level Optimization Problem
A consensus problem, i.e. , the lower-level optimization problem in Eq.
(3), needs to be solved in a distributed manner if an exact ϕ ( v ) is desired.
Following existing works (Li et al., 2022;
Gould et al., 2016; Yang et al., 2021) for bilevel optimization, instead of pursuing the exact ϕ ( v ) , an estimate of ϕ ( v ) could be utilized.
For this purpose, we first obtain the first-order
Taylor approximation of g i ( v , { y ′ i } ) with respect to v , i.e. , for a given point v , (cid:101) g
i ( v , { y ′ i } )
=
g i ( v , { y ′ i } )
+ ∇ v g i ( v , { y ′ i } )
⊤ ( v − v ) .
Then, similar to many works that use K steps of gradient descent (GD) to approximate the optimal solution of lower-level optimization problem (Ji et al., 2021;
Yang et al., 2021; Liu et al., 2021b), we utilize the results after K communication rounds between workers and master to approximate ϕ ( v ) .
Specifically, given (cid:101) g
i ( v , { y ′ i } ) , the augmented
Lagrangian function of the lower-level optimization problem in Eq.
(3) can be expressed as,
[Equation]
where φ i ∈ R m is the dual variable, and µ> 0 is a penalty parameter.
In ( k +1) th iteration, we have, (1) Workers update their local variables as follows,
[Equation]
where η y is the step-size.
Then, workers transmit the local variables
y ′
i,k , y ′
i,k +1 to the master.
(2) After receiving updates from workers, the master updates variables as follows,
[Equation]
where η z and η φ are step-sizes.
Next, the master broadcasts z ′
k +
and φ i,k +1 to workers.
As mentioned above, we utilize the results after K communication rounds to approximate ϕ ( v ) , i.e. ,
[Equation]
3.2 Polyhedral Approximation
Considering ϕ ( v ) in Eq.
(9), the relaxed problem with respect to the problem in Eq.
(4) is,
[Equation]
where ε > 0 is a constant.
Assuming that h ( v , { y i } , z ) is a convex function with respect to ( v , { y i } , z ) , which is always satisfied when we set K = 1 in Eq.
(10) according to the operations that preserve convexity (Boyd et al., 2004).
Since the sublevel set of a convex function is convex (Boyd et al., 2004), the feasible set with respect to constraint h ( v , { y i } , z ) ≤ ε is a convex set.
In this paper, inspired by the cutting plane method (Boyd & Vandenberghe, 2007; Michalka, 2013; Franc et al., 2011), a set of cutting planes is utilized to approximate the feasible region with respect to constraint h ( v , { y i } , z ) ≤ ε in Eq.
The set of cutting planes forms a polytope, let P t denote the polytope in ( t + 1) th iteration, which can be expressed as,
[Equation]
where a l ∈ R n , b i,l ∈ R m , c l ∈ R m and κ l ∈ R 1 are the parameters in l -th cutting plane, and | P t | denotes the number of cutting planes in P t .
Thus, the approximated problem in ( t + 1) th iteration can be expressed as follows,
[Equation]
And the cutting planes will be updated every k pre iterations, details are given in Section 3.4.
3.3 Asynchronous Algorithm
In the proposed ADBO, we solve the distributed bilevel optimization problem in an asynchronous manner.
The Lagrangian function of Eq.
(12) can be written as:
[Equation]
where λ l ∈ R 1 and θ i ∈ R n are dual variables.
Following (Xu et al., 2020), the regularized version of Eq.
(13) is employed to update all variables as follows,
[Equation]
where c t 1 and c t 2 denote the regularization terms in ( t + 1) th iteration.
In each iteration, we set that | P t | ≤ M, ∀ t .
c t 1 = η λ ( t +1) 14 ≥ c 1 , c
t 2 = η θ ( t +1) 14 ≥ c 2 are two nonnegative non-increasing sequences, where η λ and η θ are positive constants, and c 1 > 0 , c 2 >
meet Mc 12 + Nc 22 ≤ ϵ( ϵ > 0 is the error-tolerance which will be introduced in Definition 2).
Following (Zhang & Kwok, 2014), to alleviate the staleness issue in ADBO, we set that master updates its variables once it receives updates from S active workers every iteration and every worker has to communicate with the master at least once every τ iterations.
In ( t + 1) th iteration, let Q t + denote the index set of active workers, the proposed optimization algorithm proceeds as follows, Active worker updates the local variables as follows,
[Equation]
where ˆ t i denotes the last iteration during which worker i was active, η x and η y are step-sizes.
Then, the active workers transmit the local variables x
t +1 i and y t
+1 i to the master.
(2) After receiving the updates from active workers, the master updates the variables as follows,
[Equation]
[Equation]
where η v , η z , η λ and η θ are step-sizes.
Next, the master broadcasts v t +1 , z t +1 , θ t +1 i and { λ t +1 l } to worker i,
i ∈ Q t +1 (active workers).
Details are summarized in Algorithm 1.
3.4 Updating Cutting Planes
Every k pre iterations ( k pre > 0 is a pre-set constant, which can be controlled flexibly), the cutting planes are updated based on the following two steps (a) and (b) when t < T 1 : (a) Removing the inactive cutting planes,
[Equation]
where cp l represents the l -th cutting plane in P t and Drop( P t , cp l ) represents the l -th cutting plane cp l is removed from P t .
The dual variable set { λ t +1 } will be updated as follows,
[Equation]
where { λ t +1 } and { λ t } represent the dual variable set in ( t + 1) th and t th iterations, respectively.
Drop( { λ t } , λ l ) represents that λ l is removed from the dual variable set { λ t } .
(b) Adding new cutting planes.
Firstly, we investigate whether ( v t +1 , { y t +1 i } , z t +1 ) is feasible for the constraint h ( v , { y i } , z ) ≤ ε .
We can obtain h ( v t +1 , { y t +1 i } , z t +1 ) according to ϕ ( v t +1 ) in
Eq.
If ( v t +1 , { y t +1 i } , z t +1 ) is not a feasible solution to the original problem (Eq.
(10)), new cutting plane cp t +1 new will be generated to separate the point ( v t +1 , { y t +1 i } , z t +1 ) from the feasible region of constraint h ( v , { y i } , z ) ≤ ε .
Thus, the valid cutting plane (Boyd & Vandenberghe, 2007) a [Equation] l
[Equation]
0 must satisfy that,
[Equation]
) satisfies h ( v
[Equation]
y [Equation] , z [Equation]
≤
[Equation]
Since h ( v , { y i } , z ) is a convex function, we have that, h ( v , { y i } , z ) ≥ [Equation] v t +1 , { y t +1 i [Equation]
[Equation]
Combing Eq.
(24) with Eq.
(23), we have that a valid cutting plane (with respect to point ( v t +1 , { y t +1 i } , z t +1 ) ) can be expressed as, h ( v t +1 , { y t +1 i }
[Equation]
z t +1 )
[Equation]
[Equation]
For brevity, we utilize cp t +1 new to denote the new added cutting plane ( i.e. , Eq.
Thus the polytope P t +1 will be updated as follows,
[Equation]
where Add( P t +1 , cp t +1 new ) represents that new cutting plane cp t +1 new is added to polytope P t +1 .
The dual variable set { λ t +1 } is updated as follows,
[Equation]
where Add( { λ t +1 } , λ t +1 | P t +1 | ) represents that λ t +1 | P t +1 | is added into the dual variable set { λ t +1 } .
Finally, master broadcasts the updated P t +1 and { λ t +1 l } to all workers.
The details of the proposed algorithm are summarized in Algorithm 1.
0 i } v z
{ λ
{ θ
D ISCUSSION
Theorem 1 ( Convergence )
As the cutting plane continues to be added to the polytope, the optimal objective value of approximate problem in Eq.
(12) converges monotonically.
The proof of Theorem 1 is presented in Appendix C.
Definition 1 (Stationarity gap)
Following (Xu et al., 2020; Lu et al., 2020; Jiao et al., 2022), the stationarity gap of our problem at t th iteration is defined as:
[Equation]
Definition 2 ( ϵ -stationary point) ( { x ti } , { y ti } , v t , z t , { λ tl } , { θ ti } ) is an ϵ -stationary point ( ϵ ≥ 0 ) of a differentiable function L p , if ||∇ G t || ≤ ϵ .
T ( ϵ ) is the first iteration index such that ||∇ G t || ≤ ϵ , i.e., T ( ϵ )
=
min { t | ||∇ G t || ≤ ϵ } .
Assumption 1 (Smoothness/Gradient Lipschitz)
Following (Ji et al., 2021), we assume that L p has
Lipschitz continuous gradients, i.e., for any ω , ω ′ , we assume that there exists L > 0 satisfying that,
[Equation]
Assumption 2 (Boundedness)
Following (Qian et al., 2019), we assume that variables are bounded, i.e., || x
i || 2 ≤ α 1 , ||
v || 2 ≤ α 1 ,
||
y i ||
2 ≤ α 2 , ||
z || 2 ≤ α 2 , ||
λ l || 2 ≤ α 3 ,
||
θ
i ||
2 ≤ α 4 .
And we assume that before obtaining the ε -stationary point (i.e., t ≤ T ( ε ) − 1 ), the variables in master satisfy that
||
v t
+1 − v t ||
+ ||
z t +1 − z t ||
+ (cid:80) l ||
λ t +1 l [Equation] ϑ , where ϑ > 0 is a relative small constant.
The change of the variables in master is upper bounded within τ iterations:
[Equation]
where k 1 > 0 is a constant.
Theorem 2 (Iteration complexity) Suppose Assumption 1 and 2 hold, let η x = η y
= η v = η
z =
[Equation]
+
[Equation]
And η λ ≤ min {
[Equation]
and η
[Equation]
2 L +
[Equation]
02 are positive constants.
For a given ϵ , we have:
[Equation]
where α 3 , α 4 , γ , k d , − d , d 6 and d 7 are constants.
The detailed proof is given in Appendix B.
E XPERIMENT
In this section, experiments 1 are conducted on two hyperparameter optimization tasks ( i.e. , data hyper-cleaning task and regularization coefficient optimization task) in the distributed setting to evaluate the performance of the proposed ADBO.
The proposed ADBO is compared with the stateof-the-art distributed bilevel optimization method FEDNEST (Tarzanagh et al., 2022).
In data hypercleaning task, experiments are carried out on MNIST (LeCun et al., 1998) and Fashion MNIST (Xiao et al., 2017) datasets.
In coefficient optimization task, following (Chen et al., 2022a), experiments are conducted on Covertype (Blackard & Dean, 1999) and IJCNN1 (Prokhorov, 2001) datasets.
5. D ATA H YPER -C LEANING
Following (Ji et al., 2021; Yang et al., 2021), we compare the performance of the proposed ADBO and distributed bilevel optimization FEDNEST on the distributed data hyper-cleaning task (Chen et al., 2022b) on MNIST and Fashion MNIST datasets.
Data hyper-cleaning involves training a classifier in a contaminated environment where each training data label is changed to a random class number with a probability ( i.e. , the corruption rate).
In the experiment, the distributed data hyper-cleaning problem is considered, whose formulation can be expressed as,
[Equation] [Equation]
where D val i and D tr i represent the training and validation datasets on i th worker, respectively.
( x j , y j ) denote the j th data and label.
σ ( . ) is the sigmoid function, L is the cross-entropy loss,
C r is a regularization parameter and N is the number of workers in the distributed system.
In
MNIST and Fashion MNIST datasets, we set N = 18 , S = 9 and τ = 15 .
According to Cohen et al.
(2021), we assume that the communication delay of each worker obeys the heavy-tailed distribution.
The proposed ADBO is compared with the state-of-the-art distributed bilevel optimization method FEDNEST and SDBO (Synchronous Distributed Bilevel Optimization, i.e. , ADBO without asynchronous setting).
The test accuracy versus time is shown in Figure 1, and the test loss versus time is shown in Figure 2.
In both figures, we notice that SDBO is relatively more efficient than
FEDNEST since each step in SDBO only requires gradient descent/ascent, which is computationally efficient.
We can also observe that the proposed ADBO is the most efficient algorithm since 1) the asynchronous setting is considered in ADBO, the master can update its variables once it receives updates from S active workers instead of all workers; and 2) ADBO is a single-loop algorithm and only gradient descent/ascent is required in each iteration, thus ADBO is computationally more efficient.
5. R EGULARIZATION C OEFFICIENT O PTIMIZATION
Following (Chen et al., 2022a), we compare the proposed ADBO with baseline algorithms (FEDNEST and SDBO) on the regularization coefficient optimization task with Covertype and
IJCNN1 datasets.
The distributed regularization coefficient optimization problem is given by,
[Equation]
where ψ ∈ R n , w ∈ R n , w ′ =
[ w ′ 1 , . . . , w ′ n ] , and L denote the regularization coefficient, model parameter, and the logistic loss, respectively.
In Covertype and IJCNN1 datasets, we set N = 18 ,
S = 9 , τ = 15 and N = 24 , S = 12 , τ = 15 , respectively.
We also assume that the delay of each worker obeys the heavy-tailed distribution.
Firstly, we compare the performance of the proposed
ADBO, SDBO and FEDNEST in terms of test accuracy and test loss on Covertype and IJCNN dataset, which are shown in Figure 3 and 4.
It is seen that the proposed ADBO is most efficient because of the same two reasons we gave in Section 5.1.
[Figure]
[Figure] [Figure]
r
[Figure]
c y
[Figure]
[Figure] [Figure]
[Figure]
We also consider the straggler problem,
i
[Figure]
e. , there exist workers with high delays (stragglers) in the distributed system.
In this case, the efficiency of the bilevel optimization method with the synchronous distributed setting will be affected heavily.
In the experiment, we assume there are three stragglers in the distributed system, and the mean of (communication + computation) delay of stragglers is four times the delay of normal workers.
The results on Covertype and IJCNN1 datasets are reported in Figure 5 and 6.
It is seen that the efficiency of the synchronous distributed algorithms (FEDNEST and SDBO) will significantly be affected, while the proposed ADBO does not suffer from the straggler problem since it is an asynchronous method and is able to only consider active workers.
C ONCLUSION
Existing bilevel optimization works focus either on the centralized or synchronous distributed setting, which will give rise to data privacy risks and suffer from the straggler problem.
As a remedy, we propose ADBO in this paper to solve the bilevel optimization problem in an asynchronous distributed manner.
To our best knowledge, this is the first work that devises the asynchronous distributed algorithm for bilevel optimization.
We demonstrate that the proposed ADBO can effectively tackle bilevel optimization problems with both nonconvex upper-level and lower-level objective functions.
Theoretical analysis has also been conducted to analyze the convergence properties and iteration complexity of ADBO.
Extensive empirical studies on real-world datasets demonstrate the efficiency and effectiveness of the proposed ADBO.
A C UTTING P LANE M ETHOD FOR B ILEVEL O PTIMIZATION
In this section we introduce the proposed Cutting Plane method for Bileve Optimization (CPBO) in centralized setting.
Defining ϕ
( x ) =
arg min
y ′ f
[Equation]
y
[Equation]
and h ( x , y ) =
||
y − ϕ ( x ) || 1 , we can reformulate problem in Eq.
(1) as:
[Equation]
Following the previous works (Li et al., 2022; Gould et al., 2016; Yang et al., 2021) in bilevel optimization, it is not necessary to get the exact ϕ ( x ) , and the approximate ϕ ( x ) is given as follows.
Initialization: iteration t = 0 , variables x 0 , y 0 , { λ 0 l } and polytope P 0 .
repeatif t < T 1 then updating variable x t +1 , y t +1 and λ t +1 l according to Eq.
(41), (42) and (43); if t mod k pre == 0 then updating the polytope P t +
according to Eq.(44) and Eq.(48); updating the dual variable set { λ t +1 } according to Eq.(45) and Eq.(49); end if else updating variable x t +1 and y t +1 according to Eq.
(50) and (51); end if t = t + 1 ; until termination.
Firstly, as many work do (Ji et al., 2021; Yang et al., 2021), we utilize the K steps of gradient descent (GD) to approximate ϕ ( x ) .
And the first-order Taylor approximation of f ( x , y ′ )
with respect to x is considered, i.e. , for a given point x , (cid:101) f
( x , y [Equation]
=
[Equation] x
[-Thus, we have, ϕ ( x ) = y ′ 
 − (cid:80) K − 1 k =
 η-][Equation]
y
[Equation] x f ( x , y
[Equation]
) ⊤ ( x − x ) . Thus, we have,
[Equation]
Considering the estimated ϕ ( x ) in Eq.
(35), the relaxed problem with respect to problem in Eq.
(34) is considered as follows,
[Equation]
Assuming that h ( x , y ) is a convex function with respect to ( x , y ) , which is always satisfied when we set K = 1 in Eq.
(35) according to the operations that preserve convexity (Boyd et al., 2004). Since the sublevel set of a convex function is convex, we have that the feasible set of ( x , y ) , i.e. ,
[Equation]
is a convex set.
This paper utilizes a set of cutting plane constraints ( i.e. , linear constraints) to approximate the feasible set
Z relax .
The set of cutting plane constraints form a polytope, which can be expressed as follows,
[Equation]
where a l ∈ R n , b l ∈ R m and κ l ∈ R 1 are parameters in l -th cutting plane, and L represents the number of cutting planes in P .
Considering the approximate problem, which can be expressed as follows,
[Equation]
where P t is the polytope in ( t + 1) th iteration, and | P t | denotes the number of cutting planes in
P t .
Then, the Lagrangian function of Eq.
(39) can be written as,
[Equation]
[Equation]
The proposed algorithm proceeds as follows in ( t + 1) th iteration:
If t + 1 < T 1 , the variables are updated as follows,
[Equation]
[Table] [Equation] [Table]
[Equation]
(43) where η x , η y and η λ l represent the step-sizes.
And every k pre iterations ( k pre > 0 is a pre-set constant, which can be controlled flexibly)
the cutting planes will be updated based on the following two steps: (a) Removing the inactive cutting planes, that is, P t +1 = (cid:26) Drop( P t , cp l ) , if λ t +1 l and λ tl = 0 , l = 1 , · · · , | P t | P t , if λ t
[Equation]
where cp l represents the l -th cutting plane in P t and Drop( P t , cp l ) represents removing the l -th cutting plane cp l from P t .
And the dual variable set { λ t } will be updated as follows,
[Equation]
where { λ t +1 } and { λ t } respectively represent the dual variable set in ( t + 1) th and t th iteration.
And Drop( { λ t } , λ t +1 l ) represents remove λ t +1 l from the dual variable set { λ t } .
(b) Adding new cutting planes.
Firstly, we investigate whether ( x t +1 , y t +1 ) is a feasible solution to the original problem in Eq.
If ( x t +1 , y t +1 ) is not a feasible solution to the original problem, that is h ( x t +1 , y t +1 ) >
ε , new cutting plane is generated to separate the point ( x t +1 , y t +1 ) from
Z relax , that is, the valid cutting plane a l ⊤ x + b l ⊤ y + κ l ≤ 0 must satisfy that,
[Equation]
Since h ( x , y ) is a convex function, we have that,
[Equation]
x y (cid:21) − (cid:20) x t + y t + (cid:21)(cid:19) .
According to Eq.
(47), h ( x t +1 , y t +1 )
+ (cid:34) ∂h
( x t +1 , y t +1 ) ∂
x ∂h ( x t +1 , y t +1 )
∂ y (cid:35) ⊤ (cid:18)(cid:20) x y (cid:21) − (cid:20) x
t + y t + (cid:21)(cid:19) ≤ ε is a valid cutting plane at point ( x t +1 , y t +1 ) which satisfies Eq.
For brevity, we utilize cp t +1 new to denote this cutting plane.
Thus, we have that,
[Equation]
[Figure] [Figure] where Add( P t +1 , cp t +1 new ) represents that new cutting plane cp t +1 new is added to polytope P t +1 .
And the dual variable set is updated as follows,
[Equation]
Else if t +1 ≥ T 1 , the polytope P T 1 and dual variables will be fixed.
Variables x , y will be updated as follows,
[Equation]
where ˆ L p ( x , y ) =
F ( x , y )
+ | P
T 1 | (cid:80) l =1 λ l max { 0 , a l ⊤ x + b l ⊤ y + κ l } .
And details of the algorithm are summarized in Algorithm 2.
The comparison about the convergence results between the proposed method and state-of-the-art are summarized in Table 1.
A. E XPERIMENT
To evaluate the performance of the proposed CPBO, experiments are carried out on two applications: 1) hyperparameter optimization, 2) meta-learning.
In hyperparameter optimization, we compare
CPBO with baseline algorithms stocBio (Ji et al., 2021), STABLE (Chen et al., 2022a) and VRBO (Yang et al., 2021)), AID-CG (Grazzi et al., 2020) on the regularization coefficient optimization task (Chen et al., 2022a) on Covertype (Blackard & Dean, 1999) and IJCNN1 (Prokhorov, 2001) datasets.
We compare the performance of the proposed CPBO with all competing algorithms in terms of both the test accuracy and the test loss, which are shown in Figure 7 and 8.
In meta-learning, we focus on the bilevel optimization problem in (Rajeswaran et al., 2019).
And we compare the proposed CPBO with baseline algorithms MAML (Finn et al., 2017), iMAML (Rajeswaran et al., 2019) and ANIL (Raghu et al., 2019) on
Omniglot dataset
(Lake et al., 2015) and CIFAR-FS dataset (Bertinetto et al., [Figure] [Figure]
And the comparison between the proposed method with the baseline algorithms are shown in Figure 9 and 10.
It is seen that the proposed CPBO can achieve relatively fast convergence rate among all competing algorithms since 1) the iteration complexity of the proposed method is not high 2) every step in CPBO is computationally efficient.
A. D ISCUSSION
Definition A.1 ( x , y ) is an ϵ -stationary point of a differentiable function ˆ L p , if ||∇ x ˆ L p ( x , y ) ||
2 + ||∇ y
ˆ L p ( x , y ) ||
2 ≤ ϵ .
Assumption A.1 (Smoothness/Gradient Lipschitz)
Following (Ji et al., 2021), we assume that L p has Lipschitz continuous gradients, i.e., for any ω , ω ′ , we assume that there exists L > 0 satisfying that, |
[Equation]
Assumption A.2 (Boundedness)
Following (Qian et al., 2019), we assume that variables have boundedness, i.e., || x
|| 2 ≤ β 1 , || y || 2 ≤ β 2 .
Theorem 3 ( Iteration Complexity ) Under Assumption1 and setting the stepsizes as η x < 2 L , η y <
L , the iteration complexity (also the gradient complexity) of the proposed algorithm to obtain ϵstationary point is bounded by O ( 1 ϵ ) .
Proof of Theorem 3:
According to the Assumption A.1, when t > T 1 , we have,
[Equation]
Similarly, according to the Assumption A.1, we have,
[Equation]
Combing Eq.
(53) and Eq.
(54), we have,
[Equation]
According to the setting of η x
[Equation] η y [Equation] we have η x − Lη x 2> 0 , η y , we have η y − Lη y 2> 0 .
And we set that d = min { η x − Lη x 2 2 , η y − Lη y 2 2 } , thus we can obtain that,
[Equation]
Summing both sides of Eq.
(56) for t =
{ T 1 , · · · , T − 1 } , we obtain that,
[Equation]
where ˆ L ∗ p =
min
ˆ L p ( x , y ) .
This implies that the number of iterations required by Algorithm 2 to return an ϵ -stationary point is bounded by
[Equation]
B P ROOF OF T HEOREM
In this section [Equation] we provide complete proofs for Theorem 2.
Firstly [Equation] we make some definitions about our problem [Equation]
Definition B.1 The stationarity gap at t th iteration is defined as:
[Equation]
And we also define:
[Equation]
It follows that,[-||∇ G t || 2 = N (cid:88) i = ( || ( ∇ G t ) x i || 
-]
[-+ || ( ∇ G t ) v || 
-]
[Equation]
Definition B.2 At t th iteration, the stationarity gap w.r.t (cid:101)
L p ( { x i } , z , h, { λ l } , { θ i } ) is defined as:
[Equation]
We further define:
[Equation]
It follows that,
[Equation]
Definition B.
In our asynchronous algorithm, for the worker i in t th iteration, we define the last iteration where worker i was active as ˆ t i .
And we define the next iteration that worker i will be active as t i .
For the iteration index set that worker i is active during T + τ iteration, we define it as
V i ( T ) .
And the j th element in V i ( T ) is defined as ˆ v i ( j ) .
Then, we provide some useful lemmas used for proving the main convergence results in Theorem 2.
Lemma 1 Suppose Assumption 1 and 2 hold, we can obtain that,
[Equation]
Proof of Lemma 1:
Utilizing the Lipschitz properties in Assumption 1, we can obtain that,
[Equation]
Summing up the above inequalities in Eq.
(66), we can obtain that,
[Equation]
Combing
[Equation]
with Eq.
[Equation]
we have that,
[Equation]
Combing the Cauchy-Schwarz inequality with Assumption
[Equation] [Equation]
we can get,
[Equation]
i
[Equation]
≤
[Equation]
Thus, According to Eq.
(67), (68) and (69), we can obtain that,
[Equation]
i
[Equation]
Similarly, using the Lipschitz properties in Assumption 1, we have,
[Equation] [Equation]
Combing ∇ y
i
L p ( { x t +
i
[Equation]
with Eq.
(16), we can obtain that,
[Equation]
Combing the Cauchy-Schwarz inequality with Assumption 1, 2, we can get,
[Equation]
Thus, combing Eq.
(71), (72) and (73), we have,
L p
[Equation]
x t +1 i
[Equation] [Equation]
y
[Equation]
Combing the Lipschitz properties in Assumption 1 with Eq.
(17), we have,
[Equation]
Similarly, combing the Lipschitz properties in Assumption 1 with Eq.
(18), we have,
[Equation] p
[Equation]
x
[Equation]
i }
[Equation]
y
[Equation]
By combining Eq.
(70), (74), (75), (76), we conclude the proof of Lemma 1.
Lemma 2 Suppose the Assumption 1 and 2 hold, ∀ t ≥ T 1 , we have:
L p ( { x t +1 i } , { y t +1 i } , v t +1 , z t +1 , { λ t +1 l } , { θ t +1 i } ) − L p ( { x ti } , { y ti } , v t , z t , { λ tl } , { θ ti } ) ≤
( L +12 − 1 η x + | P t | L 2 2 a 1 + | Q t +1 | L 2 2 a) N (cid:80) i =
1 || x t +1 i {+− x ti || 
 + ( L +12 − 1 η+} y + | P t | L 2 2 a 1 + | Q t +1 | L 2 2 a) N (cid:80) i = 1 || y t
+1 i − y ti ||+ ( L +6 τk 1 NL 2− 1 η v + | P t | L 2 2 a 1 + | Q t +1 | L 2 2 a) || v t
{++1 − v t || 
+}
+ ( L +6 τk 1 NL 2− 1 η z + | P t | L 2 2 a 1 + | Q t +1 | L 2 2 a) ||
z t +1 − z t ||+( a 1 +6 τk 1 NL 2− c t − 1 1 − c t 1+ 12 η λ ) | P t | (cid:80) l
{+=1 || λ t +1 l − λ tl || 
+}
+( a 3 2 − c t − 1 2 − c t 2+ 12 η θ )
N (cid:80) i =
1 || θ t
+1 i − θ ti ||+ c t − 1 1 2 | P t | (cid:80) l =1 ( || λ t +1 l || 2 −|| λ tl || 2 )
+ 12 η λ | P t | (cid:80) l =1 || λ
{+tl − λ t − 1 l || 2 )+ c t − 1 
 N (cid:80) i =1 ( || θ t +1 i || 2 −|| θ ti || 2 )+ 12 η θ N (cid:80) i =1 || θ ti − θ t − 1 i || 2 , (77)+} where a 1 > 0 and a 3 > 0 are constants.
Proof of Lemma 2:
According to Eq.
(19), in ( t + 1) th iteration, ∀ λ ∈ Λ , it follows that:
[Equation]
Let λ = λ tl , we can obtain:
[Equation]
Likewise, in t th iteration, we can obtain:
[Equation]
Since (cid:101) L p ( { x i }
[Equation]
{ y i }
[Equation]
v [Equation]
z , { λ l } , { θ i } ) is concave with respect to λ l and follows from Eq.
(79) and
Eq.
(80), ∀ t ≥ T 1 , we have,
[Equation]
Denoting v
[Equation]
Firstly
[Equation]
we focus on the ( 1 a ) in Eq.
(82), ( 1 a ) can be expressed as follows,
[Equation]
Combing Cauchy-Schwarz inequality with Assumption
[Equation]
we can obtain,
[Equation]
where a 1 > 0 is a constant
[Equation]
Combining Eq.
(83) with Eq.
[Equation]
we can obtain that,
[Equation]
P t |
[Equation] [Equation]
Then, we put attention on the ( 1 b ) in Eq.
According to Cauchy-Schwarz inequality, ( 1 b ) can be expressed as follows,
[Equation]
where a 2 > 0 is a constant.
Then, we focus on the ( 1 c ) in Eq.
Defining L 1 ′ =
L + c 01 , according to the Assumption 1 and the trigonometric inequality, ∀ λ l , we have,
[Equation]
Following from Eq.
(87) and the strong concavity of (cid:101)
L p ( { x i } , { y i } , v , z , { λ l } , { θ i } ) w.r.t λ l (Nesterov, 2003)
[Equation]
we can obtain that,
[Equation]
In addition
[Equation]
the following inequality can be obtained,
[Equation]
Combining Eq.
(81), (82), (85), (86), (88), (89), η λ 2 ≤ 1 L 1 ′ + c 01 , and setting a 2 = η λ , we have:
L p ( { x t +1 i } , { y t +1 i } , v t +1 , z t +1 , { λ t +1 l } , { θ
[Equation]
p
[Equation]
x
[Equation]
[Equation]
{ y t +
[Equation]
t
[Equation]
t
[Equation]
According to Eq.
(20), in ( t + 1) th iteration, ∀ θ ∈ Θ , it follows that, (cid:68) [Equation] [Equation] [Equation]
[Equation]
(cid:69) = 0 .
Choosing θ
= θ ti , we can obtain, (cid:28) ∇
[Equation]
(cid:101)
L p ( { x [Equation] +1 i } [Equation]
[Equation]
Likewise, in t th iteration, we have,
[Equation]
Since (cid:101) L p ( { x i } , { y i } , v , z , { λ l } , { θ i } ) is concave with respect to θ
i
and follows from Eq.
(93): (cid:101)
L p
[Equation]
{ x t +1 i } , { y t +1 i
[Equation]
, θ
[Equation]
−
[Equation]
Denoting v t +12 ,l = θ t +1 i − θ ti − ( θ ti − θ t − 1 i ) , we have that,
N
[Equation]
=
[Equation] [Equation]
[Equation]
We firstly focus on the ( 2 a ) in Eq.
(95), we can write the ( 2 a ) as,
[Equation]
And combing the Cauchy-Schwarz inequality with Assumption
[Equation]
we can obtain,
[Equation]
where a 3 > 0 is a constant.
Thus, we can obtain the upper bound of ( 2 a ) by combining Eq.
(96) with Eq.
(97), that is,
[Equation]
Next we focus on the ( 2 b ) in Eq.
According to Cauchy-Schwarz inequality we can write the ( 2 b ) as,
[Equation]
where a 4 > 0 is a constant.
Then
[Equation]
we focus on the ( 2 c ) in Eq.
Defining L 2 ′ =
L + c 02 , according to Assumption 1 and the trigonometric inequality
[Equation]
we have,
[Equation]
Following Eq.
(100) and the strong concavity of (cid:101)
L p ( { x i } , { y i } , v , z , { λ l } , { θ i } ) w.r.t θ i , we can obtain the upper bound of (
[Equation]
In addition
[Equation]
the following inequality can also be obtained,
[Equation]
|
[Equation] [Equation]
Combining Eq.
(94), (95), (98), (99), (101), (102), η θ 2 ≤ 1 L 2 ′ + c 2 (0)
[Equation]
and setting a 4 = η θ , we have,
L p ( { x t +1 i } , { y t +1 i } , v t +1 , z t +1 , { λ t +1 l } , { θ t +1 i } ) − L p ( { x t +1 i }
[Equation]
By combining the Lemma 2 with Eq.
(90) with Eq.
(103), we conclude the proof of Lemma 2.
Lemma
Firstly, we denote S t + , S t + and F t +1 as,
[Equation]
F t +1 =
L p ( { x t +1 i } ,
[Equation]
t +
[Equation]
t
[Equation]
Defining a 5 = max { 1 , 6 τk 1 NL 2 } , ∀ t ≥ T 1 , we have,
[Equation]
Proof of Lemma 3 :
Let a 1 = 1 η λ , a 3 = 1 η θ and substitute them into the Lemma 2, ∀ t ≥ T 1 , we have,
L p ( { x t +1 i } , { y t +1 i }
[Equation]
l
[Equation]
According to Eq.
[Equation]
in ( t + 1) th iteration
[Equation]
it follows that:
[Equation]
Similar to Eq.
(109), in t th iteration
[Equation]
we have,
[Equation]
Thus
[Equation]
∀ t ≥ T
[Equation]
we can obtain the following inequality,
[Equation]
Since we have the following inequality,
[Equation]
it follows that,
[Equation]
where b t 1 > 0 .
According to the setting that c
[Equation]
− c t 1 2 .
Multiplying both sides of the inequality Eq.
(113) by 8 η λ c t 1 , we have,
[Equation]
Setting b t 1 =
c t 1 2 in Eq.
(114) and using the definition of S t 1 , we have,
[Equation]
According to Eq.
[Equation]
it follows that,
[Equation]
In addition
[Equation]
since
[Equation]
it follows that,
[Equation]
According to the setting c 02 ≤ L
[Equation]
[Equation]
we have −
[Equation]
Multiplying both sides of the inequality Eq.
(118) by [Equation] θ c [Equation]
2 , we have,
[Equation]
Setting b t 2 =
c t 2 2 in Eq.
(119) and using the definition of S t
[Equation]
we can obtain,
S t +
[Equation]
According to the setting about c t 1 and c t
[Equation]
we have η λ
[Equation]
Defining a 5 = max { 1 , 6 τk 1 NL 2 } .
Using the definition of F t +1 and combining it with Eq.
(115) and Eq.
(120), ∀ t ≥ T
[Equation]
we have,
[Equation]
Firstly
[Equation]
we set that,
[Equation]
where γ > 2 is a constant.
According to the setting of η x , η y , η v , η z and c t 1 , c t 2 , we have,
[Equation]
Combining with Lemma
[Equation]
∀ t ≥ T
[Equation]
it follows that,
[Equation]
Combing the definition of ( ∇ (cid:101) G t ) x i with trigonometric inequality, Cauchy-Schwarz inequality and the Assumption 1 and 2, we have,
[Equation]
Combing the definition of ( ∇ (cid:101) G t ) x i with trigonometric inequality, Cauchy-Schwarz inequality, we have,
[Equation]
Combining the definition of ( ∇ (cid:101) G t ) v with trigonometric inequality and Cauchy-Schwarz inequality, we have,
[Equation]
[Equation]
Combining the definition of ( ∇ (cid:101) G t )
z with trigonometric inequality and Cauchy-Schwarz inequality, we have,
[Equation]
[Equation]
Combining the definition of ( ∇ (cid:101) G t ) λ l with trigonometric inequality and Cauchy-Schwarz inequality,
[Equation]
Combining the definition of ( ∇ (cid:101) G t ) θ i with Cauchy-Schwarz inequality and Assumption 2, we have,
[Equation]
According to the Definition B
[Equation]
2 as well as Eq.
(128), (129), (130), (131), (132) and Eq.
(133), we can obtain,
[Equation]
We set that,
[Equation]
where k τ , η x , η y , η v and η z are positive constants.
Thus, combining Eq.
(134) with Eq. [Equation] Eq.
[Equation] [Equation]
we can obtain,
[Equation]
Let d t 5 denote a nonnegative sequence d t 5 =
[Equation]
It is seen that d 05 ≥ d t 5 , ∀ t ≥ 0 .
And we denote the lower bound of d t 5 as d 5 , it appears that d t 5 ≥ d 5 ≥ 0 , ∀ t ≥ 0 .
And we set the constant k τ satisfies k τ ≥ max {
[Equation]
NL 2 ) } , where η x and η y are the step-sizes in the first iteration (it is seen that η x ≥ η x , η y ≥ η y ).
We can obtain the following inequality from Eq.
(139) and the definition of d t 5 :
[Equation]
Combining the definition of d t 5 with Eq.
(127) and according to the setting || λ tl || 2 ≤ α 3 , ||
θ ti || 2 ≤ αand d 05 ≥ d t 5 ≥ d 5 , ∀ t ≥ T 1 , thus, we have,
[Equation]
= [Equation] x
[Equation]
{++
 i+}
[Equation]
Denoting (cid:101) T ( ϵ ) as (cid:101) T ( ϵ ) =
min { t | ||∇ (cid:101) G T 1 + t || ≤ ϵ 4 , t ≥ 2 } .
Summing up Eq.
(141) from t = T 1 + to t = T 1 + (cid:101) T ( ϵ ) , we have,
T 1 + (cid:101) T ( ϵ ) (cid:80) t = T 1 + d t 5 ||∇ (cid:101) G
t ||≤ F 2 − L − + 4 η λ ( c 01 c 11 + c 11 c 21 )
{+Mα 
+}
{++ c 11 
+}
Mα 3 + 72 η λ Mσ 32 + 3 d 04 ( c 11 ) 2 Mα+ 4 η θ ( c 01 c 11 + c 11 c 21 )
{+Nα 4 + c 12 
 Nα 4 + 72 η θ Nσ 42 + T 1 + (cid:101) T ( ϵ ) (cid:80) t = T 1 + 3 d+} [Equation]
(( c ˆ t
i
[Equation]
[Equation]
− ( c t i
[Equation]
1) 2 ) Nα+ 1 10 τη θ
T 1 + (cid:101) T ( ϵ ) (cid:80) t = T 1 +
N (cid:80) i =
[Equation]
it i −
[Equation]
ti || 2 − 1 10 η θ
T 1 + (cid:101) T ( ϵ ) (cid:80) t = T 1 +
N (cid:80) i =1 || θ
it
+1 − θ ti ||+
[Equation]
[Equation]
η
[Equation]
+
[Equation]
NL 2 ) T [Equation] where σ 3 =max {|| λ 1 − λ 2 ||} , σ 4 = max {|| θ 1 − θ 2 ||} and L − =
min L p ( { x ti } , { y ti } , v t , z t , { λ tl } , { θ ti } ) , which satisfy that, ∀ t ≥ T 1 + 2 ,
[Equation]
For each worker j , we have that t i − ˆ t i ≤ τ , thus,
[Equation]
Since the idle workers do not update their variables in each master iteration, for any t that satisfies ˆ v i ( j − 1) ≤ t < ˆ v i ( j ) , we have θ ti =
θ ˆ v
i ( j ) − 1 i .
And for t / ∈ V i ( T ) , we have || θ ti − θ t − 1 i || 2 = 0 .
Combing with ˆ v i ( j ) − ˆ v i ( j − 1) ≤ τ , we can obtain that,
[Equation]
Similarly, for any t that satisfies ˆ v i ( j − 1) ≤ t < ˆ v i ( j ) , we have x ti = x ˆ v i ( j ) − 1 i , y ti =
y ˆ v i ( j ) − 1 i .
And for t / ∈ V i ( T ) , we have || x ti − x t − 1 i ||
= 0 ,
|| y ti − y t − 1 i
||
= 0 .
Combing with ˆ v i ( j ) − ˆ v i ( j − 1) ≤ τ , we can obtain,
[-N (cid:80) i =
-]
[-N (cid:80) i =
-]
[-N (cid:80) i =
-]
[-N (cid:80) i =
-]
[Equation]
[-N (cid:80) i =
-]
[-i || 
-]
It follows from Eq.
(142), (144), (145), (146) that,
[Equation]
where − d and k d are constants.
d 6 is given by,
[Equation]
Thus
[Equation]
we can obtain that,
[Equation]
And it follows from Eq.
(150) that,
[Equation]
According to the setting of c t
[Equation]
c t
[Equation]
we have,
[Equation]
Summing up 1 a t 6 from t = T 1 + 2 to t = T 1 + (cid:101) T ( ϵ ) , it follows that,
[Equation]
The second inequality in Eq.
(153) is due to that ∀ t ≥ T 1 + 2 , we have,
[Equation]
The last inequality in Eq.
(153) follows from the fact that T 1 +
[Equation]
Thus [Equation] plugging Eq.
(153) into Eq.
[Equation] we can obtain:
[Equation]
Let d 7 = 4( γ − 2)
L 2 ( Mη λ + Nη θ )
[Equation]
and according to the definition of (cid:101) T ( ϵ )
[Equation]
we have:
[Equation]
[Equation]
Combining the definition of ∇ G t and ∇ (cid:101) G t with trigonometric inequality
[Equation]
we then get:
[Equation]
If t > ( 4 Mα 3 η λ 2 + 4 Nα 4 η θ 2 ) 2 1 ϵ 2 , then we have
[Equation]
ti || 2 ≤
[Equation]
Combining it with Eq.
(156), we can conclude that there exists a T (
[Equation]
such that ||∇ G t || ≤ ||
[Equation]
+
[Equation]
ti |
[Equation]
, which concludes our proof.
C P ROOF OF T HEOREM
Assuming that there are cutting planes added every k iteration, i.e. ,
[Equation]
[Equation]
Let R k denote the feasible region of problem in Eq.
(12) in k th iteration, and let R ′ denote the feasible region of problem in Eq.
(10), we have that,
[Equation]
Let F ( { x k ∗ i } , { y k ∗ i } , v k ∗ , z k ∗ ) denote the optimal objective value of the problem in Eq.
(12) in k th iteration and let F ∗ denotes the optimal objective value of the problem in Eq.
According to
Eq.
(160), we have that,
[Equation]
[Equation]
And we can obtain that,
[Equation]
It is seen from Eq.
(162) that the sequence {
[Equation]
F ( { x
[Equation]
∗ i } , { y k ∗ i }
[Equation]
k ∗
[Equation]
z k ∗ ) } is monotonically nonincreasing.
When nk → ∞ , the optimal objective value of the problem in Eq.
(12) monotonically converges to β ( β ≥ 1 ).
[Table]
D D ETAILS OF EXPERIMENTS D. A DDITIONAL RESULTS
In this section, additional experiment results on CIFAR-10 (Krizhevsky et al., 2009) and Australian (Quinlan, 1987)
datasets are reported in Figure 11 and Figure 12.
It is seen from Figure 11 and
Figure 12 that the proposed ADBO also achieves faster convergence rate.
The details of experiment setting are reported in Section D.2.
[Figure] [Figure]
[Figure] [Figure]
D. D ETAILS OF EXPERIMENTS
In this section, we provide more details of the experimental setup in this work.
In data hyper-cleaning task, experiments are carried out on MNIST, Fashion MNIST and CIFAR-10 datasets.
Following (Ji et al., 2021), we utilize the same model in data-hypercleaning task for MNIST, Fashion MNIST and CIFAR-10 datasets, and SGD optimizer is utilized.
And the step-sizes are summarized in Table 2.
In MNIST and Fashion MNIST datasets, we set N = 18 , S = 9 , τ = 15 .
And in CIFARdataset, we set N = 18 , S = 9 , τ = 5 .
We set that the (communication + computation) delays of each worker obeys log-normal distribution LN(3 . 5 , 1) .
In regularization coefficient optimization task, experiments are carried out on Covertype, IJCNN and Australian datasets.
Following (Chen et al., 2022a), we utilize the same logistic regression model, and SGD optimizer is used.
And the step-sizes are summarized in Table 2.
In Covertype dataset, we set N = 18 , S = 9 , τ = 15 ; in IJCNN1 dataset, we set N = 24 , S = 12 , τ = 15 ; and in Australian dataset, we set N = 4 , S = 2 , τ = 5 .
In the experiments that consider straggler problems, three stragglers are set in the distributed system, and the mean of (communication + computation) delay of stragglers is four times the delay of normal workers.
Codes are available in https://github.com/ICLR23Submission6251/adbo .
E P ARAMETER S ERVER A RCHITECTURE
In this section, we give the illustration of the parameter server architecture, which is shown in Figure 13.
In parameter server architecture, the communication is centralized around a set of master nodes (or servers) that constitute the hubs of a star network, and worker nodes (or clients) pull the shared parameters from and send their updates to the master nodes.