R ANDOMIZED P RIMAL -D UAL C OORDINATE M ETHOD FOR L ARGE - SCALE L INEARLY C ONSTRAINED N ONSMOOTH N ONCONVEX O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The large-scale linearly constrained nonsmooth nonconvex optimization ﬁnds wide applications in machine learning, including non-PSD Kernel SVM, linearly constrained Lasso with nonsmooth nonconvex penalty, etc.
To tackle this class of optimization problems, we propose an efﬁcient algorithm called Nonconvex
Randomized Primal-Dual Coordinate (N-RPDC) method.
At each iteration, this method only randomly selects a block of primal variables to update rather than updating all the variables, which is suitable for large-scale problems.
We provide two types of convergence results for N-RPDC.
We ﬁrst show that any cluster point of the sequence of iterates generated by N-RPDC is almost surely (i.e., with probability 1) a stationary point.
In addition, we also provide an almost sure asymptotic convergence rate of O (1 / √ k ) .
Next, we establish the expected O ( ε − 2 ) iteration complexity of N-RPDC in order to drive a natural stationarity measure below ε in expectation.
The fundamental aspect to establishing the aforementioned convergence results is a surrogate stationarity measure we discovered for analyzing
N-RPDC.
Finally, we conduct a set of experiments to show the efﬁcacy of NRPDC.
I NTRODUCTION
Many large scale problems arising in machine learning amounts to solving the following linearly constrained nonsmooth nonconvex optimization problem:
[Equation]
where A ∈ R n × d and b ∈ R n .
Throughout this paper, we impose the following assumptions on problem (P): the function f : R d → R is possibly nonconvex and continuously differentiable with its gradient ∇ f being L f -Lipschitz continuous, X is a convex and compact set of R d , i.e., there exits a positive number M such that M = max x,x (cid:48)
∈ X (cid:107) x − x (cid:48) (cid:107) , and g : R d → R is nonsmooth and nonconvex.
More precisely, g is assumed to be lower semicontinuous (l.s.c) ρ g -weakly convex and has bounded subgradients over X , i.e., there exists L g > 0 such that (cid:107) s (cid:107)
≤ L g , ∀ s ∈ ∂g
( x ) , x ∈ X , where ∂g is the subdifferential of g (see (4) for deﬁnition).
Let F ∗ be the optimal value of (P).
Recall that a function g is said to be ρ g -weakly convex if g ( · ) + ρ g 2 (cid:107) · (cid:107) 2 is convex for some constant ρ g ≥
(Vial, 1983).
It is worth mentioning that a wide class of nonsmooth nonconvex functions belong to the weakly convex class; see, e.g., (Vial, 1983; Davis & Drusvyatskiy, 2019) for more discussions on weak convexity.
We further assume g ( x ) = (cid:80) N i
[-=
-]=
1 g i ( x i ) is block separable with respect to the space decomposition
X =
X 1 × X 2 × · · · × X N , [Equation]
and
[Equation]
Let A =
( A 1 , A 2 , . . .
, A N ) ∈ R n × d be the corresponding partition of A , where A i ∈ R n × d i .
When
N = d , each X i ⊂ R corresponds to a box constraint on the coordinate x i for i = 1 , . . .
, d .
Motivations.
Our motivation for studying large scale problem (P) stems from the fact that this problem class ﬁnds wide applications in machine learning.
To be more speciﬁc, we will present several practical examples below that give rise to (P).
Application 1: Non-PSD kernel support vector machine.
Support vector machine (SVM) is a widely utilized technique for supervised learning (Boser et al., 1992; Cortes & Vapnik, 1995).
In order to improve the interpretability and enhance the robustness, some non-PSD kernels are used in the kernalized SVM method, such as the sigmoid kernel (Lin & Lin, 2003), the jittering kernel (DeCoste & Sch¨olkopf, 2002), the tangent distance kernel (Haasdonk & Keysers, 2002, August), to name a few.
Formally, the kernalized SVM problem can be written as
[Equation]
where Q is a d × d non-PSD matrix, c ∈ R is the upper bound of all variables, y ∈ {− 1 , 1 } d is the vector of labels, and 1 d is an d -dimensional vector of all 1 s.
Application 2: Linearly constrained Lasso with nonsmooth nonconvex penalty.
The Lasso is one of the most popular methods for variable selection.
The standard Lasso uses (cid:96) 1 -norm to select important variables.
Some works propose to utilize nonsmooth nonconvex regularizers to further improve the performance of the Lasso; see, e.g., (Breheny & Huang, 2011; 2015; Rakotomamonjy et al., 2019).
If additional prior information is available, the works (Deng et al., 2020, May; Gaines et al., 2018;
James et al., 2013; Won et al., 2019) propose the constrained Lasso model.
The linearly constrained
LASSO with nonsmooth nonconvex penalty can be formulated as follows:
[Equation]
where x = ( x 1 , ..., x d ) (cid:62) , X ⊂ R d , A ∈ R n × d is the design matrix, b ∈ R n is the response vector,
B ∈ R m × d and c ∈ R m are given constraints, and φ is a nonsmooth nonconvex regularizer such as
MCP (Zhang et al., 2010) or SCAD (Fan & Li, 2001) penalty.
There are plenty of other applications that give rise to (P), e.g., the robust M-estimators, distributed learning, etc.
Due to the limitation of space, we will not expand them in details here.
Related works.
Linearly constrained nonconvex optimization.
Perhaps the most widely utilized class of algorithms for solving constrained optimization problems are the Primal-dual methods.
Previous works on single-loop primal-dual methods mainly consider linearly contained convex optimization problems.
The work (Bot¸ & Nguyen, 2020) considers a nonconvex instance with a special linear constraint Ax − z = 0 , where both x and z are decision variables and the matrix A is assumed to be surjective. Under these assumptions, convergence results were established for a proximal alternating direction method of multipliers (ADMM). The recent works (Zhang & Luo, 2020a;b) provide convergence results of ADMM using gradient-based updates for linearly constrained smooth nonconvex optimization problems with a general linear constraint Ax = b , i.e., problem (P) with g ≡ 0 .
The main insights of their results are a construction of an proximally regularized auxiliary problem that dates back to (Bertsekas, 1979) and a properly constructed Lyapunov function.
Then, they established descent property on the Lyapunov function based on the dual error bound condition (see also (Hong & Luo, 2017)), which leads to iteration complexity results.
The method introduced in (Zhu et al., 2020) can be used to solve problem (P) with speciﬁc linear constraints.
This method is based on the auxiliary problem principle of augmented Lagrangian (APP-AL) method (Cohen & Zhu, 1984; Zhao & Zhu, 2019; 2021), which can be seen as a forward-backward splitting method and applies a Jacobian updating strategy.
However, both ADMM-type and APP-AL-type methods compute the full gradient of primal variables and update all primal variables at each iteration.
Therefore, the computation complexity of one iteration of these two types of methods are expensive for large-scale problems (See Remark 2.1).
Randomized coordinate methods for unconstrained optimization.
In the past decade, big data applications are ubiquitous in machine learning.
The formulated optimization problems often involve very large datasets, and hence computing the function value or the gradient can be very expensive.
These observations motivate the study of the randomized coordinate methods.
The researches on this type of methods can be traced back to (Nesterov, 2012; 2014).
In (Nesterov, 2012), Nesterov studied the iteration complexity of the randomized coordinate gradient (RCG) descent method for smooth convex optimization.
Later, in (Nesterov, 2014), the same author analyzed the randomized coordinate subgradient method for a set of piece-wise linear nonsmooth convex optimization problems.
The works (Richt´arik & Tak´aˇc, 2014) and (Lu & Xiao, 2015) extends Nesterov’s results to convex additive composite optimization.
Iteration complexity and linear convergence of RCG were studied in (Patrascu & Necoara, 2015) and (Karimi et al., 2016) for nonsmooth nonconvex optimization and smooth nonconvex optimization, respectively.
Moreover, by employing an asynchronous point to evaluate the gradient in each iteration, The works (Liu & Wright, 2015) and (Liu et al., 2014) establish the iteration complexity results of asynchronous RCG for convex smooth optimization and additive composite optimization, respectively.
A Few results on randomized coordinate methods for linearly constrained optimization.
Though the randomized coordinate-type methods for unconstrained optimization are extensively studied, there are only a few results for this type of methods concerning coupled linearly constrained optimization problems.
The works (Gao et al., 2019; Wang et al., 2014) study the randomized primal-dual coordinate (RPDC) method and provides iteration complexity results for linearly constrained convex optimization.
Better complexity results of RPDC were obtained for linearly constrained strongly convex optimization (Xu & Zhang, 2018).
The recent works (Zhu & Zhao, 2020; Latafat et al., 2019; Fercoq & Bianchi, 2019; Alacaoglu et al., 2020) provide almost sure (i.e., with probability 1 ) asymptotic convergence results of RPDC-type methods for linearly constrained convex optimization.
It is worth emphasizing that the works (Latafat et al., 2019; Fercoq & Bianchi, 2019; Alacaoglu et al., 2020) establish sequential convergence results (i.e., the convergence of the whole sequence of iterates).
However, the above mentioned works only considers either ADMM-type and APP-AL-type methods for linearly constrained smooth nonconvex optimization problem or the randomized coordinate methods for linearly constrained convex optimization.
To the best of our knowledge, no previous result concerns convergence and iterate complexity of the randomized coordinate methods for linearly constrained nonsmooth nonconvex optimization , which will be the main focus of this paper.
Main contributions.
In this paper, we aim to solve the large-scale linearly constrained nonsmooth nonconvex optimization problem (P).
To tackle it, we propose the Nonconvex Randomized PrimalDual Coordinate (N-RPDC) method based on an auxiliary problem (see Section 2).
Relying on the mild local uniform metric subregularity property (see Section 3.3), we provide two types of convergence results for N-RPDC (see Section 4).
We ﬁrst show that any cluster point of the sequence of iterates generated by N-RPDC is almost surely (i.e., with probability 1) a stationary point of (P).
In addition, we also provide an almost sure asymptotic convergence rate of O (1 / √ k ) , where k represents iteration number.
Next, we establish the expected iteration complexity of N-RPDC.
Namely,
N-RPDC needs at most O ( ε − 2 ) number of iterations in order to drive a surrogate stationarity measure below ε in expectation.
The fundamental aspect to our convergence analysis is a surrogate stationarity measure we discovered for analyzing N-RPDC (see Section 3.1), which is one of the main contributions of this work.
The standard stationarity measure (i.e., checking KKT conditions) is not directly applicable here due to the stochastic nature of N-RPDC.
Instead, we deﬁne the notion of reference point and use its distance to the iterate generated by N-RPDC as a surrogate stationarity measure.
Then, such a surrogate stationarity measure is clariﬁed by showing that it is an upper bound of the standard stationarity measure at the reference point up to a numerical constant (see Proposition 3.1 and Remark 3.1).
There are also some other interesting techniques that we utilized for establishing the above convergence results.
For example, the utilization of the local uniform metric subregularity is crucial in order to deal with the nonsmooth nonconvex term g in problem (P), as it is far from obvious how to prove the previously used error bound condition if g is not null (see Section 3.3).
Notations.
We use (cid:104)· , ·(cid:105) and (cid:107) · (cid:107) to denote the Euclidean inner product and the Euclidean norm, respectively.
For a matrix A , its minimum eigenvalue is denoted by λ min ( A ) .
The spectral norm of a matrix A is denoted by (cid:107) A (cid:107) .
Let S be a subset of R d .
We use proj S ( z ) to denote the orthogonal projector onto S and dist( z, S ) :
= inf x ∈ S (cid:107) x
− z (cid:107) to denote the distance between x and S .
When
S = ∅ , we set dist( z, S ) =
+ ∞ .
I S ( z ) =
(cid:26) 0 , z ∈ S + ∞ , z / ∈ S represents the indicator function of the set S .
If S is a convex set, then the limiting normal cone to S is deﬁned as N S ( z ) = { ξ : (cid:104) ξ, ζ − z (cid:105) ≤ 0 , ∀ ζ ∈ S } .
A UXILIARY P ROBLEM AND P ROPOSED A LGORITHM
In this section, we will deﬁne an auxiliary problem of (P).
An important feature of the auxiliary problem is that it has exactly the same set of KKT points to the original problem (P).
Furthermore, the auxiliary problem has a better geometric structure, which makes it easier to solve.
Finally, we will propose a primal-dual coordinate method based on the auxiliary problem.
Let us ﬁrst present some preliminaries on the subdifferential and KKT conditions.
Due to the fact that problem (P) is highly nonsmooth and nonconvex, our general purpose is to design an efﬁcient algorithm for ﬁnding a stationary point rather than a globally optimal solution.
Hence, we will deﬁne certain suitable stationarity measures in this subsection.
Recall that the function g in problem (P) is τ -weakly convex.
By (Vial, 1983, Proposition 4.6), we have
[Equation]
where h is the associated convex function
such that g ( x ) = h ( x ) − τ 2 (cid:107) x
|| 2 (see (Vial, 1983, Proposition 4.3) for the guarantee of the existence of such a function h ) and ∂h ( x ) is the usual convex subdifferential.
Thus, the subdifferential of a weakly convex function is always well deﬁned.
The Lagrangian of problem (P) can be written as
[Equation]
where p is the dual variable.
Then, we have the following KKT conditions for problem (P):
[Equation]
A feasible point ( x, p ) ∈
X × R n is called a stationary point of (P) if it satisﬁes the above KKT conditions, i.e., it satisﬁes dist (0 , ∂ L ( x, p )) = 0 .
2. T HE A UXILIARY P ROBLEM
Thanks to the construction used in (Bertsekas, 1979), we introduce an auxiliary problem to (P) in the following:
[Equation]
where σ > L f + ρ g is a regularization parameter.
We now characterize the set of stationary points of (P + ).
Towards that end, let w
=
( x, z, p ) ∈
X × R d × R n .
The Lagrangian of (P + ) is given by
[Equation]
Thus, we have the following KKT conditions for problem (P + ):
[Equation]
A point w =
( x, z, p ) ∈
X × R d × R n satisfying the above KKT conditions is called a stationary point of (P + ).
One crucial feature of the auxiliary problem is that it has exactly the same set of stationary points as that of the original problem (P), which is presented in the following lemma.
Lemma 2.1 (relation between stationary points of (P) and (P + )) .
The following two statements are equivalent: (a) ( x ∗ , p ∗ ) is a stationary point of problem (P); and (b) ( x ∗ , x ∗ , p ∗ ) is a stationary point of problem (P + ). [Equation]
Proof.
We ﬁrst prove that part (a) implies part (b).
Since ( x ∗ , p ∗ ) is a stationary point of (P), we have 0 ∈ ∂
L ( x ∗ , p ∗ ) .
Upon plugging z = x ∗ in the L + ( x ∗ , z, p ∗ ) shows that 0 ∈ ∂
L + ( x ∗ , x ∗ , p ∗ ) .
The inverse direction is proved by a similar argument.
The auxiliary problem can be viewed as a quadratically regularized version of the original problem (P) with an additional variable z .
Recall that f has L f -Lipschitz gradient and g is ρ g -weakly convex.
We have the following benign properties of the auxiliary problem (P + ), whose derivations can be found in Appendix A.2:
[Equation]
[Equation]
[Equation]
2 (cid:107) x − z (cid:107) 2 is continuously differentiable in ( x, z ) with Lipschitz continuous gradient; (2) F + ( x, z ) =
f ( x ) + g ( x ) + σ 2 (cid:107) x − z (cid:107) 2 is weakly convex in ( x, z ) with parameter L f + ρ g ; (3) F + ( x, z ) is bi-strongly convex,
i
e., F + ( x, z ) is strongly convex in x for every ﬁxed z and F + ( x, z ) is strongly convex in z for every ﬁxed x .
2. N ONCONVEX R ANDOMIZED P RIMAL -D UAL C OORDINATE M ETHOD
As we established in the last subsection, problem (P + ) has the same set of stationary points as that of the original problem (P) and the former has a series of benign properties.
These observations motivate us to design algorithm for solving the original problem by targeting on (P + ).
Similar to standard primal-dual methods, our algorithm for solving problem (P + ) builds on the following augmented Lagrangian of (P + ):
[Equation]
With the augmented Lagrangian, our algorithm is designed by performing an APP-AL update over ( x, z, p ) at each iteration.
Furthermore, the updates for variables x and z are achieved through randomized coordinate steps.
We depict our algorithmic procedures—i.e., the N-RPDC method— in Algorithm 1.
The proximal term D ( x, x k ) used in N-RPDC is the so-called Bregman distance function .
Let β K be the strong convexity parameter of the Bregman distance function D ( x, x k ) .
It is safe for the readers to think D ( x, x k )
= 12 (cid:107) x − x k (cid:107) 2 for now.
We put the more general choices of D ( x, x k ) in Appendix A.3.
It is worth mentioning that the subproblem for updating x k +1 in N-RPDC has a closed form solution once the Bregman distance function D ( x, x k )
=
12 (cid:107) x − x k (cid:107) 2 and g i ( x i ) is SCAD, MCP, quadratic, or (cid:96) ν -norms with ν ∈ { 1 , 2 , ∞} .
Remark 2.1 (computational complexity) .
It is claimed in (Nesterov, 2012; Wright, 2015) that a coordinate method can have a much lower computational complexity than its full counterpart.
Let us take the second motivating application (i.e., nonsmooth nonconvex constrained Lasso, problem (3) in Application 2) in Section 1 as an illustrative example to show that the computational complexity of one iteration of N-RPDC with N = d can be much cheaper than the full primal-dual counterpart (ADMM-type or APP-type method).
Note that in this case, each X i is a box constraint on the coordi- nate x i .
Suppose we set D ( x, x k )
= 12 (cid:107) x − x k (cid:107) 2 .
Then, the update of w k +1 =
( x k +1 , z k +1 , p k +1 ) consists of: 1)
p k +
= p k + ηs k . 2)
x k +1 i ( k )
=
prox α x , I X i ( k ) + φ (cid:16) x ki ( k ) − α
x
( (cid:0) A
i ( k ) (cid:1) (cid:62) r k + σ ( x k − z k ) i ( k ) + (cid:0) B i ( k ) (cid:1) (cid:62) ( p k +1 + γs k ))
(cid:17) and x k +1 j (cid:54) =
i ( k ) =
x kj (cid:54) =
i ( k ) , where I X i ( k ) is the indicator function of X i ( k ) .
z k +1 i ( k ) =
z ki ( k ) − α z σ ( z k − x k ) i
( k ) with z k +1 j (cid:54) =
i ( k ) =
z kj (cid:54) =
i ( k ) .
We also update the residual vectors as r k +
=
r k + A i ( k ) ( x k
+1 i ( k ) − x ki ( k ) ) and s k
+1 =
s k + B i ( k ) ( x k
+1 i ( k ) − x ki ( k ) ) .
If we precompute r 0 =
Ax 0 − b and s
=
Bx 0 − c , and the proximal mapping prox α x , I X i ( k ) + φ admits a closed form solution (which is the case when φ is the SCAD or MCP penalty), then the above updates have a total complexity O (max { m, n } ) .
Therefore, the complexity of N-RPDC is d × cheaper than its full counterpart (which has a computational complexity of O ( d max { m, n } ) in one iteration).
S TATIONARITY M EASURE , L YAPUNOV F UNCTION , AND U NIFORM M ETRIC S UBREGULARITY
In this section, we will establish a series of important results, which serve as the foundations for our later convergence analysis.
3. T HE S TATIONARITY M EASURE
Consider the current iterate w k .
If w k +1 is updated by the full primal-dual counterpart to N-RPDC (i.e., N = 1 in Algorithm 1), then the step length (cid:107) w k − w k +1 (cid:107) is a natural stationarity measure due to the facts that dist(0 , ∂ L + ( w k +1 ))
=
O ( (cid:107) w k − w k +1 (cid:107) ) with some step sizes hidden in the big-O and lim k
→ + ∞ (cid:107) w k − w k +1 (cid:107) = 0 can be established by an easy argument based on an appropriate Lyapunov function.
However, if w k +1 is updated by our N-RPDC with N (cid:54)
= 1 , which is a stochastic algorithm that only updates part of w k to w k +1 , it is no longer obvious to show lim k
→ + ∞ (cid:107) w k − w k +1 (cid:107) = 0 .
Therefore, due to the stochastic nature of N-RPDC, the step length (cid:107) w k − w k +1 (cid:107) cannot play the role of stationarity measure.
This observation motivates us to work with a surrogate stationarity measure.
Towards that end, for a given point w = ( x, z, p ) , we deﬁne the reference point of w as T ( w ) :
= ( T x ( w ) , T z ( w ) , T p ( w )) and  
[Equation]
One possible way to understanding the reference point is to let w k =
( x k , z k , p k ) , then T ( w k ) is nothing else other than the next iterate
w k +1 if the full primal-dual method is utilized, i.e., N = 1 in
N-RPDC.
However, it is worthing mentioning in the case where N > 1 , we do not need to compute the reference point T ( w k ) explicitly.
Instead, N-RPDC only use a block coordinate for updating.
The next proposition provides us a valid stationary measure, which is crucial for our later convergence analysis.
Proposition 3.1 (surrogate stationarity measure) .
For all w = ( x, z, p ) ∈
X × R d × R n , we have dist (cid:0) 0 , ∂
L + ( T ( w )) (cid:1) ≤ c (cid:107) w − T ( w ) (cid:107) =:
Φ( w ) .
Here, T ( w ) is the reference point of w deﬁned in {+(8) and c := √ 3 max (cid:110) L K α x + L f +2 σ + (cid:107) A (cid:107) , 1 α z + 2 σ, γ (cid:107) A (cid:107) +
 η (cid:111) is a positive constant.+}
Remark 3.1 (interpretation of the surrogate stationarity measure) .
The mapping Φ( w ) will serve as the surrogate stationarity measure for analyzing the convergence of N-RPDC.
Particularly, it is clear that dist (cid:0) 0 , ∂ L + γ ( T ( w )) (cid:1)
=
when Φ( w ) = 0 .
In order to characterize the iteration complexity of N-RPDC, we focusing on achieving Φ( w ) ≤ ε since it implies that w is ε -close to the reference point T ( w ) which is an ε -stationary point (here, T ( w ) is deﬁned to be ε -stationary since dist (0 , ∂ L + ( T ( w ))) ≤ ε ).
These interpretations clarify our surrogate stationarity measure.
Note that similar notion of surrogate stationarity measure dates back to Ekeland’s variational principle (Ekeland, 1974) and also appeared in recent advances of the analysis of the subgradient-type methods; see, e.g., (Davis & Drusvyatskiy, 2019; Li et al., 2021).
It is interesting to note that the main difference between our approach and the existing ones lies in the choice of the reference point.
Let us take the approach used in (Davis & Drusvyatskiy, 2019) for analyzing subgradient-type method as an example.
The authors deﬁne the proximal mapping of the objective function as the reference point, while we utilize the update of the full primal-dual method (i.e., N = 1 in Algorithm 1) as the reference point.
Such a main difference comes from different sources of difﬁculties.
The main difﬁculty for the analysis of subgradient-type method in (Davis & Drusvyatskiy, 2019) comes from the fact that this algorithm is intrinsically not a descent method on the objective function, while our main difﬁculty is due to the stochastic nature of N-RPDC.
3. T HE L YAPUNOV F UNCTION AND O NE -S TEP A NALYSIS
Recall that the function F + deﬁned in (P + ) is strongly convex in x for every ﬁxed z (see our analysis in Section 2.1).
Let us deﬁne
[Equation]
In addition, we also deﬁne
[Equation]
where L + γ is deﬁned in (7).
Next, mimicking the construction of the Lyapunov function used in (Zhang & Luo, 2020b), we deﬁne
[Equation]
as the Lyapunov function for our convergence analysis.
The ﬁrst part of Λ( · ) is augmented Lagrangian L + γ ( w ) of (P + ).
The second part is dual gap of (P + ) with ﬁxed z .
Additionally, by the deﬁnition of ψ γ ( z, p ) , we have that L + γ ( w ) ≥ ψ γ ( z, p ) .
By the strong duality, we have ν ( z ) =
max p ψ γ ( z, p ) .
Therefore, we have the fact that Λ( w ) ≥
ν
( z ) ≥ F ∗ .
Standard analysis for primal-dual methods develops certain descent property on the Lyapunov function.
Before that, we ﬁrst present a preliminary one-step analysis for N-RPDC.
Note that indices i ( k ) , k = 0 , 1 , 2 , . . .
in N-RPDC are random variables.
Thus, it generates a random output.
We use
[Equation]
to denote the ﬁltration generated by the random variables i (0) , i (1) , . .
, i ( k ) .
Clearly, we have
F k ⊂ F k +1 .
We use E i ( k ) and E F k to denote the expectations taken over the random variable i ( k ) and the ﬁltration F k , respectively.
The following lemma provides estimations for ν ( z ) − ψ γ ( z, p ) and L + γ ( w ) after one-step execution of N-RPDC.
Lemma 3.1 (one-step analysis of N-RPDC) .
Suppose 0 < α x ≤ β K / (cid:2)
L f
+ 2 σ + γ (cid:107) A (cid:107) 2 + 5 (cid:3) and 0 < η < 1 / (cid:34)
[Equation]
For all k ≥ 0 , we have Λ( w k )
− E i ( k ) Λ( w k +1 )
[Equation]
where σ is deﬁned in (P + ) and λ is any positive constant.
Toward establishing certain decent property on the Lyapunov function in terms of the stationarity measure Φ( w k ) , we have to connect all the terms on the right-hand side of the inequality of
Lemma 3.1 to Φ 2 ( w k ) =
(cid:107) w k − T ( w k ) (cid:107) 2 .
Thus, it remains to upper bound (cid:107) x
( z k , T p ( w k ))
− x ( z k ) (cid:107) by (cid:107)
Ax ( z k , T p ( w k ))
− b (cid:107) , which needs the so-called uniform metric subregularity property.
3. U NIFORM M ETRIC S UBREGULARITY AND ITS C ONSEQUENCE
Let us deﬁne the parameter set-valued mapping
[Equation]
as the subdifferential of L + γ with respect to ( x, p ) with ﬁxed z .
Clearly, we have ∂ L zγ ( x ( z, p ) , p ) =
(cid:18)
Ax ( z, p [Equation] − b (cid:19) due to the optimality of x ( z, p ) in (10).
To proceed, we introduce the notion of local uniform metric subregularity property for the parameter set-valued mapping.
Deﬁnition 3.1 (local uniform metric subregularity, cf.
(Kruger & Duy Cuong, 2021)) .
Let H z ( u ) : U ⇒ V be a set-valued mapping.
We call that H z ( u ) satisﬁes the locally metric subregularity property with parameter ( δ, κ ) uniformly over all z ∈ Z if
[Equation]
for any u ∈ { u (cid:48) ∈ U : dist(0 , H z ( u (cid:48) )) ≤ δ } , where δ, κ > 0 are uniform constants.
This local error bound condition gives the following result.
Lemma 3.2.
Suppose the set-valued mapping ∂ L zγ ( x, p ) deﬁned in (13) satisﬁes the local uniform metric subregularity property, then there is κ > 0 (independent of z ) such that
[Equation]
− b (cid:107) , whenever (cid:107) Ax ( z, p ) − b (cid:107) ≤ δ .
Equipped with this lemma, we are able to upper bound (cid:107) x
( z k , T p ( w k ))
− x ( z k ) (cid:107) by (cid:107)
Ax ( z k , T p ( w k ))
− b (cid:107) locally in order to establish the descent property on the Lyapunov function.
We will discuss this descent property in the next section.
Now, let us provide the conditions on problem (P) in order to ensure the local uniform metric subregularity assumed in Lemma 3.2.
Lemma 3.3 (sufﬁcient condition for local uniform metric subregularity) .
Consider problem (P).
Suppose that ∇ f ( x ) is piecewise afﬁne and ∂g ( x ) is polyhedral multifunction and X is a polyhedral set, then ∂ L zγ ( x, p ) deﬁned in (13) satisﬁes the local uniform metric subregularity property.
Remark 3.2 (generality of local uniform metric subregularity) .
This sufﬁcient condition ensures that ∂ L zγ ( x, p ) must satisfy the local uniform metric subregularity property whenever f is quadratic (can be nonconvex, i.e., its Hessian is not required to be positive semideﬁnite), g can be written as (cid:107) · (cid:107) 1 − ρ g 2 (cid:107) · (cid:107) 2 , (cid:96) 2 -norm, (cid:96) ∞ -norm structure, and X is a polyhedron, which covers all the two motivating applications listed in Section 1.
Remark 3.3 (comparison to the error bound condition utilized in (Zhang & Luo, 2020b)) .
Recall that the work (Zhang & Luo, 2020b) considers problem (P) with g ≡ 0 .
They use the so-called dual error bound condition (see (Hong & Luo, 2017)) instead of the local uniform metric subregularity property.
However, we argue that it is far form obvious how to extend their technique to cover our nonsmooth nonconvex case, as it is not clear whether one can establish the dual error bound condition if g is not null.
Remark 3.4 (further comments on the surrogate stationarity measure Φ( w ) deﬁned in Proposition 3.1) .
It is interesting that the sufﬁcient condition in Lemma 3.3 also ensures that ∂ L + ( w ) satisﬁes the local metric subregularity when Φ( w ) ≤ ε with sufﬁcient small ε (Robinson, 1981), i.e., dist( w, W ) ≤ κ (cid:48)
dist(0 , ∂ L + ( w )) for some κ (cid:48) > 0 , where W is the set of stationary points of (P + ).
Then, we have dist( w, W ) ≤ (1 /c + κ (cid:48) )
ε which follows from dist( T ( w ) , W ) ≤ κ (cid:48)
ε (due to Φ( w ) ≤ ε ) and the triangle inequality.
C ONVERGENCE AND I TERATION C OMPLEXITY OF N-RPDC
Equipped with all the machineries developed in Section 3, we are now ready to establish the convergence properties of N-RPDC.
4. D ESCENT PROPERTY ON THE L YAPNOV F UNCTION
Upon invoking Lemma 3.2 in Lemma 3.1, we can derive the descent property in expectation on the
Lyapnov Function.
Lemma 4.1 (expected sufﬁcient decrease for Λ k ) .
Suppose that Assumptions of Lemma 3.1 hold.
If the set-valued mapping ∂ L zγ ( x, p ) deﬁned in (13) satisﬁes the local uniform metric subregularity property (see Deﬁnition 3.1).
Let λ =
min { δη 2 σM 2 +1 , η σκ 2 +1 } , the parameter α z satisfy 0 <
α z < 1 / (cid:104) 2 σ + σλ + 3 σ 2 σ − L f − ρ g + 1 (cid:105) , and c 1 = min (cid:110) 1 ,
N 2 η ( σκ 2 +2)+ (cid:111) .
Then, we have
[Equation]
4. A LMOST S URE C ONVERGENCE R ESULTS AND E XPECTED I TERATION C OMPLEXITY
In this subsection, we establish the almost sure (i.e., with probability 1) convergence results for
N-RPDC.
Theorem 4.1 (almost sure convergence result) .
Under the setting of Lemma 4.1, then
(a) lim k
→ + ∞ Φ( w k ) = 0 almost surely.
(b) The sequence { w k } is almost surely bounded.
(c) Any cluster point of the sequence { w k } is almost surely a stationary point of (P).
In addition, we can also provide the almost surely asymptotic rate of convergence in the sense of limit inferior for our N-RPDC method.
Theorem 4.2 (almost sure O (1 / √ k ) convergence rate) .
Under the setting of Lemma 4.1, we have lim inf k →
[Equation]
∞ √ [Equation] [Equation] ) = 0 almost surely i.e., the asymptotic O (1 / √ k ) convergence rate holds almost surely.
The almost sure convergence rate established above is of asymptotic nature, which concerns the behavior of N-RPDC when k → ∞ .
Next, we also provide the iteration complexity results of
N-RPDC in expectation.
Theorem 4.3 (expected iteration complexity) .
Under the setting of Lemma 4.1.
The sequence { w k } is generated by N-RPDC.
Then
[Equation]
Consequently, to achieve E F t Φ( w ¯ t ) ≤ ε for some 0 ≤ ¯ t ≤ t , N-RPDC needs at most t =
Nc
[Equation]
(Λ
[Equation]
w
[Equation]
number of iterations.
Remark 4.1.
Note that the closely related work (Zhang & Luo, 2020b) provides a similar iteration complexity result (i.e., O ( ε 2 ) ) if g ≡ 0 . By contrast, our results can deal with the additional nonsmooth nonconvex term g .
In addition, we also provide the almost sure convergence results, complementing the results in (Zhang & Luo, 2020b).
Let us also emphasize that the main motivation of our algorithm lies in that N-RPDC is a randomized coordinate method, which is more suitable for modern large-scale problems.
Remark 4.2. The recent work (Zhang & Luo, 2020a) establishes a global dual error bound condition, which allows them to avoid the compactness assumption on X .
Due to the existence of g , we utilize the local uniform metric subregularity rather than this dual error bound condition. To further relax the compactness assumption on X in problem (P), one possible direction is to establish a global version of the utilized local uniform metric subregularity. We leave this direction as a future work.
N UMERICAL E XPERIMENTS
Because of the limitation of space, we put the numerical experiments in Appendix A.1.
We can observe that N-RPDC with larger N slightly outperforms that with smaller N in terms of convergence speed.
We also compared our N-RPDC to the algorithm proposed in (Zhang & Luo, 2020b); see Figure 1. We can observe that N-RPDC slightly outperforms their algorithm. Note that the main motivation of N-RPDC is that it is more suitable for modern large-scale problems due to its reasonably fast convergence speed and low computational complexity at each iteration.
A A PPENDIX
The Appendix is organized as follows.
Subsection A.1 discusses implementation details for NRPDC and presents numerical experiments on non-PSD kernel SVM and sum to zero constrained
LASSO problem with SCAD penalty.
Some properties of the auxiliary problem (P + ) are given in subsection A.2.
Some examples of Bregman distance function are given in A.3.
Subsection A. shows the proof of Proposition 3.1.
Subsection A.5 provides the proof of Lemma 3.1.
Proof of
Lemma 3.2 is given in subsection A.6.
Subsection A.7 provides the proof of Lemma 3.3.
Subsection A.8 shows the proof of Lemma 4.1.
Subsection A.9 provides the proof of Theorem 4.1.
Subsection A.10 gives the proof of Theorem 4.2.
The proof of Theorem 4.3 is given in subsection A.11.
A. N UMERICAL E XAMPLES
This subsection discusses experiments conducted using MATLAB R2020a on a personal computer with Intel Core i5-6200U CPU (2.40GHz) and 8.00 GB RAM.
A.1. N ON -PSD KERNEL SUPPORT VECTOR MACHINE PROBLEM
Consider the non-PSD kernel support vector machine problem,
[Equation]
where x ∈ R
d are the decision variables, and
Q ∈ R
d × d is a matrix, possibly non-PSD.
Let Q =
( Q (cid:62) 1 , Q (cid:62) 2 , · · · , Q (cid:62) N ) (cid:62)
∈ R
d × d be an appropriate partition of matrix Q and Q i be an d i × d matrix.
L + γ of Non-PSD kernel SVM can be written as
[Equation]
x (cid:107) 2 . [Figure] Thanks of Proposition 3.3, for problem Non-PSD kernel SVM, ∂ L zγ is locally metric subregular in ( x, z, p ) uniformly in z at point 0 .
N-RPDC with D ( x, x k ) =
12 (cid:107) x − x k (cid:107) 2 for non-PSD kernel support vector machine problem is
[Equation]
Choose i ( k ) from { 1 , 2 , . .
. , N } with equal probability
[Equation]
Thus, the primal subproblem of N-RPDC has the closed form   x k
[Equation]
(cid:20) x
[Equation]
k )
[Equation]
, x k +1 j (cid:54) = i ( k ) = x kj (cid:54) = i ( k ) ; and
[Equation]
We used LIBSVM dataset heart scale and ionosphere in the experiment.
Q was generated using the sigmoid kernel (Lin & Lin, 2003), and we selected c = 1 .
We compared two algorithms: nonconvex ADMM in (Zhang & Luo, 2020b) ( nADMM ), and N-RPDC from this paper on these two datasets.
For N-RPDC algorithm, we partitioned the variables N = 10 , 90 blocks for heart scale and
N = 10 , 70 blocks for ionosphere .
In order to verify the theoretical results of this paper, we compute the sequence of reference point { T ( w k ) } along with the sequence { w k } generated by N-RPDC.
In the real world applications of
N-RPDC, the reference point sequence is not necessary to compute.
In Figure 1, the left graph show the number of blocks and min 0 ≤ k ≤ t Φ( w k ) with respect to epoch count for heart scale ; the right graph show the number of blocks and min 0 ≤ k ≤ t Φ( w k ) with respect to epoch count for ionosphere .
For both datasets, min 0 ≤ k ≤ t Φ( w k ) of each epoch of N-RPDC is the average value of 50 times result.
Moreover, we can draw the conclusion that N-RPDC is slightly outperforms nADMM and larger N is slightly outperforms small N .
A.1. S UM TO ZERO CONSTRAINED LASSO PROBLEM WITH SCAD PENALTY
Consider the sum to zero constrained LASSO problem with SCAD penalty:
[Equation]
where x ∈ R
d is the decision variables and A ∈ R n × d is the regressors matrix.
Let A
=
( A 1 , A 2 , · · · , A N
) ∈ R n × d be an appropriate partition of matrix A and A i be an n × d i matrix.
b ∈ R n is the response vector.
1 d is an d -dimensional vector of 1 s. φ ( · ) is the SCAD penalty:
[Equation]
L + γ of (CLASSO-SCAD) can be written as
[Equation]
Thanks of Proposition 3.3, for problem (CLASSO-SCAD), ∂ L zγ is locally metric subregular in ( x, z, p ) uniformly in z at point 0 .
We take D ( x, x k ) =
12 (cid:107) x − x k (cid:107) 2 and construct N-RPDC algorithm for (CLASSO-SCAD) as following: [Equation] k
[Equation]
η
[Equation]
[Equation]
d x k ) ,
Choose i ( k ) from { 1 , 2 , . .
. , N } with equal probability
[Equation]
where q k = p k +1 + γ ( 1 (cid:62) d x k ) .
Thus, the primal subproblem of RPDC has the closed form  
[Equation]
with ζ k = x ki ( k ) − α x
[ A (cid:62) i ( k )
( Ax k − b )
+ σ ( x k − z k ) i
( k ) + q k 1 n i ( k ) ] and µ = α x . (cid:40)
[Equation]
In this experiment, the elements of A ∈ R n × d are selected i.i.d.
from a Gaussian N (0 , 1) distribution.
To construct a sparse true solution x ∗ ∈ R d , given the dimension d and sparsity s , we select s entries of x ∗ at random to be nonzero and N (0 , 1) normally distributed, and set the rest to zero.
The measurement vector b ∈ R n is obtained by b =
Ax ∗ + δ with the elements of the noise vector δ ∈ R n are i.i.d.
N (0 , 0 . 001) .
We choose θ = 2 . 3 with n = 360 , d = 1280 , s = 8 for the ﬁrst dataset and n = 720 , d = 2560 , s = 16 for the second dataset.
We partitioned the variables N = 10 , 40 , 80 blocks.
Thus d i = 128 , 32 , 16 for the ﬁrst dataset and d i = 256 , 64 , 32 for the second dataset.
In order to verify the theoretical results of this paper, we compute the sequence of reference point { T ( w k ) } along with the sequence { w k } generated by N-RPDC.
In the real world applications of
N-RPDC, the reference point sequence is not necessary to compute.
In Figure 2, the left graph [Figure] d=1280,n=360,s=
N= N= N=   1000. d=1280,n=360,s=
N= N= N= , we can draw the conclusion that larger N is slightly outperforms
A. S OME PROPERTIES OF THE AUXILIARY PROBLEM P +
The auxiliary problem can be viewed as a quadratically regularized version of the original problem (P) with an additional variable z .
Recall that f has L f -Lipschitz gradient and g is ρ g -weakly convex.
We have the following benign properties of the auxiliary problem (P + ):
(1) f
( x ) + σ 2 (cid:107) x − z (cid:107) 2 is continuously differentiable in ( x, z ) with Lipschitz continuous gradient; (2) F + ( x, z ) =
f ( x ) + g ( x ) + σ 2 (cid:107) x − z (cid:107) 2 is weakly convex in ( x, z ) with parameter L f + ρ g ; (3) F + ( x, z )
is bi-strongly convex, i.e., F + ( x, z ) is strongly convex in x for every ﬁxed z and
F + ( x, z ) is strongly convex in z for every ﬁxed x .
The ﬁrst property is trivially true.
To see the second property, we note that the L f -Lipschitz gradient property of f implies that f ( x ) ≥ f ( y ) + (cid:104) s, x − y (cid:105) − L f 2 (cid:107) x − y (cid:107) 2 , ∀ x, y ∈ R d (Nesterov, 2003,
Lemma 1.2.3).
This, together with (Vial, 1983, Proposition 4.8), establishes that f is L f -weakly convex 2 .
By (Vial, 1983, Proposition 4.1), we have that f
+ g is weakly convex with parameter
L f + ρ g .
We have shown the second property y recognizing that ( x, z ) (cid:55)→ σ 2 (cid:107) x − z (cid:107) 2 is a convex mapping.
The third property is a direct consequence of the deﬁnition of weak convexity, which yields that f ( x ) + g ( x ) + σ 2 (cid:107) x − z (cid:107) 2 is σ − ( L f +
ρ g ) strongly convex in x .
A. S OME EXAMPLES OF B REGMAN DISTANCE FUNCTION
The proximal term D ( x, x k ) used in N-RPDC is the so-called Bregman distance function , which is deﬁned as
[Equation]
Here, K : R d → R is the core function, which is β K -strongly convex and has L K -Lipschitz continuous gradient on R d .
One popular choice of the core function is K ( x ) = 1 2 (cid:107) x (cid:107) 2 , which yields
D ( x, x k ) =
12 (cid:107) x − x k (cid:107) 2 .
However, more generally, the core function can also be selected as the
Q -quadratic proximal regularization term K ( x ) = 12 (cid:107) x (cid:107) 2 Q , where Q is a positive deﬁnite matrix.
This choice covers some second-order methods as we can set Q to be the regularized Hessian matrix.
A. P ROOF OF P ROPOSITION 3.
Proof.
The optimal condition of
[Equation]
and
[Equation]
yields that
[Equation]
Let
[Equation]
and
[Equation]
Then we have that
[Equation]
By the expression (15-17) of ξ
[Equation]
α z and η are given positive numbers and Assumptions of problem (P), there exists positive number c ≥ √
[Equation]
and the following inequality holds
[Equation]
Combining ξ ∈ ∂
L + ( T ( w k ))
and it follow the desired statement.
A. P ROOF OF L EMMA 3.
In order to provide the Lemma 3. [Equation] we need the following two propositions as preparation.
Proposition A.1 (Lipschitz properties of x ( z ) and x ( z, p ) ) .
Suppose σ > L f + ρ g .
Then
(i) (cid:107) x
( z ) − x ( z (cid:48) )
(cid:107) ≤
[Equation]
[Equation]
) (cid:107)
[Equation]
(cid:48) (cid:107) ; (ii) (cid:107) x
( z, p ) − x ( z (cid:48) , p ) (cid:107) ≤
[Equation]
− ( L f +
[Table]
Proof.
(i) Let φ ( x, z ) =
f ( x ) + g ( x ) + σ 2 (cid:107) x − z (cid:107)
+
I
˜
X
( x ) with ˜ X
= { x ∈ X | Ax − b = 0 } .
Since σ > L f + ρ g , we have φ ( x
[Equation]
z ) is σ − ( L f + ρ g ) -strongly convex in x . Therefore
[Equation]
using the strongly convex of φ ( x, z ) in x , we have that
[Equation]
Therefore
[Equation]
(ii) Let ϕ ( x, z, p )
=
L + γ ( x, z, p ) + I X ( x ) .
Since σ > L f + ρ g , function ϕ
( x, z, p ) also is σ − ( L f + ρ g ) -strongly convex in x .
Therefore
[Equation]
Again using the strongly convex of ϕ ( x, z, p ) in x , we have that
[Equation]
p ) [Equation] Therefore, (cid:107) x ( z, p ) − x ( z (cid:48) , p ) (cid:107) ≤ σ σ − ( L f +
[Equation]
(cid:107)
z − z (cid:48) (cid:107) .
Proposition A.2.
Suppose σ > L f + ρ g .
Let { w k } be generated by N-RPDC.
For given w k , the primal output ( x k +1 , z k +1 ) are random variables, and the following assertion hold:
[Equation]
Proof.
Since x ( z k , T p ( w k )) =
arg min x ∈ X L + γ ( x, z k , T p ( w k )) and L + γ ( x, z, p ) is σ − L f − ρ g -strongly convex in x , then ∀ ζ ∈ ∂g
( T x ( w k ))
+ N X
( T x ( w k ))
we have that
[Equation]
It follows that
[Equation]
Again using the optimal condition of
[Equation]
we have that
[Equation]
Therefore
[Equation]
with
[Equation]
By Assumptions on (P)
[Equation]
we obtain that
[Equation]
and
[Equation]
Then by the triangle inequality
[Equation]
above inequality yields the desired result.
With these two propositions in hands, we can provide the technical proof of Lemma 3.1.
Proof of Lemma 3.
Step 1: Estimate the term L + γ ( w k ) − E
i ( k ) L + γ ( w k +1 ) .
By the dual update of N-RPDC, we have that
[Equation]
Since x k +1 is the solution of subproblem of x in N-RPDC scheme, we have that
[Equation]
Take x = x k in (20), by the fact that g i ( k ) ( x ki ( k ) ) − g i ( k ) ( x k +1 i ( k ) ) = g ( x k ) − g ( x k +1 ) and (cid:104)∇ i
[Equation]
p
[Equation]
−
[Equation]
=
[Equation]
Ax k
[Equation]
b
[Equation]
] , x
[Equation]
x
[Equation]
[Equation]
, we have that
[Equation]
It yields that
[Equation]
Since (cid:104)
[Equation]
therefore
[Equation]
Additionally, since z k +1 is the solution of subproblem of z in N-RPDC
[Equation]
we have that
[Equation]
Take z
= z k in (22), by the fact that
[Equation]
we have that
[Equation]
Since
[Equation]
above inequality yields that
[Equation]
By the combination of (19), (21) and (23), we obtain that
[Equation]
Take expectation with respect to i ( k ) on both side of above inequality
[Equation]
we have that
[Equation]
By the fact that E i ( k ) (cid:107) x k − x k +1 (cid:107) 2 = 1 N
[Equation]
E i
[Equation]
k [Equation] − z
[Equation]
+ [Equation] 2 = 1 N
[Equation]
z k − T z ( w k ) (cid:107)and p k +1 =
T p ( w k ) [Equation] above inequality follows that
[Equation]
Step 2: Estimate the term ν ( z k ) − ν ( z k +1 ) .
By the Danskin’s theorem, we have that ∇ ν
( z ) =
σ ( z − x ( z )) .
By statement (i) of Proposition A.1, we have that
[Equation]
The gradient Lipschitz property of ν follows that
[Equation]
Step 3:
Estimate the term ψ γ ( z k +1 , p k +1 ) − ψ γ ( z k , p k ) .
Since x ( z k , p k ) =
arg min x ∈ X L + γ ( x, z k , p k ) , we have ψ [Equation] z k +1 , p k +1 ) [Equation]
ψ
[Equation]
z k , p
[Equation]
+1 ) =
[Equation]
+
[Equation]
By the concavity of ψ γ ( z, · ) and ∇ ψ γ ( z, p )
=
Ax ( z, p ) − b , we have
[Equation]
By the dual update of N-RPDC [Equation] the above inequality yields that
[Equation]
By Proposition A.
[Equation]
(26) guarantees that
[Equation]
By the combination of (25) and (27)
[Equation]
we obtain that
[Equation]
Step 4:
Estimate the term [ ν ( z k ) − ψ γ ( z k , p k )] − E i
[Equation] k ) [ ν ( z k +1 ) − ψ γ ( z k +1 , p k +1 )] .
Summing statement (24) and (28), we obtain the variation of dual gap:
[Equation]
By Proposition A [Equation] [Equation] we have that
[Equation]
with λ > 0 is any positive number.
By the combination of (29) and (30)
[Equation]
we have that
[Equation]
Take expectation with respect to i ( k ) on both side of above inequality
[Equation]
we have that
[Equation]
By the fact that E i ( k ) (cid:107) z k − z k +1 (cid:107) 2 = 1 N {+(cid:107) z k − T z ( w k ) (cid:107) 2 and p k +
 = T p ( w k )+}
[Equation]
[-z k − T z ( w k ) (cid:107) 2 and p k +
-]
above inequality follows that [Equation] −
[Equation]
Step 5: Estimate the term Λ( w k ) − E
i
( k ) Λ( w k +1 ) .
By the combination of Step 1 and Step 4, we obtain that
[Equation]
Since 0 < α x ≤ β K / [ L f + 2 σ + γ (cid:107) A (cid:107) 2 + 5] , then β K α x − ( L f + 2 σ + γ (cid:107) A (cid:107) 2 ) ≥ 5 .
Moreover, 0 < η ≤ 1 /
[Equation]
implies
[Equation]
A. P ROOF OF L EMMA 3.
Proof.
The result is directly by the fact that dist (cid:0) 0 , ∂
L zγ ( x ( z, p ) , p ) (cid:1) =
(cid:107)
Ax ( z, p ) − b (cid:107) .
A. P ROOF OF L EMMA 3.
Proof.
The claim is provided by the results of Proposition 1 and Corollary of (Robinson, 1981).
A. P ROOF OF L EMMA 4.
Proof.
Taking λ =
min { δη 2 σM 2 +1 , η σκ 2 +1 } and 0 < α z < 1 / (cid:104)
σ + σλ + 3 σ 2 σ − L f − ρ g + 1 (cid:105) in
Lemma 3.1, we have that
[Equation]
For the last two terms of above inequality, we have two cases to discuss.
• Case 1: (cid:107)
Ax ( z k , T p ( w k )) −
[Equation]
Since ∂ L zγ ( x, p ) is locally metric subregular in ( x, p ) uniformly in z over R d at pointwith parameters κ, δ > 0 , (35) yields that
[Equation]
• Case 2: (cid:107)
Ax [Equation] z [Equation]
> 2 σλM
[Equation]
Since M = max x,x (cid:48) ∈ X (cid:107) x − x (cid:48) (cid:107) , we have that
[Equation]
For both case
[Equation]
(35) yields that
[Equation]
By Proposition A
[Equation]
[Equation]
above inequality yields that
[Equation]
[Equation]
with c 1 = min (cid:110) 1 ,
N 2 η
[Equation]
2 +2)
[Equation]
[Equation]
A. P ROOF OF T HEOREM 4.
In order to provide the Theorem 4.1, we need the famous Robbins-Siegmund theorem.
Theorem A.1.
(Robbins-Siegmund Theorem, Robbins & Siegmund (1971), Theorem 1.3. in Duﬂo (2013), Theorem 2.27 in Carpentier et al. (2015)) Let { Λ k } k ∈ N , { µ k } k ∈ N , { ν k } k ∈ N and { η k } k ∈ N be four positive sequences of real-valued random variables adapted to the ﬁltration { ξ k } k ∈ N .
Assume that
[Equation]
and that
[Equation]
Then, the sequence { Λ k } k ∈ N almost surely converges to a ﬁnite 3 random variable Λ ∞ , and (cid:80) k ∈ N η k < + ∞ , a.s..
Then we can derive the technical proof of this theorem.
Proof of Theorem 4.1: (a) By Lemma 4.1 and the fact that Λ( w ) ≥ F ∗ , we have that
[Equation]
By the Robbins-Siegmund Theorem (Theorem A.1), we obtain that, lim k
→ + ∞ Λ( w k ) is almost surely exists and + ∞ (cid:80)
[Equation]
a.s. .
Therefore, we have lim k
→ + ∞ Φ( w k ) =
; (b) By the compact of X , we have the sequences { x k } and { x ( z k ) } are bounded.
By statement (a), we have lim k
→ + ∞ Λ( w k ) is almost surely exists.
It follows that the sequence { Λ( w k ) } is almost surely bounded.
By the deﬁnition of Λ( w k ) , we also have that Λ( w k ) ≥ ν
( z k ) ≥ F ( x ( z k )) ≥ F ∗ .
Therefore the sequence { ν ( z k ) } is almost surely bounded.
Now we show the almost surely boundness of the sequence { z k } by contradiction.
Suppose P (cid:8) [Equation] z k (cid:107) → + ∞ (cid:9) > 0 .
Since the sequence { x ( z k ) } is bounded, there exists a subsequence { z k (cid:48) } such that x ( z k (cid:48) )
→ ¯ x .
Therefore
P
[Equation]
+ σ 2 (cid:107) ( x
[Equation]
z k (cid:48) ))
[Equation] z k [Equation] 2 → + ∞ (cid:111) > 0 , which follows a contradiction with that { ν ( z k (cid:48) ) } is almost surely bounded.
Therefore, we obtain the almost surely boundness of the sequence { z k } .
Next we propose the boundness of { p k } .
By the optimal condition of problem of T x ( w k ) =
arg min x ∈ X
[Equation]
[Equation]
, we have
[Equation]
Since T x ( w k ) =
N E i ( k ) x k +1 − ( N − 1) x k and the boundness of the sequence { x k } , we have the sequence { T x ( w k ) } is bounded.
By bounded of subgradient of g , the boundness of { x k } , { T x ( w k ) } and the almost surely boundness of { z k } , it follows that (cid:107) A (cid:62) T p ( w k ) (cid:107) <
+ ∞ , ∀ k > 0 , a.s. .
By the fact that p k +
=
T p ( w k ) , we have that [Equation]
a
[Equation]
By update of p in N-RPDC, we have that p
[Equation]
when ˆ p
[Equation]
+
[Equation]
η k (cid:88)
[Equation]
= (
[Equation]
[Equation]
k [Equation] [Equation] − x ∗ ) ∈
Im ( A
with x ∗ be the optimal solution of (P). Since p 0 ∈ Null( A (cid:62) ) , we have A (cid:62) p
= 0 and (cid:107) A (cid:62) ˆ
p k +1 (cid:107) <
+ ∞ , a.s. .
Then we have two cases to discuss.
• Case 1: rank ( A ) = n .
Obviously, we have that
[Equation]
with λ min ( AA (cid:62) ) be the smallest eigenvalue of matrix AA (cid:62) .
• Case 2: rank ( A ) =
r < n .
Without loss of generality, assume the ﬁrst r rows of A (denoted by A r ∈ R r × d ) are linearly independent, we have
[Equation]
where I r × r ∈ R r × r is the indentity matrix and B ∈ R ( n − r ) × r .
Let Q := I r × r + B (cid:62) B .
It is easy to show that Q is symmetric and positive deﬁnite.
By the fact that ˆ p k +1 =
ηA (cid:80) kj =1 ( x j − x ∗ ) , we have that
[Table]
[Equation]
where λ min (cid:0) A r ( A r ) (cid:62) (cid:1) is the smallest eigenvalue of positive semideﬁnite matrix A r ( A r ) (cid:62) and λ min ( Q (cid:62) Q ) is the smallest eigenvalue of positive semideﬁnite matrix Q (cid:62) Q .
Using the fact that
[Equation]
where λ max ( Q ) is the largest eigenvalue of matrix Q .
Therefore, we have that
[Equation]
Therefore, for both cases, there exists d > 0 such that
[Equation]
which yields the almost surely boundness of { p k } by the triangle inequality
Therefore, the sequence { w k } is almost surely bounded.
(c) By lim k
→ + ∞ (cid:107) w k − T ( w k ) (cid:107) =
a.s. in statment (a), Lemma 2.1, Proposition 3.1, the almost surely boundness of { w k } , and the closedness of ∂ L + γ ( · ) , we obtain the desired result.
A. P ROOF OF T HEOREM 4.
Proof.
Suppose this statement does not hold, that is,
[Equation]
for some d > 0 .
Then for k 0 large enough, for all k ≥ k 0 we have
[Equation] > 0 .
Therefore
[Equation]
Since the fact that + ∞ (cid:80) k
= kd 2 k
+
=
+ ∞ , we have that above inequality is contradicted with the fact that + ∞ (cid:80) k
=
(Φ( w k ))
2 < + ∞ a.s. in the proof of Theorem 4.1.
Therefore, we obtain the desired result.
A. P ROOF OF T HEOREM 4.
Proof.
Recalling the inequality of Lemma 4.1 and the fact that Λ( w k ) ≥ ν ( z k ) ≥ F ∗ , we have
[Equation]
Taking expectation with respect to F t , t > k for above inequality, we obtain that
[Equation]
By the fact Λ( w k ) ≥ ν ( z k ) ≥ F ∗ , it follows
[Equation]
By the Jensen’s inequality and the convexity of function h ( x ) = x 2 , x ∈ R , it follows that
[Equation]
By the fact that E F t Φ( w k ) ≥ 0 and h ( x ) = x 2 , x ∈ R is monotonic increasing in x on [0 , + ∞ ) , we have that
[Equation]
Here comes that
[Equation]