F ULLY P OLYNOMIAL -T IME R ANDOMIZED A PPROXIMATION S CHEMES FOR G LOBAL O PTIMIZATION OF H IGH -D IMENSIONAL M INIMAX C ONCAVE P ENALIZED G ENERALIZED L INEAR M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Global solutions to high-dimensional sparse estimation problems with a folded concave penalty (FCP) have been shown to be statistically desirable but are strongly NP-hard to compute which implies the non-existence of pseudopolynomial time global optimization schemes in the worst case.
This paper shows that, with high probability, a global solution to generalized linear models with minimax concave penalty (MCP), a speciﬁc type of FCP, coincides with a stationary point characterized by the signiﬁcant subspace second order necessary conditions (S 3 ONC).
Given that the desired S 3 ONC solution admits a fully polynomialtime randomized approximation scheme (FPRAS), we are able to demonstrate the existence of an FPRAS for this strongly NP-hard problem.
We further demonstrate two versions of the FPRAS for generating the desired S 3 ONC solutions.
One follows the paradigm of an interior point trust region algorithm
and the other is the well-studied local linear approximation (LLA).
Our analysis thus provides new techniques for global optimization of certain NP-Hard problems and new insights on the effectiveness of LLA
I NTRODUCTION
This paper concerns global optimization of a folded concave penalized formulation of highdimensional learning generalized linear models, which belongs to statistical/machine learning problems such that the number of dimensions (or number of ﬁtting parameters) p is (much) larger than the number of samples n .
This type of problem has recently become very common in a variety of engineering and scientiﬁc applications (Fan et al., 2014a; Fan & Li, 2006) including computational biology, speech recognition and image processing (Huang et al., 2016; Algamal & Lee, 2015;
Xing et al., 2001; Palmer & Ostendorf, 2001; Bagaria et al., 2012; Paul et al., 2013).
Globally minimal solutions to such a nonconvex learning formulation have been shown effective to guarantee desirable statistical performance in order to address high dimensionality (Zhang & Zhang, 2012).
Nonetheless, generating a global solution admits no pseudo polynomial-time algorithm, unless “P = NP”;
Indeed, global optimality is shown strongly NP-hard to achieve by (Chen et al., 2017; Huo & Chen, 2010) while Chen et al.
(2014) show similar results for several related problems in regularized minimization.
In contrast to the existing pessimistic result, we derive herein a fully polynomial-time randomized approximation scheme (FPRAS, as deﬁned in 4) that theoretically ensures global minimality to 2 with high probability.
Speciﬁcally, consider a high-dimensional generalized linear model (GLM) as follows.
Let X =
( x 1 , ..., x n ) (cid:124) be the n × p design matrix with x i =
( x i 1 , ..., x ip ) (cid:124) , i = 1 , ..., n , and Y =
( y 1 , ..., y n ) (cid:124) be the n -dimensional response vector.
We will assume the design matrix X is ﬁxed, while the mean of the response is given by E
[ y i ]
= ψ (cid:48)
( x (cid:62) i β true ) for some known link function ψ : Θ → (cid:60) , where Θ ⊆ (cid:60) and β true =
( β true, ..., β truep ) is the unknown vector of true parameters of the model.
Such a setup can be seen as a generalization of linear regression models with the link function allowing for nonlinear transformations that enable a more ﬂexible approach to model estimation.
The high-dimensional regression problem is to estimate β true given knowledge of X , Y , and ψ in the undesirable scenario where p (cid:29) n > 0 .
To that end, traditional statistical learning schemes often resort to the following formulation:
[Equation]
which, according to traditional statistical theories, would result in over ﬁtting in general under the high-dimensional setting.
To resolve over ﬁtting, modern statistical theories favor a modiﬁed formulation as below:
[Equation]
where P λ ( | · | ) is sparsity-inducing regularization term that penalizes any nonzero dimensions in the minimizer, and λ > 0 is a tuning parameter.
Under the assumption that the true regression parameter β true is sparse, a global optimizer to equation 2 has been shown effective to address over ﬁtting for many choices of speciﬁc regularization functions P λ .
Indeed, one of the most successful choices of P λ is the much studied Lasso-based regularized (Tibshirani, 1996), aka, the (cid:96) 1 (-norm) penalty, which has been demonstrated to entail desirable statistical properties (Bickel et al., 2009;
Negahban et al., 2012).
Another admirable property of the Lasso is that, especially when applied to least squares linear regression, it yields an extremely tractable problem via a variety of algorithms (Friedman et al., 2008; 2010).
However, per (Zhao & Yu, 2006; Fan & Li, 2001), Lasso is not selection consistent without a strong irrepresentable condition and may sometimes introduce nontrivial estimation bias.
As a popular alternative to Lasso, the folded concave penalty (FCP) was ﬁrst introduced by (Fan & Li, 2001).
There are mainstream examples of FCP functions, including the SCAD by (Fan & Li, 2001) and MCP by (Zhang, 2010).
This paper will focus on MCP, deﬁned as P λ ( | t | ) =
(cid:82) | t | 0 ( aλ − s ) + a ds for some ﬁxed parameter a > 0 .
In contrast to the Lasso, the FCP regulaizations achieve variable selection consistency non-contingent on the irrepresentable condition and is demonstrated to be unbiased (Fan & Li, 2001).
Furthermore, (Zhang & Zhang, 2012) demonstrated that the global solution to the FCP-regularized formulation leads to desirable recovery of the oracle solution.
Nonetheless, FCP problems are signiﬁcantly harder to solve than Lasso, the new penalty term moves the problem outside the realm of convex optimization, (Chen et al., 2017) even showed that any estimation problem with convex loss and folded concave regularization to be strongly NP-hard, ruling out the possibility of a pseudo-polynomial-time global optimization algorithm.
(Liu et al., 2016) were seemingly the ﬁrst to propose a global approach to the problem called MIPGO which reformulates the problem into a mixed integer program.
Yet, the worst-case complexity of MIPGO is in exponential time.
Perhaps for this reason, current literature tends to focus on local algorithms for the FCP-regularized learning problems.
The local quadratic approximation algorithm by (Fan & Li, 2001) is an example of a majorization minimization algorithm, an approach which is also related to the local linear approximation (LLA) algorithm proposed by (Zou & Li, 2008).
LLA was further explored by (Fan et al., 2014b) showing the oracle property can be obtained with high probability despite the local approach.
(Mazumder et al., 2011; Fan & Lv, 2011) demonstrate coordinate optimization approaches for FCP while (Wang et al., 2014) used an approximate regularization path-following algorithm to obtain the optimal convergence rate to statistically desirable local solution.
(Wang et al., 2013) analyzed the CCCP algorithm and showed under certain conditions that it asymptotically ﬁnds the oracle estimator.
(Liu et al., 2017) took an algorithm agnostic approach by analyzing local solutions satisfying second order KKT conditions and showed desirable statistical properties like recovering the oracle solution and sparisty.
These results discussed above primarily relate to FCP-regularized linear regression, a special case of GLM where ψ is speciﬁcally the identity function.
For analyses which encompass GLM’s with FCP regularizers, (Fan & Lv, 2011) showed that GLM’s, even in ultra high dimensional variable selection problems, have oracle properties when using FCP regularization and demonstrated a coordinate optimization algorithm for ﬁnding local solutions.
In the area of Mestimators, which is a further generalization of our estimation method beyond even GLMs, (Loh &
Wainwright, 2013; Loh et al., 2017a) showed that under certain conditions all local solutions must be within statistical precision of the true parameter and its support while (Loh et al., 2017b) demonstrate a two-step algorithm involving composite gradient descent to ﬁnd a local solution.
Bian &
Chen (2014) demonstrate a optimality conditions for a class of nonconvex optimization problems using nonlipscitz regularization.
From the numerous results pertaining to local solution schemes above, our research question is why local solutions are repetitively successful.
In other words, are there certain geometric properties of the learning formulation equation 2 with FCP that allow all local schemes to perform well independent of the speciﬁc designs of the algorithmic procedures?
Our answer to this question is afﬁrmative; we show herein that all local solutions within an efﬁciently achievable sub-level set are actually globally optimal.
Those local solutions are characterized by the signiﬁcant subspace second-order necessary conditions (S 3 ONC) provable admit FPRAS’s.
The S 3 ONC are weaker conditions than the standard second-order KKT conditions.
As per this result, all S 3 ONC-guaranteeing algorithms (which include a large spectrum of local algorithms) belong to the class of FPRAS’s for global optimization of the strongly NP-hard FCP-based learning problem.
We subsequently develop theories for two speciﬁc algorithms of this type: one gradient-based method and the other is the same as the
LLA.
It is worth noting that (Zhang, 2010) provides conditions to establish the uniqueness of local solutions to FCP-based learning.
When local solutions are unique, then any local optimization algorithms would ensure global optimality.
However, a few critical assumptions are necessary to achieve the uniqueness result and, furthermore, many report numerical experiments, e.g., those in (Fan et al., 2014b; Liu et al., 2017; 2016; Fan & Li, 2001) indicate the non-uniqueness of local solutions, instead.
In contrast, our results in this paper imposes only standard assumptions commonly shared by a ﬂexible set of high-dimensional GLMs and are applicable even if the local solutions are non-unique.
To our knowledge, this is the ﬁrst geometric proof that global solutions coincide with pseudo-polynomial-time computable local solutions in an FCP-based regression formulation with high probability.
The resulting algorithms are the ﬁrst few FPRAS’s to this problem.
Two works with notable relations to our own are Liu et al.
and Liu & Ye (2019).
The ﬁrst applies a similar analytical framework to linear regression problems however our generalization to
GLMs adds signiﬁcant ﬂexibility and it was unknown for their result that the oracle solution implies global optimality since it was only as of Zhang & Zhang (2012) that global optimality was known to potentially imply the oracle solution.
Further, it is nontrivial to extend their existing result to global optimal results.
(Liu & Ye, 2019) on the other hand is a more general setup than our own though the tradeoff is that our rate is better and S 3 ONC solutions do not ensure global optimality to the in-sample training error for their setup.
The rest of this paper is organized as follows.
Section 2 goes through speciﬁc problem assumptions and explains the S 3 ONC.
Section 3 contains our main result for global optimality and uses it to make additional claims for LLA.
Section 4 contains numerical results to verify our theoretical ﬁndings.
Section 5 provides concluding remarks.
In this paper we will use (cid:107)·(cid:107) 0 to denote the number of nonzero entries, | · | to denote the (cid:96) 1 -norm if the argument is a vector, or cardinality if the argument is a set, (cid:107)·(cid:107) to denote the (cid:96) 2 -norm, (cid:107)·(cid:107) max to denote the maximum norm and (cid:107)·(cid:107) min to denote the absolute value of the entry with the smallest magnitude.
( · ) + is used equivalently to max(0 , · ) .
For any vector v , v Q is intended as ( v j : j ∈ Q ) .
For any set Q , we denote the complement as Q c .
In particular, let S be the true support set, that is, S :=
{ j : β truej (cid:54) = 0 } and its complement is S c .
We occasionally use the term the “oracle solution” to refer to the solution β oracle deﬁned as β oracle ∈
[Equation]
L ( β ) .
The oracle solution is a hypothetical solution which assumes prior knowledge of the true support set S and thus can be considered a theoretical benchmark.
S ETUPS , PRELIMINARIES , AND ASSUMPTIONS 2. SETUPS AND ASSUMPTIONS
Our analysis will focus on sparse GLMs that have a ﬁxed design matrix and satisfy the following assumptions:
(A1) Assume that
(i) b u ≥ ψ (cid:48)(cid:48) ( x (cid:124) i β ) ≥ b l > 0 for all x (cid:124) i β ∈ Θ ; (ii) There exists K > 0 such that the design matrix satisﬁes 1 n (cid:107) X j (cid:107) 2 < K for all j ∈ [ p ] .
Let the tuning parameter a in P λ satisfy K < ( b u a ) − 1 .
(A2)
The vector of residuals W ∈ (cid:60) n such that W :=
y − E
[ y | X ] is subgaussian( σ ) which means it satisﬁes that P [ |(cid:104) W, v (cid:105)| ≥ t ] ≤ 2 exp ( − t 2 / 2 σ 2 ) , for all v : (cid:107) v (cid:107) = 1 and any t > 0 ; (A3)
There exists a sequence { r d ≥ 0 :
d = 1 , 2 , ..., p } such that the following are satisﬁed: (i) For any d 1 , d 2 : 1 ≤ d 1 ≤ d 2 ≤ p , we have r d 1 ≥ r d 2 ; (ii) There exists some ˜ p ∗ : 2 |S| ≤ ˜ p ∗ ≤ p such that r ˜ p ∗ > 0 ; (iii) For all d : 1 ≤ d ≤ p , it holds that n − 1 (cid:107) Xβ (cid:107) 2 ≥ r d (cid:107) β (cid:107) 2 for any β ∈ (cid:60) p : (cid:107) β (cid:107) 0 ≤ d .
Remark 1.
Part (i) of (A1) states that our link function is both strongly convex and continuously differentiable; that is, the gradient being Lipschitz continuous.
Many types of traditional GLM problems satisfy this constraint including those for normal (linear regression), categorical (logistic regression), binomial, gamma and Poisson distributions, although in some cases the mild assumption on the boundedness of Θ has to be made.
Even though the original domain of the link function can be unbounded, one may still consider its bounded subset given that it contains the vector of true parameters.
Part (ii) of (A1) can be assumed without loss of generality by normalizing the design matrix columns.
Remark 2.
(A2) is a common assumption in the literature, for example (Negahban et al., 2012)
(Wang et al., 2013).
Remark 3.
Both (i) and (A2) are satisﬁed by a number of GLM setups, one example is linear regression.
In such a setup, the response Y takes a gaussian distribution, while the gradient of the link function (encoding E [ Y | X ] ) is the identity.
Note, it is difﬁcult to have one without the other, since the loss formulation 1 is simply a log-likelihood maximization applied to a distribution within the class of exponential dispersion models (Jørgensen, 1987).
Another classic example is logistic regression which is used for a Bernoulli or binomial distributed Y along with a logit link function.
It should be noted that we treat the matrix X as ﬁxed, so its generative distribution is not important to the analysis outside of whether it satisﬁes the assumptions and constraints mentioned.
In the numerical experiments in Section 4, we use i.i.d.
gaussian generation method since, as discussed for Deﬁnition 1, it means our design matrices will satisfy (A3) with high probability.
Remark 4.
Assumption (A3) can be understood to be a lower bound on the eigenvalues for principal sub-matrices of X (cid:124)
X of dimension d × d for all d ∈ [ p ] .
For every d : d ≤ ˜ p ∗ , the lower bounds are positive, meaning that the smallest eigenvalues of the d × d principal sub-matrices are assumed positive.
According to (Liu et al., 2017), Assumption (A3), for certain parameters, is provably a weaker condition than the restricted eigenvalue (RE) condition, as deﬁned in Deﬁnition 1 below and ﬁrst introduced by (Bickel et al., 2009) as a plausible assumption to allow for the desired recovery quality of Lasso.
The RE condition is a common assumption in the high-dimensional learning literature, such as (Zhang et al., 2014) and (Fan et al., 2014b).
Deﬁnition 1.
(RE condition (Zhang et al., 2014))
The matrix X ∈ (cid:60) n × p is said to satisfy the RE condition if, for some r e > 0 , it holds that 1 n (cid:107) Xδ (cid:107) 2 ≥ r e (cid:107) δ (cid:107) 2 for all δ ∈ (cid:83)
| ˆ S | =
s C ( ˆ S )
where
C ( ˆ S ) :
= { δ :=
( δ i ) ∈ (cid:60) p :
|
δ ˆ S c | ≤ 3 | δ ˆ S |} , δ ˆ S c :=
( δ j : j ∈ ˆ S c ) , and δ ˆ S :=
( δ j : j ∈ ˆ S ) .
Furthermore, the largest possible r e is said to be the restricted eigenvalue constant of X .
Random design matrices with with i.i.d.
rows generated following subgaussian distributions as in (A2) have been shown to satisfy the RE condition with high probability (Zhou, 2009) while proposition 1 in Loh & Wainwright (2013) includes a proof that with high probability, restricted strong convexity (RSC) is satisﬁed for a setup equivalent to our own.
Note that satisfaction RSC implies the RE condition above.
Thus, within our setup, (A3) is also satisﬁed with high probability for our setting.
2. P RELIMINARIES ON S 3 ONC
Our results focus on the S 3 ONC solutions, which has been formerly introduced by (Liu et al., 2017) in the special case of high-dimensional linear regression as a relaxation of the standard second-order
KKT conditions.
The deﬁnition of S 3 ONC depends on the notion of ﬁrst order necessary conditions (FONC) as below.
Deﬁnition 2 (FONC) .
A solution β ∗ satisﬁes the ﬁrst order necessary conditions (FONC) if
[Equation]
where ∂ ( | · | ) denotes the subdifferential of | · | .
Deﬁnition 3 (S 3 ONC) .
A solution β ∗ satisﬁes the signiﬁcant subspace second-order necessary condition (S 3 ONC) if it satisﬁes FONC and for all j ∈ { j : β ∗ j (cid:54) = 0 } ,
[Equation]
if the second derivative exists.
Remark 5.
The S 3 ONC can be intuited as the second order necessary condition applied only to the dimensions where β j (cid:54) = 0 , i.e., the signiﬁcant dimensions.
Since the S 3 ONC is weaker than the standard second-order KKT conditions, any algorithm that guarantees the second-order KKT conditions can be used to obtain an S 3 ONC solution, by requiring a more stringent optimality condition, may be slower than necessary.
One speciﬁcally S 3 ONC guaranteeing approach, presented in (Liu & Ye, 2019), utilizes an interior point trust region algorithm in order to guarantee an S 3 ONC solution in polynomial time.
This is the scheme which will be used later in Section 4.
M AIN RESULTS
We now present our theoretical results for global optimization of FCP penalized GLMs.
All proofs can be found in the appendix.
We will make use of a short-hand notation:
[Equation]
Theorem 1.
Suppose assumptions (A1), (A2), and (A3) with any ˜ p ∗ : ˜ p ∗ ≥ 2 |S| .
Let β ∗ be an arbitrary S 3 ONC solution to equation 2 with P λ speciﬁed as the MCP.
Assume that Q ( β ∗ ) ≤
Q ( β true )
+ Γ for an arbitrary Γ ≥ 0 .
(i) Let the sub-optimality gap satisﬁes Γ < P λ ( aλ ) − σ
[Equation]
(cid:1)
[Equation]
ii) choose P
[Equation]
and (iii) assume that the minimal signal strength satisfy
[Equation]
then the following two statements hold
(a) β ∗ is an oracle solution with probability at least 1 − exp( − t + ˜
p ∗ ln( pe ˜ p ∗ ))
− exp( − (˜ p ∗ + 1)( t (cid:48) − ln p )) · 1 − exp( − ( p − ˜ p ∗ )
( t (cid:48) − ln p ))
1 − exp( − t (cid:48) +ln p ) .
(b) β ∗ is both an oracle solution and an globally optimal
solution to equation 2 with probability
[Equation]
Remark 6.
Theorem 1 (especially in the second statement) is perhaps the ﬁrst result that establishes a set of conditions for any S 3 ONC solution to be globally optimal with high probability.
Further, this result is algorithm independent which allows for greater ﬂexibility compared to most existing results as in (Loh et al., 2017b) and (Fan & Lv, 2011) which rely on a speciﬁc algorithm choice.
Remark 7.
The second part follows quite easily from the ﬁrst due to the uniqueness of β oracle as well as the fact that β opt must also be an S 3 ONC solution.
Thus by applying the ﬁrst part of the
Theorem to β opt we are able to show that both our arbitrary β ∗ and β opt coincide with the unique β oracle .
Remark 8.
The above constraints on Γ , P λ ( aλ ) and (cid:107) β true S (cid:107) min may initially seem disparate but can all be converted to constraints on the sample size n as is shown in Corollary 1 below.
This is possible because Γ can be bounded by some function of n − γ for some γ > 0 .
Given that, it can be seen that the lesser side of inequalities (i),(ii) and (iii) go to 0 as n grows.
Further discussion of how this is achieved for Corollary 1 can be found in Remark 12.
Corollary 1.
Assume ln p ≥ 1 , b l ≤ 1 , and s ≥ 1 .
Let β ∗ be an S 3 ONC solution to equation 2.
Let assumptions (A1), (A2), and the RE condition as deﬁned in Deﬁnition 1 hold.
Assume that
Q ( β ∗ ) ≤ Q ( β Lasso ) almost surely, where β Lasso is the optimal solution to the Lasso problem with penalty coefﬁcient λ Lasso = σ (cid:113) lnp n 1 − γ/ 2 where γ ∈ [0 , 1] is an arbitrary scalar.
Let λ = σr e (cid:113) ln p n γ/and a ∈ [ . 8 , 1) .
There exist problem independent constants C 1 > 0 , C 2 > 0 and C 2 > 0 such that if
[Equation]
Then β ∗ is the global solution to 2 with probability at least 1 − C 4 exp( − C 5 sn γ/ 2 ln p ) −
C 6 exp( − C 7 b u n γ/
ln
( p ))
for problem independent constants C 4 , C 5 , C 6 and C
Remark 9.
Corollary 1 indicates that for γ > 0 , the global optimal solution coincides with computable S 3 ONC solution with overwhelming probability given that the sample size meets certain requirements.
It should speciﬁcally be noted that the relationship between n and p require only ln p n γ/
= O (1) , which ensures the applicability to the high-dimensional setting even if n (cid:28) p .
Remark 10.
(Liu & Ye, 2019) has derived a gradient-based algorithm that provably ensures an
S 3 ONC solution at pseudo-polynomial-time complexity.
When n is properly large, this pseudopolynomial-time algorithm enables a straightforward design of an FPRAS for generating the global optimal solution as follows.
FPRAS:
A pseudo-polynomial-time algorithm that generates global optimum at high probability
Step 1.
Initialize the parameters δ, λ, a, ˆ a, k = 0 and β Lasso by solving equation
Step 2.
If Case 1: | β kj | ∈ (0 , aλ ) for some j = 1 , ..., p , then choose an arbitrary ι ∈ { j : | β kj ∈ (0 , aλ ) } and solve
[Equation]
[-+
-]and let β k +1 j = β kj ∀
j (cid:54) =
ι .
Go to Step 3.
Else Case 2: If — β kj / ∈ (0 , aλ ) for all j = 1 , ..., p then for all j = 1 , ..., p : — If β k j = 0 then β k +1 j
= ˆ a · (cid:2) | [ ∇L ( β k )]
j | − λ (cid:3) + · sign ( − [ ∇L ( β k )] j ) .
— If | β kj | ≥ aλ , then β k +1 j = β kj − ˆ a · [ ∇L ( β k )]
j .
Go to Step 3.
Step 3.
Algorithm stops if | β kj | / ∈ (0 , aλ ) and (cid:13)(cid:13)
β kj − β k
+
j
(cid:13)(cid:13) <
δ .
Otherwise, let k :
=
k + 1 and go to Step 2.
Remark 11.
As shown by (Liu & Ye, 2019), the above algorithm has time complexity in
O (cid:16)(cid:0) Q ( β Lasso ) − Q ( β opt ) (cid:1) · max { (1 / (2 a )
− b u / 2) − 1 , 2 b − 1 u , (1 /a − b u / 2) − 1 } · 1 /δ 2 (cid:17) which can be seen as a pseudo-polynomial time complexity.
Note that the objective function terms are polynomially dependent on the problem dimensionality and input.
Deﬁnition 4.
(FPRAS) Let f : L → (cid:60) +0 be a function representing the optimal solution to a problem.
Let A be a probabilistic algorithm, which takes as an input an instance of the problem x ∈ L , and a parameter (cid:15) > 0 .
We call A a fully polynomial randomized approximation scheme, if it has the following properties
• the algorithm returns a value within the required precision with probability at least 1 / 2 .
P (cid:2) |A
( x, (cid:15) )
− f ( x ) | ≤ (cid:15)f ( x ) (cid:3) > 1 /
• the running time of A is polynomial in both the size of the input x and the precision 1 /(cid:15)
Note that the speciﬁc probability bound is fairly arbitrary as long as it is a constant probability bounded away from 1/2 (Vazirani, 2013).
Remark 12.
We are able to remove Γ from the result by bounding the performance difference between β true and β lasso using similar techniques as in Bickel et al.
In order to use this bound for our S 3 ONC solution, we require that Q ( β ∗ ) ≤ Q ( β Lasso ) .
However, this can generally be obtained by initializing any S 3 ONC guaranteeing algorithm with β Lasso in a similar fashion to (Fan et al., 2014b) for LLA.
The FPRAS above follows the same initialization scheme.
Remark 13.
The above speciﬁcation of values for a, λ and λ Lasso can be thought of as examples rather than strict requirements.
A closer examination of the proof for Corollary 1 will reveal that the values for λ and λ Lasso can be chosen in a much more ﬂexible fashion, though the corresponding values of C 1 through C 7 may be different for different combinations of λ and λ Lasso .
The techniques used in the proof of Theorem 1 can be used to provide insights into other optimization schemes.
As an example, we can apply the same analysis to the state-of-the-art FCP-based algorithm, LLA, using the framework in (Fan et al., 2014b) as a starting point.
LLA: local linear approximation.
Step 1.
Set k = 0 .
Initialize the algorithm with β 0 =
β Lasso , where β Lasso is generated by solving equation 5.
Let N be the maximal iteration number.
Step 2.
For all k = 1 , ..., N , solve the following convex program to generate β k +1 :
[Equation]
where P (cid:48) λ is the ﬁrst derivative of P λ .
Let k :
=
k + 1 .
We can show that in fact the LLA is another FPRAS that achieves the global optimal solution.
The proof of this can be found in the appendix.
Corollary 2.
For problem equation 2.
If (cid:107) β true S
[Equation]
min >
[Equation]
a + 1) λ , λ
[Equation]
max { 3 λ Lasso s 1 / 2 b l r e
[Equation]
l
[Equation]
an
[Equation]
[Equation]
[Equation]
} and the RE condition in Deﬁnitionholds, the following holds.
(a) The LLA algorithm initialized with β Lasso converges to the oracle solution in two iterations with probability 1 − φ 0 − φ 1 − φ 2 , where
[Equation]
(b) If in addition (A1) and (A2) holds, while the parameters of ( a, λ ) satisfy that P [Equation] and the and P
[Equation]
aλ ) >
[Equation]
˜ p ∗ + 2 √ ˜ p ∗ t 3 + 2 t 3 (cid:1) and let the minimal signal strength satisfy (cid:107) β true S
[Equation]
p
[Equation]
p
[Equation]
t [Equation] [Equation] l
[Equation]
r [Equation] p |S| , P λ ( aλ ) |S|} then the LLA algorithm initialized by β Lasso converges to the global solution in two iterations with probability at least
[Equation]
where
[Equation]
and t 1 , t 2 , t 3 , t 4 > 0 are arbitrary constants.
Remark 14.
Since each iteration of the LLA solves a convex program, which can be done within polynomial-time.
When n is properly large, the above theorem then indicates that the LLA is another
FPRAS in globally optimizing the FCP-based nonconvex formulation.
N UMERICAL EXPERIMENTS 4. E XPERIMENTAL SETUP
We focus our tests on sparse logistic regression.
Our problem and data are implemented in a similar way as (Fan et al., 2014b).
We construct β true as below: Firstly, β true S is constructed randomly by choosing 10 elements of β and choosing the magnitude of each to be a uniform value within [1 , 2] .
Each value is chosen to be negative with probability .
Then, the remaining entries β true S c are set to be 0.
The design matrix X ∈ (cid:60) n × p is constructed by generating n iterations of x i ∼ N p (0 , Σ) where Σ = ( . 5 | j − j (cid:48) | ) p × p .
We then generate Y using a Bernoulli distribution where P ( y i = 1) =
(1 + e − x (cid:124) i β true )
− 1 .
With this data, we train a logistic regression model by invoking Algorithm 1 in solving equation 2 with MCP for S 3 ONC solutions initialized with Lasso implemented in Python 3.
The tuning parameters λ and a are obtained by cross validation following (Fan et al., 2014b).
We would like to ascertain whether our FCP classiﬁer, obtained using S 3 ONC methods, is actually the global optimal solution.
We do this by taking each element of the FCP classiﬁer and perturbing it to ﬁnd a new potential solution.
Each element’s perturbation is independent and generated by a
N (0 , 1 /p 1 / 2 ))
-random variable.
We then check if this perturbed classiﬁer has better FCP regularized performance on the training data than the FCP classiﬁer.
If not, we repeat until either a better solution is found, or until 2000 perturbations have been tried.
Additionally, we compare our solution’s statistical performance to those of other popular regularization methods.
Using the data generation method above, we obtain two sets of data, both withsamples.
One set is for training the model, and the other is the test set for out-of-sample tests.
We repeat the above process for 100 times to generate 100 training-and-test instances, each withsamples.
We compare those trained using the method described above with Lasso solutions generated by the global minimizer to equation 5 and an estimator generated by solving equation 2 when
P λ is substantiated by an (cid:96) 2 penalty.
The Lasso and (cid:96) 2 classiﬁers are solved using the scikit learn python library.
We compare the above estimators in terms of statistical performance for both (cid:96) 1 loss: | β ∗ − β true | and (cid:96) 2 loss: (cid:107) β ∗ − β true (cid:107) .
4. N UMERICAL RESULTS
n =n =n =n =
[Table]
=
[Table]
=
[Table]
=
[Table]
Table 1 contains the numbers from optimality analysis.
This technique did not yield a single perturbed solution that could beat the FCP classiﬁer obtained from the FPRAS in any of our thousands of iterations.
Table 2 shows the numerical results for the statistical performance measurements.
We show the two performance measures for each of the three classiﬁers for tphree different problem types.
As expected, the FCP classiﬁer generally outperformed the lasso and (cid:96) 2 classiﬁers.
The margins are fairly thin between FCP and lasso, especially compared to the standard deviation.
Other values of n and p were tried but the results generally followed the same pattern.
As a result we tentatively conclude that our numerical results align with our theoretical results though further testing of the global optimality probability would be valuable.
[Table]
C ONCLUSIONS
This paper investigates both the theoretical and empirical performance of FPRAS’s on MCP regularized GLMs.
Despite such a problem being strongly NP-Hard, we have shown two FPRAS that achieve global optimality.
To our knowledge this is the ﬁrst probability bound for global optimization of FCP regularized GLMs using an FPRAS.
Further, the same technique can be used to extend other results in order to obtain global optimization bounds for a wide variety of problems.
Though this paper focuses on GLMs
[Table]
further exploration will focus on the question whether similar results can be found for more general problem classes under weaker assumptions.
High-dimensional
M-estimation problems could potentially be a future avenue of investigation
A A PPENDIX
The Appendix is organized as below: Section A.1 presents the proofs for the main results, Sections
A.2 and A.3 present central lemmata to be useful in Section A.1.
A. P ROOF O F MAIN RESULTS
A useful relationship in our proofs is that, for an S 3 ONC solution β ∗ within { β ∗ : Q ( β ∗ ) ≤
Q ( β true ) + Γ } for any Γ ≥ 0 , we have the following useful inequality under Assumption (A1):
[Equation]
where δ ∗ = β ∗ − β true .
This is obtained by invoking the strong convexity of ψ , which leads to
[Equation]
Proof of Theorem 1.
First, given our assumption that (A1) holds, that (i) ˜
p ∗ ≥ 2 |S| , (ii) β ∗ is
S 3 ONC satisfying Q ( β ∗ ) ≤ Q ( β true ) + Γ for some Γ ≥ 0 , and (iii) P λ ( aλ ) > σ 2 2 nb l (1 + 2 √ t (cid:48)
+
[Equation]
, we can apply Lemma 5 with ˜ p
=
˜
p ∗ .
This means that (cid:107) β ∗ − β true (cid:107) 0 ≤ ˜ p ∗ with probability at least 1 − exp( − (˜ p ∗ + 1)( t (cid:48) − ln p )) · 1 − exp( − ( p − ˜ p ∗ )( t (cid:48) − ln p ))
1 − exp( − t (cid:48) +ln p ) .
From this, given the additional assumption that (A3) holds, we can apply the second part of Lemma 4 with ˜ p
=
˜
p ∗ to get that for any t > 0 , 1 n (cid:107) X
( β ∗ − β true ) (cid:107) 2 ≤
[Equation] [Equation]
n
[Equation]
p
[Equation]
√ ˜ p
[Equation]
+ 8 b l min {
[Equation] [Equation]
(cid:107) β ∗ (cid:107) 0 )
r − 1 ˜ p ∗ , P λ ( aλ )˙
( |S| − (cid:107) β ∗ (cid:107) 0 )
+ Γ } holds with probability at least 1 − exp( − t + ˜
p ∗ ln( pe ˜ p ∗ )) .
Given that for 2 arbitrary sets A and B
P
[Equation]
Therefor they hold simultaneous with probability at least 1 − exp( − t + ˜ p
ln( pe ˜ p ∗
[Equation]
) − exp( − (˜ p ∗ +
[Equation]
The same sequence of arguments can be used to show that β opt also satisﬁes (cid:107) β opt − β true (cid:107) ≤ ˜ p ∗ and 1 n (cid:107) X ( β opt − β true ) (cid:107) 2 ≤ 8 σ 2 b 2 l n (cid:0) ˜
p ∗ + 2 √ ˜
p ∗ t + 2 t (cid:1) + 8 b l min { λ 2 ( |S|−(cid:107) β ∗ (cid:107) 0 )
r − 1 ˜ p ∗ , P λ ( aλ )˙
( |S|− (cid:107) β ∗ (cid:107) 0 )
+Γ } with the same probability.
Using again the union bound and DeMorgan’s law, we say β ∗ and β opt satisfy the above conditions simultaneously with probability 1 − 2 exp( − t + ˜ p ∗ ln
( pe ˜ p ∗ ))
−
[Equation]
exp
[Equation]
+ln p ) .
With this, our Γ assumption and our minimal signal strength assumption, we can apply Lemma 6 to show that β ∗ = β opt with probability at least
[Equation]
Proof of Corollary 1.
First we need to bound Γ .
In order to do this we use the lasso problem
Q lasso ( β ) =
(cid:80) i ∈N (cid:96) ( β, x i , y i )
+ (cid:80) j ∈P λ lasso
| β j | as well as the concavity of MCP over positive values to get the following 2 inequalities
[Equation]
and (cid:88)
[Equation]
We also need 2 results from the proof for φ 0 in Corollary 2 which shows that both | δ (cid:96) S c | ≤ 3 | δ (cid:96) S | and b l n (cid:13)(cid:13)
Xδ (cid:96) (cid:13)(cid:13) 2 ≤ 3 λ lasso | δ (cid:96) S | conditional on A where δ (cid:96) =
β lasso − β true .
Given our restricted eigenvalue assumption (cid:107) Xδ (cid:96) (cid:107) 2 n (cid:107) δ (cid:96) (cid:107) 2 ≥ r e , this can be used to show
[Equation]
δ (cid:96) S (cid:13)(cid:13) 2 (cid:13)(cid:13) δ (cid:96)
S (cid:13)(cid:13) ≤√ s (cid:13)(cid:13) δ (cid:96) (cid:13)(cid:13)(cid:13)(cid:13) δ (cid:96) S (cid:13)(cid:13) ≤
[Equation]
r
[Equation]
n
[Equation]
≤
[Equation]
r
[Equation] [Equation]
|
[Equation]
(cid:96) S
| b l (cid:13)(cid:13) δ (cid:96) S
(cid:13)(cid:13)
≤√ s r e 3 λ lasso √ s (cid:13)(cid:13)
δ (cid:96) S (cid:13)(cid:13)
b l (cid:13)(cid:13)
δ (cid:96) S (cid:13)(cid:13) (12) which means | δ (cid:96) | ≤ 12 λ lasso s b l r e with conditional on A which occurs with probability
at least 1 − 2 p exp( − ( λ lasso ) 2 nb u a 8 σ)
Finally we are able to bound gamma by combining the above
[Equation]
[Equation] [Equation]
≤
[Equation]
Next consider the conditions necessary to apply Theorem 1.
We have assumptions (A1) and (A2) and (A3) per our assumption that the RE condition holds combined with 7.
That leaves the 3 requirements on Γ , P λ ( aλ ) and (cid:107) β true S (cid:107) min .
We will convert each of these to inequalities on n
Utilizing 16 and substituting λ =
Qσr e (cid:113) ln p n γ/ 2 and λ lasso =
(cid:15)σ (cid:113) ln p n 1 − γ/ 2 , where Q, (cid:15) > 0 are arbitrary constants, and setting ˜ p ∗ = 4 s , t = ˜
p ∗ n γ/ 2 ln p , t (cid:48)
=
n γ/ 2 ln p we get the following
[Equation]
For some constants C 1 , C 2 and C
We can then apply Theorem 1 (conditional on A ) substitute our values and simplify to get that β ∗ is the global solution with probability at least
[Equation]
We then use the same technique as in Theorem 1 to combine this number with the probability of A to get the ﬁnal non-conditional probability that β ∗ is the global solution with probability at
[Equation]
For some constants C 4 , C 5 C 6 and C 7 .
Note that these constants, as well as C 1 , C 2 and C 3 , are dependent only on the value of a Q and (cid:15) , as far as problem dependencies are concerned.
Thus given that a Q and (cid:15) are chosen to be any positive constant value, as in the statement of Corollary 1, C 1 through C 7 are problem independent, which is the desired result.
Proof of Corollary 2.
The ﬁrst result is simply Corollary 2 in (Fan et al., 2014b).
If we initialize the
LLA algorithm with β lasso , the solution to LASSO using λ lasso as the LASSO constant, then the
LLA algorithm converges to the oracle solution in 2 iterations with probability 1 − φ 0 − φ 1 − φ 2 .
However we still need to solve for the actual values of φ 0 , φ 1 , φ 2 for GLM.
First consider φ
= P ( (cid:13)(cid:13) β lasso − β true (cid:13)(cid:13) max > a 0 λ ) .
Similar to Lemma B.1.
in Bickel et al.
(2009), to bound this we will start by noticing that for the lasso penalized loss function Q lasso ( β ) =
(cid:80) i ∈N l ( β, x i , y i ) + λ lasso (cid:80)
j ∈P | β
j | we have that Q lasso ( β lasso ) ≤ Q lasso ( β true ) .
If we then let δ (cid:96) =
β lasso − β true we can use the same tactic as in the derivation of 8 to get b l 2 n (cid:13)(cid:13)
Xδ (cid:96) (cid:13)(cid:13) 2 − 1 n W (cid:124)
Xδ
[Equation]
≤ λ lasso (cid:80) j ∈P | β truej | − | β lassoj | , which can then be rearranged to get.
[Equation]
Next let A = (cid:84) j ∈P {| 1 n W (cid:124)
X j | ≤ λ lasso / 2 } .
We can combine this with 25 to get that b l 2 n (cid:13)(cid:13)
Xδ (cid:96) (cid:13)(cid:13) 2 + λ lasso / 2 (cid:80) j ∈P | β lassoj − β truej | ≤ λ lasso (cid:80) j ∈P | β lassoj − β truej | + λ lasso (cid:80) j ∈P | β truej | − | β lassoj | conditional on A .
From this notice that the right term goes to zero when β truej =
so we then have that b l 2 n (cid:13)(cid:13)
Xδ (cid:96) (cid:13)(cid:13) 2 + λ lasso / 2 (cid:80) j ∈P | β lassoj − β truej | ≤ λ lasso (cid:80) j ∈S | β lassoj − β truej | + | β truej | − | β lassoj | .
Using the triangle inequality and the deﬁnition of δ (cid:96) we can simplify this to
[Equation] [Equation]
conditional on A .
By relaxing different parts of the equation, this can be further simpliﬁed to both b l n (cid:13)(cid:13)
Xδ (cid:96) (cid:13)(cid:13) 2 ≤ 3 λ lasso | δ (cid:96) S | ≤ 3 λ lasso s 1 / 2 ||
δ (cid:96) S || 2 and | δ (cid:96) S c | ≤ 3 | δ (cid:96) S | .
Note that the second of these shows that δ (cid:96) satisﬁes the constraint for the RE condition 1.
Therefor we have that (cid:107) Xδ (cid:96) (cid:107) 2 n (cid:107) δ (cid:96) (cid:107) 2 ≥ r e .
If this is combined with the ﬁrst of the two equations, we can get that
[Equation]
(cid:13)(cid:13)
[Equation]
r e ) 1 /conditional on A .
Next, using this we can show that conditional on A we have that (cid:13)(cid:13)
δ (cid:96) (cid:13)(cid:13) max ≤ (cid:13)(cid:13) δ (cid:96) (cid:13)(cid:13) ≤ (cid:13)(cid:13)
Xδ (cid:96) (cid:13)(cid:13) 2 2 / ( (cid:13)(cid:13) δ (cid:96) (cid:13)(cid:13) nr e ) ≤ 3 λ lasso s
[Equation]
2 b l r e < a
[Equation]
if
[Equation]
>
[Equation]
s
[Equation]
This is the inverse of the condition that deﬁnes φ 0 .
Thus, we can bound φ 0 with φ 0 ≤ P ( A c ) =
P ( (cid:83) j ∈P | 1 n W (cid:124)
X j | > λ lasso / 2) =
P ( (cid:83) j ∈P | W (cid:124)
X j | / (cid:107) X j (cid:107) > nλ lasso / (2 (cid:107) X j (cid:107) )) ≤ pP ( |(cid:104) W, v (cid:105)| > λ lasso n 2 (cid:107) X j (cid:107) ) ≤ pP ( |(cid:104) W, v (cid:105)| > λ lasso ( nb u a ) 1 / 2) ≤ 2 p exp − ( λ lasso ) 2 nb u a 8 σwhich uses both (A1)(ii) and (A2) as long as λ > 3 λ lasso s 1 / 2 b l a 0 ( r e ) per (A2).
Next consider φ 1 = P ( (cid:13)(cid:13)(cid:13) (cid:79)
[Table]
[Equation]
where v ∈ (cid:60) n is some vector with (cid:107) v (cid:107) = 1 as indicated in (A2) and δ o = β oracle − β true .
From this, using Demorgan’s law and the union bound, we notice that P ( A + B ≥ C ) ≤ P ( A ≥
C/ 2) + P ( B ≥ C/ 2) which can be used to further simplify
[Equation] [Equation]
We can then simplify both terms individually.
For the ﬁrst term, P
[Equation]
≥ (1 / 2) a 1 λ ( ab u n ) 1 / 2 ) , given the fact that the oracle solution and true solution have the same support, the oracle solution must be in the Γ = 0 level set of the true solution.
Using similar arguments to Lemma 5, we have that b l 2 n (cid:107)
Xδ o (cid:107) 2 ≤ 1 n W (cid:124)
Xδ o From here Lemma 2 can be applied since we know (cid:13)(cid:13) β oracle − β true (cid:13)(cid:13) 0 ≤ s .
With some simpliﬁcation this gives that (cid:107)
Xδ o (cid:107) ≤
[Equation]
Utilizing Lemma 3 with s in place of ˜ p shows that
P
[Equation]
≤ ( pes ) s exp( − t 1 ) .
This is the ﬁrst half of φ 1 as long as (1 / 2) a 1 λ ( an/b u ) 1 / 2 ≥ 2 b l σ (cid:112) s + 2 √ st + 2 t which is equivalent to the assumed condition λ ≥ 4 σ √ s +2 √ st 1 +2 tb l a 1 ( an/b u ) 1 /
Next, the second term can be easily bounded using (A2):
P
[Equation] [Equation]
Therefor φ 1 ≤ ( pes ) s exp(
[Equation]
1 ) + 2 exp(
[Equation]
a
[Equation]
2 ab u n 8 σ)
Next consider φ 2 =
P
(cid:13)(cid:13) β oracle S (cid:13)(cid:13) min ≤ aλ )
First, given the assumption (cid:13)(cid:13) β true ] (cid:13)(cid:13) min > ( a + 1) λ we can see that φ 2 = P ( (cid:13) (cid:13) β oracle S (cid:13) (cid:13) min ≤ aλ ) ≤ P ( (cid:13) (cid:13) β oracle S − β true S (cid:13) (cid:13) max > λ ) ≤
P ( (cid:13)(cid:13) β oracle − β true (cid:13)(cid:13) 2 > λ ) = P ( (cid:107) δ o (cid:107) 2 > λ ) .
Next, since we know that the support of β oracle and β true is S , we know that | δ o S c | = 0 ≤ 3 | δ o S | which is the constraint for the RE condition.
Therefor we know that (cid:107)
Xδ o (cid:107) 2 n (cid:107) δ o (cid:107) 2 ≥ r e .
With this and a similar line of argument as in φ 1 we get that φ 2 ≤ P ( (cid:107) δ o (cid:107) > λ ) ≤ P ( (cid:107)
Xδ o (cid:107) > λ √ nr e ) ≤ P ( 2 b l (max S ˜ p : | S ˜ p
| = s (cid:13)(cid:13)(cid:13) ˜
U (cid:124) S
˜ p W (cid:13)(cid:13)(cid:13) > λ √ nr e ) =
[Equation]
) assuming that
[Equation]
≥
[Equation]
st + 2 t which is equivalent to the condition λ ≥ 2 σ
[Equation]
This, combined with the fact that for MCP, a 0 = a 1 = a 2 = 1 shows the ﬁrst result.
The second result can be seen by ﬁrst noting all assumptions of Theorem 1 part 2 are satisﬁed, where (A3) with r 4 s is implied by 7.
Thus by using the same arguments as in Theorem 1 partwhich shows that the oracle solution is unique and that the global solution is the oracle solution with some probability, since the global solution is almost surely S 3 ONC with Γ = 0 .
If we use t = t 3 and t (cid:48)
=
t 4 we get that the probability that the global solution is not the oracle solution as
[Equation]
This combined with the ﬁrst result shows that the LLA algorithm converges to the global solution in 2 iterations with probability 1 − φ 0 − φ 1 − φ 2 − φ 3 which is the second result.
A. C ENTRAL LEMMAS AND THEIR PROOFS
Lemma 1.
Let β ∗ be a S 3 ONC solution to 2.
If assumption (A1) holds, then
P
[Equation] [Equation]
] = 1 .
Proof of Lemma 1.
First, deﬁne events γ j and δ j as
[Equation]
First, for any given j ∈ P , we solve for P [ γ j ∩ δ j ] given our assumptions.
We can start with ∂ 2 Q ( β ) ( ∂β j
[Equation]
2 (cid:12)(cid:12)(cid:12) β = β ∗ ≥ 0 which gives us 1 /n n (cid:80) i =1 ψ (cid:48)(cid:48)
( x (cid:124) i β ∗ ) x 2 i,j + P (cid:48)(cid:48) λ ( | β ∗ j | ) ≥ 0 .
We can rearrange this to get b u n (cid:80)
i
[Equation] [Equation]
≥ n
[Equation]
i
[Equation]
( x (cid:124) i β
[Equation]
) x 2 i
[Equation]
j ≥ − nP (cid:48)(cid:48) λ ( | β ∗ | ) =
n/a where we get the leftmost inequality from assumption (A1) part (i) and the rightmost equality from the deﬁnition of MCP.
More concisely we have that b u (cid:107) X j (cid:107) 2 ≥ n/a which contradicts (A1) part (ii).
Therefor we know P [ γ j ∩ δ j ] = 0 .
It should also be noted that P [ γ cj ] = 0 since β ∗ satisﬁes S 3 ONC conditions.
Thus, by applying
Demorgan’s law and then the union bound, it can be obtained that
[Equation]
We can then apply this result to all indices to get that P [ δ j ] = 0 for all j ∈ { 1 , 2 , ..., p } , which is the desired result.
Lemma 2.
Consider an arbitrary S 3 ONC solution β ∗ to 2 with MCP.
Given the event that for some integer
[Equation]
, then
[Equation]
a
[Equation]
Where
[Equation] [Equation]
else and U S ˜ p ∈ (cid:60) n × ˜ p is deﬁned as in the following Thin SVD: X S ˜ p =
U S ˜ p D S
˜ p V S ˜ p .
Proof.
Denote δ ∗ :=
( δ ∗ j )
=
β ∗ − β true , S ˜ p :=
( j : δ ∗ j (cid:54) = 0) ⊆
P , δ ∗ S ˜ p :=
( δ ∗ j : j ∈ S ˜ p ) and
X S ˜ p :=
( x ij : i ∈ N , j ∈ S ˜ p ) .
By assumption, we know that (cid:107) δ ∗ (cid:107) 0 ≤ | S ˜ p | = ˜ p .
First decompose X S ˜ p using Thin SVD to get X S ˜ p
=
U S ˜ p D S
˜ p V S ˜ p where U S ˜ p ∈ (cid:60) n × ˜ p .
Note that since and U (cid:124) S
˜ p
U S ˜ p
=
I we have that for any υ ∈ (cid:60) ˜
p we have (cid:13)(cid:13) D S
˜ p V S ˜ p
υ (cid:13)(cid:13) 2 =
[Equation]
therefor we can obtain that
[Equation]
Where
[Equation]
else
[Equation]
Lemma 3.
Consider an arbitrary S 3 ONC solution β ∗ to 2 with MCP.
If (A2) holds, then for some integer ˜
[Equation]
=˜
p (cid:13)(cid:13)(cid:13) ˜
U (cid:124) S
˜ p W (cid:13)(cid:13)(cid:13) ≤ σ (cid:112)
˜ p + 2 √ ˜ pt + 2 t (cid:105) ≥ 1 − ( pe ˜ p )
˜ p exp( − t ) .
Where ( ˜ U S ˜ p ) i,j := (cid:26) U S ˜ p , if j ∈ S ˜ p 0 , else and U S ˜ p ∈ (cid:60) n × ˜ p is deﬁned as in the following Thin SVD: X S ˜ p =
U S ˜ p D S
˜ p V S ˜ p .
Proof.
We attempt to bound (cid:16) max S ˜ p : | S ˜ p
| =
˜ p (cid:13)(cid:13)(cid:13) ˜
U (cid:124) S ˜
p W (cid:13)(cid:13)(cid:13)(cid:17) .
Given that we now have W multiplied by a square matrix, we can apply Lemma 9.
In the Lemma, let Σ u = ˜
U S ˜ p
˜ U (cid:124) S ˜ p .
The fact that Σ u Σ u = Σ u means that Σ u is an idempotent matrix with (cid:107) Σ u (cid:107) ≤ 1 and Tr (Σ u ) = rank (Σ u ) ≤ rank ( ˜ U S ˜ p ) ≤ rank ( U S ˜ p ) ≤ ˜ p .
Lemma 9 then states that that P (cid:104)(cid:13)(cid:13)(cid:13)
˜ U (cid:124) S
˜ p W (cid:13)(cid:13)(cid:13) ≤ σ (cid:112)
˜ p + 2 √ ˜ pt + 2 t (cid:105) ≥ 1 − exp( − t ) .
From this we can show that
[Equation]
Where the ﬁrst inequality can seen by noting that if η k ∈ (cid:60) k is a sequence of i.i.d random variables and θ ∈ (cid:60) is a scalar, by applying De Morgan’s Law and then using the union bound, it can be obtained that P [max k ∈ K η k ≤ θ ] =
P [ (cid:84) k ∈ K η k ≤ θ ] = 1 − P [ (cid:83) k ∈ K η s ≥ θ ] ≥ 1 − (cid:80) k ∈ K P [ η k ≥ θ ] = 1 − | K | (1 − P [ η k ≤ θ ]) which yields the same inequality as in 44.
This is the desired result.
Lemma 4.
Consider an arbitrary S 3 ONC solution β ∗ to 2 with MCP.
Let Assumptions (A1) and (A2) hold.
Given the simultaneous occurrence of (i) the event that Q ( β ∗ ) ≤ Q ( β true ) + Γ holds for some Γ ≥ 0 ; (ii) the event that for some integer ˜ p : (cid:107) β ∗ − β true (cid:107) 0 ≤ ˜ p .
Then for any t > 0 , 1 n (cid:107) X ( β ∗ − β true ) (cid:107) 2 ≤ 4 σ 2 b 2 l n (˜ p +2 √ ˜
pt +2 t )
+ 8 b l min { (cid:80) j ∈ S P (cid:48)
λ
( | β ∗ j | ) | β ∗ j | , P λ ( aλ )( |S|−(cid:107) β ∗ (cid:107) 0 )
+ Γ } holds with probability at least 1 − exp( − t
+ ˜ p
ln( pe ˜ p )) .
If in addition (A3) holds with ˜ p ∗ ≥ ˜ p , then 1 n (cid:107) X
( β ∗ − β true ) (cid:107) 2 ≤ 8 σ 2 b 2 l
2 √ pt +
+
Proof.
First, denote δ ∗ :=
( δ ∗ j )
=
β ∗ − β true , S ˜ p :=
( j : δ ∗ j (cid:54) = 0) ⊆
P , δ ∗ S ˜ p :=
( δ ∗ j : j ∈ S ˜ p ) and
X S ˜ p :=
( x ij : i ∈ N , j ∈ S ˜ p ) .
By assumption, we know that (cid:107) δ ∗ (cid:107) 0 ≤ | S ˜ p | = ˜ p .
Further, let us denote
[Equation]
We now start to deﬁne the desired bound by applying the second part of Lemma 8.
The result simpliﬁed using the above deﬁnitions becomes
[Equation]
Next, since all assumptions for Lemma 1 are satisﬁed
[Equation]
we can apply it to get
[Equation]
We can then complete the square, solving for 1 √ n (cid:107) Xδ ∗ (cid:107) to get
[Equation]
where the last inequality holds due to the value inside the square root being larger than the term outside.
From here, squaring both sides gives us
[Equation]
Finally, by applying the second part of Lemma 1 we get
[Equation]
With probability at least 1 − ( pe ˜ p ) ˜ p exp( − t ) .
Thus by the deﬁnition of T 1 , the ﬁrst result of the lemma has been shown.
For the second part we look to bound the central term of T 1 .
We ﬁrst notice (a) that since assumption (A1) holds, Corollary 4 indicates that if β ∗ j (cid:54) =
⇒ | β ∗ j | ≥ aλ for all j ∈ P ; (b) that for this range of β ∗ j , P (cid:48) λ
( | β ∗ j | )
=
; (c) that per the deﬁnition of MCP 0 ≤ P (cid:48) λ ( | β ∗ j | )
≤ λ for any β ∗ j ∈ (cid:60) .
If we combine these observations with 45 and the deﬁnition of δ ∗ , we can see that T 1 ≤ (cid:80) j ∈S P (cid:48) λ
( | β ∗ j | ) | δ ∗ | ≤ λ (cid:112) |S| − (cid:107) β
∗S (cid:107) 0 · (cid:107) δ ∗ (cid:107) .
From this, given that assumption that, for this second result (A3) holds with ˜ p ∗ ≥ ˜ p , and r ˜ p ≥ r ˜ p ∗ ≥ 0 we can use (A3) part (iii) to show that T 1 ≤ λ (cid:112) |S| − (cid:107) β
∗S (cid:107) 0 · (cid:107)
Xδ ∗ (cid:107) √
nr ˜ p .
Since this holds almost surely, it can then be combined with 47 to get
[Equation]
We can then multiply by 2 √ n/b l (cid:107)
Xδ ∗ (cid:107) to get
[Equation] [Equation]
We
2 A 2 + 2 B to get
[Equation]
Combining this with 50 yields that
[Equation]
Finally, by applying the second part of Lemma 1 and noting from ( 45 ) that T 1 ≤ P λ ( aλ )( |S| − (cid:107) β ∗ (cid:107) 0 ) + Γ we see that
[Equation]
With probability at least 1 − ( pe ˜ p ) ˜ p exp( − t ) which is the desired result.
Lemma 5.
Let Assumptions (A1) and (A2) hold.
Consider a solution β ∗ satisfying S 3 ONC of 2.
Assume that Q ( β ∗ ) ≤ Q ( β true )+Γ holds for an arbitrary Γ > 0 .
For any integer ˜ p : 2 |S| ≤ ˜ p ≤ p if the penalty parameters ( a, λ ) satisfy that P λ ( aλ ) > σ 2 2 nb l (1 + 2 √ t + 2 t ) + σ 2 n |S| (1+2 √ t +2 t )
+Γ b l b l (˜ p +1 − 2 |S| ) , for an arbitrary t > 0 , then (cid:107) β ∗ − β true (cid:107) 0 ≤ ˜ p with probability at least 1 − exp( − (˜ p + 1)( t − ln p )) · 1 − exp( − ( p − ˜ p )( t − ln p ))
1 − exp( − t +ln p ) .
Proof.
We start from the useful inequality deﬁned in
[Equation] [Equation]
where δ ∗ = β ∗ − β true .
Next, conditioning on the fact (i) that β ∗ is S 3 ONC , (ii) that all assumptions for Corollary 4 are satisﬁed (which implies that P λ ( | β ∗ j | ) ∈ { 0 , P λ ( aλ ) } ) and (iii) that
P λ ( | β truej | ) ≤ P λ
( aλ ) we have that
[Equation]
Now consider an event E 1 := {(cid:107) β ∗ − β true (cid:107)
= ˜
p + k } for an arbitrary integer k : 1 ≤ k ≤ p − ˜ p
Conditioning on this event, we may denote and S ˜ p + k ⊆ P such that δ ∗ j (cid:54) = 0 for all j ∈ S ˜ p + k .
By assumption we can ensure that | S ˜ p + k | =
˜ p + k .
Also denote by X S ˜ p + k = ( x ij : i ∈ N ,
j ∈ S ˜ p + k ) and let δ ∗ S ˜ p
+ k :
= ( δ ∗ j :
j ∈ S ˜ p + k ) .
Note that conditional on E 1 , the ﬁrst part of Lemma 1 (using ˜ p + k in place of ˜ p in Lemma 1) can be used to bound W (cid:124)
Xδ ∗ in 59.
Additionally, by deﬁnition (cid:107) β true (cid:107)
= |S| and conditional on E 1 we can apply the substitution (cid:107) β ∗ (cid:107) 0 ≥ ˜ p + k − |S| .
This gives us
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
In order for this equation to be feasible, we know that the quadratic formula must have real roots.
Therefor
[Equation]
Now consier another event E
[Equation]
for an arbitrary t > 0 .
Conditioning on the occurence of E 1 ∩ E 2 ( t ) we can show, using ﬁrst
E 2 ( t ) and then 61, that σ 2 (˜ p + k ) n
[Equation]
k − 2 |S| ) · P λ ( aλ ) − Γ] almost surely, which contradicts with the assumption on the parameters ( a, λ ) .
This can be seen starting from our original assumption that P λ ( aλ ) >
σ 2 2 nb l
[Equation] [Equation]
+
[Equation]
t
[Equation]
+
[Equation] [Equation]
We can then multiply both (outer) sides by 2 b l (˜ p + k − 2 |S| ) and rearrange to get σ 2 n (˜ p + k ) · (1+2 √ t +2 t ) <
2 b l [(˜ p − 2 |S| + k ) · P λ ( aλ ) − Γ] .
Given this contradiction, we know that P [ E 1 ∩ E 2 ( t )] = 0 .
Therefore, again using the union bound combined with DeMorgan’s law we get that P [ E 1 ∩E 2 ( t )] ≥ 1 − P [ E c 1 ] − P
[ E 2 ( t ) c ] which, can be simpliﬁed to
[Equation]
Since all assumptions of the Lemma 3 are satisﬁed, we can next use it to bound
P [ E 2 ( t ) c ] .
By taking the compliment of the result in the second half of Lemma 1, we get, for some t (cid:48) , that P
[Equation]
( pe ˜ p + k ) ˜ p + k exp( − t (cid:48) ) ≤ p ˜ p
[Equation]
( − t (cid:48) ) .
If we then let t (cid:48)
=
(˜ p + k ) t we get
[Equation]
˜ p
[Equation]
Thus we have that p ˜ p + k exp( −
(˜ p + k ) t ) ≥ P
[ E 2 ( t ) c ] which can be combined with 62 to show
[Equation]
With this
[Equation]
we can solve for our desired value
[Equation]
Which is the desired result
[Equation]
Lemma 6.
Consider an arbitrary S 3 ONC solution β ∗ to 2 with MCP.
Let Assumptions (A1) and (A3) with ˜ p ∗ ≥ ˜ p hold.
Assume the satisfaction of (cid:107) β ∗ − β true (cid:107) 0 ≤ ˜ p and Event E a (˜ p ) :
= { 1 n (cid:107) X
( β ∗ − β true ) (cid:107) 2 ≤ 8 σ 2 b 2 l n (cid:0) ˜ p + 2 √ ˜ pt
+ 2 t (cid:1) +
8 b l min { λ 2 r ˜ p ( |S|−(cid:107) β ∗ (cid:107) 0 ) , P λ ( aλ ) · ( |S|−(cid:107) β ∗ (cid:107) 0 )+ Γ } .
If the sub-optimality gap satisﬁes Γ < P λ ( aλ ) − σ 2 b l n (cid:0) ˜ p + 2 √ ˜ pt + 2 t (cid:1) .
If the minimum signal strength satisﬁes (cid:107) β true S (cid:107) min > (cid:113) 8 σ 2 r ˜ p b 2 l n (cid:0) ˜ p + 2 √ ˜ pt + 2 t (cid:1) + 8 r ˜ p
b l min { λ 2 r ˜ p |S| , P λ ( aλ ) |S| + Γ } then β ∗ is the oracle solution to 2.
If in addition we have the satisfaction of (cid:107) β opt − β true (cid:107) ≤ ˜ p and the event E b (˜ p ) :
= {
[Equation]
n
[Equation]
, P
[Equation]
| − (cid:107) β ∗ (cid:107) 0 )
+ Γ } then β ∗ is both the oracle solution and the global solution to 2.
Proof.
First, let us denote β ∗ − β true = δ ∗ .
We start by combining E α (˜ p ) and (A3)iii, which is possible due to our assumption (cid:107) β ∗ − β true (cid:107) 0 ≤ ˜ p .
This gives us
[Equation]
From here if we relax |S| − (cid:107) β ∗S (cid:107) 0 to just |S| , the deﬁnition of δ ∗ and note that (cid:107) δ ∗ (cid:107) ≥ (cid:13)(cid:13) δ ∗ j (cid:13)(cid:13) , we can obtain the following
[Equation]
almost surely.
From this we can bound | β ∗ j | using the square root term and | β truej | , so we know that if | β truej |− (cid:113) 8 σ 2 r ˜ p b 2 l n (cid:0)
˜ p + 2 √ ˜ pt + 2 t (cid:1) + 8 r ˜ p
b l min { λ 2 r ˜ p |S| , P λ ( aλ ) |S| + Γ } > 0 then | β ∗ j | > 0 .
From this we can obtain the inequality
[Equation]
almost surely.
We can then combine this with our minimum signal strength assumption to get that
[Equation]
We can combine this with equation 65, by focusing on the second part of the minimum term and noting the right side is always positive to get
[Equation]
which can be simpliﬁed into
[Equation] [Equation]
thus, it can be seen that if P λ ( aλ ) > σ 2 b l n (cid:0) ˜ p + 2 √ ˜ pt + 2 t (cid:1) + Γ then 1 > (cid:107) β
∗S c (cid:107)
= 0 .
This is satisﬁed by the assumption that P λ ( aλ ) − σ 2 b l n (cid:0)
[Equation]
p +
[Equation]
˜ pt + 2 t (cid:1) > Γ .
Finally, because β ∗ is an S 3 ONC solution, it has to satisfy FONC.
Per 2, this means that β ∗ ∈ arg inf { 1 n (cid:80) i ∈N (cid:96) ( β, x i , y i ) + (cid:80) j ∈P P (cid:48) λ
( | β ∗ j | )
| β j | : β ∈ (cid:60) p } .
Due to Corollary 4, we know that the penalty term goes to 0 since either β ∗ j = 0 or P (cid:48) ( | β ∗ j | )
=
P (cid:48) ( | aλ | ) = 0 .
Further we know that β ∗ j = 0 for all j ∈ S c .
Threfore we know that
[Equation]
Given right
, we shown result.
Next, Consider β opt which is the global optimal solution to 2.
Given that the S 3 ONC conditions are necessary, β opt must be an S 3 ONC solution.
With this fact and the assumption of E b (˜ p ) , we have the same set of assumptions for β opt as we had for β ∗ .
Thus the same sequence of arguments can be used to show that
[Equation]
Finally, per the strict convexity of our loss function as implied by (A1) we can see that the inﬁmum of the above problem is unique.
Therefor
[Equation]
Which is the second result.
A. A DDITIONAL LEMMAS
Lemma 7.
The RE condition in 1 implies (A3) with r 4 s ≥ r e > 0 and ˜ p ∗ ≥ 4 s .
Proof.
As in Lemma 1 in (Liu et al., 2017).
Lemma 8.
Let β ∗ be a S 3 ONC solution to 2 Given (A1) and that Q ( β ∗ ) ≤ Q ( β true ) + Γ holds for some Γ ≥ 0 then
[Equation]
Proof.
First, we know that β ∗ ∈ arg min β { n (cid:80) i =
[Equation]
β, x
[Equation]
, y
[Equation]
+ p
[Equation]
j =1 P
[Equation]
λ ( | β ∗ | )
| β j |} because the KKT conditions are the same as FONC which β ∗ satisﬁes.
This gives us that n (cid:80) i
[Equation]
This can be used along the same lines as the level set inequality in the derivation for 8 to get b l 2 n (cid:107)
Xδ
[Equation]
Xδ
[Equation]
≤ p (cid:80)
[Equation]
The ﬁrst two terms of the min function are easily obtained from this.
The last term can be obtained from 8 by noting that due to Corollary 4, β ∗ / ∈ (0 , aλ ) and that P λ ( aλ ) =
P λ ( β ) ∀ β ≥ aλ .
This gives us that b l 2 n
[Equation]
aλ )
[Equation]
S −(cid:107) β ∗
[Equation]
0 )+Γ Which is the ﬁnal term to complete the desired result.
Lemma 9.
Consider a subgaussian ˜ n -dimensional random vector ˜ W ∈ (cid:60) ˜ n as deﬁned in (A2).
Then
Proof.
We apply Theorem 2.1 in (Hsu et al., 2012) where our ˜ W , V and Σ v are equivalent to their x , A and Σ .
Note their expectation condition is equivalent to our (A2) with µ = E
[ W ] = 0 .
This gives us that for all t >
[Equation]
Given that µ = 0 , the term involving tr (Σ v µµ (cid:124) ) goes to zero and therefor the statement in 9 can be obtained by taking the complement of the probability bound.