E FFICIENT H IGH -D IMENSIONAL D ATA R EPRESENTATION L EARNING VIA S EMI -S TOCHASTIC B LOCK C OORDINATE D ESCENT M ETHODS
Anonymous authors
Paper under double-blind review
A BSTRACT
With the increase of data volume and data dimension, sparse representation learning attracts more and more attention.
For high-dimensional data, randomized block coordinate descent methods perform well because they do not need to calculate the gradient along the whole dimension.
Existing hard thresholding algorithms evaluate gradients followed by a hard thresholding operation to update the model parameter, which leads to slow convergence.
To address this issue, we propose a novel hard thresholding algorithm, called Semi-stochastic Block Coordinate Descent Hard Thresholding Pursuit (SBCD-HTP).
Moreover, we present its sparse and asynchronous parallel variants.
We theoretically analyze the convergence properties of our algorithms, which show that they have a signiﬁcantly lower hard thresholding complexity than existing algorithms.
Our empirical evaluations on real-world datasets and face recognition tasks demonstrate the superior performance of our algorithms for sparsity-constrained optimization problems.
I NTRODUCTION
In modern high-dimensional data analytics, where the variable dimension can be equal or even larger than the number of samples, sparse representation learning has become a mainstream method to explore the potential real model of the problem and provides statistically reliable results.
It has been applied to many diverse domains such as high-dimensional statistics (B¨uhlmann & Van De Geer, 2011), signal processing (Lai et al., 2013), and computer vision (Wright et al., 2008).
Many sparse learning algorithms such as (cid:96) 1 -norm convex relaxation methods (Negahban et al., 2009; Van de Geer et al., 2008) have been proposed in the past few decades.
In order to get a smaller estimation error, (cid:96) 0 -norm constrained algorithms are more prominent than (cid:96) 1 -norm convex relaxation algorithms (Zhang et al., 2010).
In this paper, we mainly focus on the following sparsity-constrained optimization problem:
[Equation]
where F ( w ) is a ﬁnite-sum convex and smooth function, each function f i ( w ) is associated with the i -th sample, (cid:107) w (cid:107) 0 is the number of nonzero entries in variable w , and s represents the sparsity level.
The goal of sparse representation learning is to recover w ∗ based on the given data.
Such a formulation encapsulates plentiful important problems, including sparse graphical model learning (Zhou et al., 2019), sparse linear/logistic regression (Blumensath & Davies, 2009; Foucart, 2011;
Pati et al., 1993; Bahmani et al., 2013), and low-rank regression (Rohde et al., 2011).
However, Problem (1) is in general NP hard, which is caused by the non-convexity of the sparsity constraint.
It makes us to obtain an approximate solution to Problem (1).
One of the most widely used methods for solving this problem is the hard thresholding based gradient descent method.
In recent years, sparse representation learning is becoming more common for large-scale and highdimensional data.
However, deterministic gradient descent hard thresholding algorithms such as fast gradient descent hard thresholding (FG-HT)
(Jain et al., 2014; Yuan et al., 2014) have to compute the gradients of all n component functions, which leads to a huge computing overhead.
To address this issue, many stochastic hard thresholding methods have been proposed.
For instance, Nguyen et al.
(2017) proposed a stochastic gradient descent hard thresholding (SG-HT) algorithm,
whose
[Table] [Table]
gradient and hard thresholding complexities are both O
( κ (cid:101) s log( 1 (cid:15) )) .
However, due to the stochastic sampling
[Table]
SG-HT can only attain a sub-optimal estimation bound, as shown in Table
[Table]
which is inferior to those of deterministic gradient methods such as FG-HT.
Another limitation of SG-HT is that it requires that the restricted condition number κ (cid:101) s should not be larger than 4 / 3 , which makes
SG-HT hard for solving high-dimensional representation learning problems.
Recently, many stochastic variance reduced methods (e.g., SAG (Roux et al., 2012), SVRG (Johnson & Zhang, 2013)) and their accelerated variants such as (Defazio, 2016; Allen-Zhu, 2018) have been proposed to accelerate stochastic gradient methods for convex optimization.
All these methods enjoy low per-iteration complexities comparable with stochastic gradient descent (SGD), and they also can attain improved convergence rates.
By incorporating the variance reduction technique into sparsityconstrained optimization domain, Li et al.
(2016b) proposed a stochastic variance reduced gradient hard thresholding (SVRG-HT) algorithm, which can converge more quickly and obtain a smaller estimation error than SG-HT.
Moreover, SVRG-HT allows an arbitrarily large condition number in its theoretical analysis similar to FG-HT
(Yuan et al., 2014).
The gradient oracle and hard thresholding complexities of SVRG-HT are O (( n + κ (cid:101) s ) log( 1 (cid:15) )) and O ( κ (cid:101) s log( 1 (cid:15) )) , respectively.
Nevertheless, it can not leverage the coordinate block to accelerate convergence.
Chen & Gu (2016) proposed an accelerated stochastic block coordinate gradient descent hard thresholding (ASBCDHT) algorithm.
The gradient oracle complexity of ASBCDHT is O (( n + κ (cid:101) s |B| k ) log( 1 (cid:15) )) , which is superior to
SVRG-HT.
Moreover, ASBCDHT also has the hard thresholding complexity, O ( κ (cid:101) s log( 1 (cid:15) )) .
However, the hard thresholding complexity of ASBCDHT still scales linearly with κ (cid:101) s , which is usually expensive for real-world sparse learning problems.
For large-scale and high-dimensional data, Li et al.
(2016a) also proposed the asynchronous parallel variant of SVRG-HT (called ASVRG-HT) by utilizing multicore architectures.
Although it makes each processor to evaluate a stochastic gradient update on a global parameter stored in a shared memory in an asynchronous and lock-free mode,
ASVRG-HT attains the similar gradient and hard thresholding complexities as its general version (i.e., SVRG-HT).
Therefore, all the algorithms mentioned above have a linearly κ (cid:101) s -dependent hard thresholding complexity.
This motivates us to address the following key issue:
Can we design such algorithms that have a κ (cid:101) s -independent hard thresholding complexity and a lower gradient oracle complexity?
To answer the above problem, we propose an efﬁcient Semi-stochastic Block Coordinate Descent
Hard Thresholding Pursuit (SBCD-HTP) algorithm and its sparse Asynchronous variant (ASBCDHTP).
The oracle complexities and statistical estimation error of the proposed algorithms and other hard thresholding methods are summarized in Table 1.
We highlight several theoretical advantages of the proposed algorithms over the state-of-the-art methods as follows:
• SBCD-HTP and ASBCD-HTP substantially improve the restricted condition on the sparsity level s , i.e., s =Ω( κ (cid:98) s s ∗ ) , while most related algorithms such as SVRG-HT require s =Ω( κ 2 (cid:101) s s ∗ ) .
• The statistical estimation error of both our algorithms is better than those of FG-HT, SVRG-HT and ASBCDHT, as our algorithms have a smaller sparsity level s =Ω( κ (cid:98) s s ∗ ) and thus have a smaller cardinality, i.e., (cid:98) s
= 2 s + s ∗ .
This makes the restricted condition number κ (cid:98) s of SBCD-HTP smaller than other algorithms.
Moreover, the statistical error of our algorithms is also smaller than that of SG-HT (i.e., 1 n (cid:80) ni
=1 (cid:107)∇ f i ( w ∗ ) (cid:107) ) due to the large magnitude of (cid:107)∇ f i ( w ∗ ) (cid:107) (Zhou et al., 2018b).
• Both SBCD-HTP and ASBCD-HTP enjoy a κ (cid:101) s -independent hard thresholding complexity, which is signiﬁcantly better than the state-of-the-art hard thresholding algorithms.
That is, the hard thresholding complexity of our algorithms is κ (cid:101) s times lower than that of FG-HT, SG-HT, SVRG-HT and ASBCDHT with (cid:101)
s = 2Ω( κ 2 (cid:101) s s ∗ )
+ s ∗ .
Since both our algorithms have a signiﬁcantly lower hard thresholding complexity, they are more suitable for handling large-scale sparse representation learning problems, especially for high-dimensional data.
• For sparsity-constrained problems, the gradient oracle complexities of both SBCD-HTP and ASBCD-HTP (or S 2 BCD-HTP) are much lower than those of the state-of-the-art hard thresholding algorithms, and that of S 2 BCD-HTP is similar to ASBCD-HTP under conditions on the delay. Actually, the gradient oracle complexity of S 2 BCD-HTP is better than SBCD-HTP when we deal with sparse data sets, since S 2 BCD-HTP only evaluates the common nonzero point of the random sample and the sampled block.
Furthermore, their gradient oracle complexities are signiﬁcantly lower than those of ASBCDHT and SVRG-HT, and much lower than that of FG-HT, since they require a smaller sparsity level s = Ω( κ (cid:98) s s ∗ ) and also have a smaller value of κ (cid:98) s with (cid:98) s
= 2 s + s ∗ compared with the restricted condition number κ (cid:101) s used in other hard thresholding algorithms (Zhou et al., 2018b).
R ELATED W ORK
In this section, we brieﬂy discuss the relevant research to our work, which beyonds the sparsityconstrained optimization domain.
Traditional gradient descent is computational expensive at each iteration, and stochastic gradient descent has a low per-iteration complexity while obtaining a large variance in estimating.
Thus, many stochastic variance reduced algorithms (Defazio et al., 2014;
Roux et al., 2012; Shalev-Shwartz & Zhang, 2013; Shang et al., 2018; Johnson & Zhang, 2013) and their variants (Schmidt et al., 2017; Kone ˇ cn ` y et al., 2015; Xiao & Zhang, 2014; Hannah et al., 2018) have been proposed.
The stochastic variance reduced methods are very promising for many machine learning problems including sparse learning problems.
In contrast to gradient descent methods, coordinate descent algorithms have received increasing attention due to their successful applications in high dimensional problems (Breheny & Huang, 2011;
Friedman et al., 2007).
Among them, randomized block coordinate descent (RBCD) updates a block of coordinate with respect to the entire training instances.
The per-iteration cost is signiﬁcantly lower than gradient descent, while it still has a relationship with n component functions.
Some stochastic block coordinate descent (SBCD) algorithms such as (Dang & Lan, 2015; Xu & Yin, 2015; Kone ˇ cn ` y et al., 2017) were proposed.
Such algorithms compute the stochastic gradient with respect to a random sample restricted to one randomized coordinate block.
Therefore, these algorithms randomly sample a block of features and data instances at each iteration.
However, they can only achieve a sublinear convergence rate (Zhang & Gu, 2016).
Recently, some mini-batch randomized block coordinate descent algorithms such as (Zhao et al., 2014;
Wang & Banerjee, 2014) were proposed to accelerate the convergence of stochastic block coordinate gradient descent.
Our work studies over the above research by considering a sparsity-constrained optimization problem.
The proposed algorithms enjoy a lower computational complexity in both gradient evaluation and hard thresholding computation while obtaining a linear convergence performance.
Moreover, its sparse and asynchronous variants are also proposed to deal with sparse high-dimensional data.
P RELIMINARIES
Throughout this paper, we use w ∗ to denote the optimal solution of Problem (1), and its optimal sparsity level is s ∗ with (cid:107) w ∗ (cid:107)
≤ s ∗ .
(cid:107) w (cid:107) is the Euclidean norm for a vector w ∈ R d and (cid:107) w (cid:107) ∞ is the largest absolute entry in w .
The hard thresholding operation HT ( w, s ) preserves the s largest entries of w in magnitude for vector w and the rest entries are set to zero.
We use supp ( w ) to denote the support set of w , i.e., the indices of its nonzero entries.
Given an index set I , we deﬁne I C as the complement set of I , and v I ∈ R d , where [ v I ] j = v j if j ∈I and [ v I ]
j =0 if j / ∈I .
Given an integer n ≥ 1 , we deﬁne [ n ] = { 1 , · · · , n } .
For a set B , we denote its cardinality by |B| .
Moreover, we use the common notations of Ω( · ) and O ( · ) to characterize the asymptotics of two real sequences.
Throughout the analysis, we make two important assumptions on the objective function, which are commonly used in the analysis of hard thresholding algorithms (Li et al., 2016b; Shen & Li, 2017;
Chen & Gu, 2016; Gao & Huang, 2018).
In high-dimensional sparse learning, the per-iteration hard thresholding operation can be time-consuming or even more expensive than gradient computation.
Thus, we also take the complexity of hard thresholding into our consideration.
Assumption 1 (Restricted Strong Convexity)
A differentiable function F ( · ) is restricted ρ − (cid:98) sstrongly convex at sparsity level (cid:98) s , if there exists a uniform constant ρ − (cid:98) s > 0 such that for any w, w (cid:48)
∈ R d with (cid:107) w − w (cid:48) (cid:107) 0 ≤ (cid:98) s , we have
[Equation]
Assumption 2 (Restricted Strong Smoothness)
For any i ∈ [ n ] , the differentiable function f i ( · ) is restricted ρ + (cid:98) s -strongly smooth at sparsity level (cid:98) s , if there exists a uniform constant ρ + (cid:98) s > 0 such that for any w, w (cid:48) ∈ R d with (cid:107) w − w (cid:48) (cid:107) 0 ≤ (cid:98) s , we have
[Equation]
[Equation]
Moreover, the restricted condition number is deﬁned as: κ (cid:98) s = ρ + (cid:98) s /ρ − (cid:98) s .
Since the complexity of one hard thresholding operation is linear with d , which is similar to a computational cost of a stochastic gradient, we deﬁne the hard thresholding complexity as a metric for hard thresholding algorithms.
Deﬁnition 1 (Hard Thresholding Complexity)
In a hard thresholding operation, a vector w is fed into HT ( · , s ) and then the output HT ( w, s ) is obtained.
Both gradient oracle and hard thresholding complexities can more comprehensively reﬂect the overall computational cost of ﬁrst-order hard thresholding algorithms.
Therefore, their per-iteration cost is dominated by both the gradient evaluation and hard thresholding operations.
We also deﬁne the optimization error and statistical error to clearly understand our theoretical results.
Deﬁnition 2 (Optimization Error and Statistical Error)
The optimization error is the difference which will decrease with the increasing of iterations.
A statistical error is the (unknown) difference between the retained value and the true value.
This means that the optimization error will decrease to zero when the number of iteration is quite large. However, a statistical error will still work during the optimized process. Therefore, the larger the statistical error, the more difference to the true value.  S EMI -S TOCHASTIC B LOCK C OORDINATE D ESCENT H ARD T HRESHOLDING P URSUIT
In this section, we propose a novel Semi-stochastic Block Coordinate Descent Hard Thresholding
Pursuit (SBCD-HTP) algorithm, and also theoretically analyze its convergence properties.
4. O UR SBCD-HTP A LGORITHM
The proposed SBCD-HTP algorithm is summarized in Algorithm 1.
In each outer-loop, we select a snapshot point (cid:101) w and compute its full gradient ∇ F ( (cid:101) w ) .
In each inner-loop, we uniformly and randomly select a mini-batch samples B and a block of coordinates G j t , where j t ∈ { 1 , 2 , . . .
, k } , {G 1 . . .
G k } is a partition of all the d coordinates and is divided uniformly at random.
We usually set k =10 in our experiments.
Note that, different from ASBCDHT, in line 5 we set S to be a union set of the support set of snapshot point (cid:101) w and G j t .
In this way, the coordinate of the support set S which includes the possibly optimal coordinates can be updated, and thus this can reduce the error caused by randomness.
Moreover, SBCD-HTP enjoys the dual advantages of randomized block coordinate descent and semi-stochastic gradient descent to optimize the objective as shown in line 6.
Note that we perform the hard thresholding operation in only the outer-loop, which is different from all existing hard thresholding algorithms.
Performing hard thresholding operations with high frequency or prematurely in inner-loop will make the gradient information lost, which is the main reason for slow convergence and low accuracy of existing hard thresholding sparsity-constrained algorithms.
Our hard thresholding algorithm not only guarantees the sparsity of model parameters, but also uses the full dimensional information without the interference of hard thresholding operations to update model parameter w t +1 .
Our theoretical analysis and experimental results show that SBCD-HTP has a faster convergence rate and more accurate results.
We have to clarify the difference between the proposed SBCD-HTP and ASBCDHT in Chen & Gu (2016) for clear improvements. There are mainly two changes: a) we design a bigger support set with the union of the sampled block G j t with the one of the current support of the snapshot point (cid:101) w , since the current support of snapshot point contain the most related optimized information, while that of Chen & Gu (2016) only use the sampled block. b) we move the hard thresholding operation outside of the inner-loop, since the truncated function (hard thresholding) may loss many information and bringing higher complexity.
Moreover, the asynchronous variant of SBCD-HTP is also proposed in this paper to optimize the sparse problem 1 in the sparse data.
This is the ﬁrst to design this type of algorithm in sparse learning ﬁeld, and the theoretical proof is also provided.
4. C ONVERGENCE A NALYSIS
We ﬁrst analyze the convergence behavior of SBCD-HTP.
The main result is summarized in the following theorem, whose proof is provided in Section C.
Theorem 1 Suppose F ( w ) is ρ − (cid:98) s -strongly convex and each function f
i ( w ) is ρ + (cid:98) s -strongly smooth with parameter (cid:98) s
= 2 s + s ∗ .
Let κ (cid:98) s
= ρ + (cid:98) s ρ − (cid:98) s and
the sparsity level s ≥ Ω( κ (cid:98) s s ∗ ) .
In addition, assume that the learning rate η ≤ 1 48 ρ + (cid:98) s , the number of inner-loops m ≥ 1800 κ (cid:98) s , and the number of blocks k =10 , then the convergence rate γ = k 2 ηmρ
[Equation]
s
[Equation]
1 − ω ) + 8 ρ + [Equation] s η ( n −|B| ) |B| (1 − ω )( n − 1) ≤ 12 .
For the sparsity-constrained problem (1), the output of Algorithm 1 satisﬁes
[Equation]
where ˜ I =
supp ( HT ( ∇ F ( w ∗ ) , 2 s )) ∪ supp ( w ∗ ) and ω = 8 ρ
+ (cid:98) s η
(1 + n −|B| |B| ( n − 1) ) .
Theorem 1 shows that SBCD-HTP enjoys a linear convergence rate for solving sparsity-constrained problem (1).
In order to ensure that the linear converging term in Eq.
(4) satisﬁes ( 12 ) r E
[ F ( (cid:101) w 0 ) −
F ( w ∗ )] ≤ (cid:15) , we can obtain the following corollary.
Corollary 1 Suppose the conditions in Theorem 1 hold.
To achieve ( 12 ) r E [ F ( (cid:101) w 0 ) − F ( w ∗ )] ≤ (cid:15) , the gradient oracle complexity of SBCD-HTP is O (( n + κ (cid:98) s |B| k ) log( 1 (cid:15) )) , and its hard thresholding complexity is O (log( 1 (cid:15) )) . {+Remark 
+}
This corollary is inferred under the high-dimensional condition.
Actually, the worst condition of the gradient oracle complexity of SBCD-HTP is O (( n + κ (cid:98) s |B| dk + s d ) log( 1 (cid:15) )) .
Due to the small sparsity level s required in our Theorem 1 and the overlaps between G j t and (cid:101) G , we can obtain the gradient oracle complexity O (( n + κ (cid:98) s |B| k ) log( 1 (cid:15) )) for SBCD-HTP.
Table 1 summarizes the properties of some hard thresholding methods.
Compared with the gradient oracle complexities of FG-HT
(Yuan et al., 2014)
and SVRG-HT (Li et al., 2016b)
(i.e.,
O ( nκ (cid:101) s log( 1 (cid:15) )) and O (( n + κ (cid:101) s ) log( 1 (cid:15) )) , respectively)
, SBCD-HTP has a much lower oracle complexity.
For ASBCDHT (Chen & Gu, 2016), whose gradient oracle complexity is O (( n + κ (cid:101) s |B| k ) log( 1 (cid:15) )) , the gradient oracle complexity of SBCD-HTP is O (( n + κ (cid:98) s |B| k ) log( 1 (cid:15) )) and is much lower than ASBCDHT, as κ (cid:98) s of SBCD-HTP is usually comparable to or even smaller than those of others (note that s in (cid:98) s
=2 s + s ∗ is required to be smaller than others as shown below).
Moreover, SBCD-HTP allows s =Ω( κ (cid:98) s s ∗ ) , which is considerably superior to the condition of s =Ω( κ 2 (cid:101) s s ∗ ) required in other hard thresholding algorithms such as (Zhou et al., 2018b).
In particular, with the same parameter settings of |B| and k , the gradient oracle complexity of our algorithm outperforms ASBCDHT.
SBCD-HTP also attains a lower hard thresholding complexity than the state-of-the-art hard thresholding methods.
That is, the hard thresholding complexity of SBCD-HTP is O (log( 1 (cid:15) )) , which is κ (cid:101) s -independent and is signiﬁcantly lower than those of other methods.
This is a signiﬁcant improvement on hard thresholding complexity.
Theorem 1 immediately implies the following results.
Corollary 2 Suppose the conditions in Theorem 1 hold.
The output of Algorithm 1 satisﬁes
[Equation]
The right-hand side of Eq.
(5) consists of two terms.
The ﬁrst term is the optimization error, which approaches zero with the increase of r .
The second term corresponds to the statistical error, which is proportional to √ (cid:98) s (cid:107)∇ F ( w ∗ ) (cid:107) ∞ .
One can also observe that the statistical error bound is better than those of FG-HT and ASBCDHT, since (cid:98) s in the statistical error O ( √ (cid:98) s (cid:107)∇ F ( w ∗ ) (cid:107) ∞ ) has a smaller cardinality (i.e., 2Ω( κ (cid:98) s s ∗ )
+ s ∗ ), while existing algorithms require a larger sparsity level s (cid:48)
=
Ω( κ 2 (cid:101) s s ∗ ) and have a larger cardinality.
Therefore, compared with other existing algorithms,
SBCD-HTP enjoys a smaller statistical error, especially for the problems with a large restricted condition number.
It is usually better than the error bound O ( √ (cid:101) s (cid:107)∇ F ( w ∗ ) (cid:107) ∞ + (cid:107)∇ ˜
I F ( w ∗ ) (cid:107) ) with (cid:101)
s =2Ω( κ 2 (cid:101) s s ∗ )
+ s ∗ in SVRG-HT.
Moreover, the error bound of SBCD-HTP is much smaller than that of SG-HT (Nguyen et al., 2017)
(i.e., O ( 1 n (cid:80) ni
=1 (cid:107)∇ f i ( w ∗ ) (cid:107) ) ), since the magnitude of the individual gradient norm (cid:107)∇ f i ( w ∗ ) (cid:107) can still be relatively large (Zhou et al., 2018b).
A SYNCHRONOUS V ARIANT OF SBCD-HTP
In this section, we propose the serial sparse and asynchronous parallel variants of SBCD-HTP for high-dimensional sparse data sets.
Our Asynchronous Sparse Semi-stochastic Block Coordinate
Descent Hard Thresholding Pursuit (ASBCD-HTP) algorithm is summarized in Algorithm 2.
The main difference between ASBCD-HTP and SBCD-HTP is sparse approximate gradients as in (Mania et al., 2017; Zhou et al., 2018a).
In order to perform fully sparse updates, we use a diagonal matrix D to re-weigh the dense fully gradient ∇ F ( (cid:101) w ) .
The entries of D are the inverse probabilities of the corresponding coordinates belonging to a uniformly sampled support T i t of sample i t .
Let
P i t be the projection matrix for the support T i t , we can get D
i t since D i t =
P i t D .
Then we can ﬁnd that E i t
[ D i t ∇ F ( (cid:101) w )]= ∇ F ( (cid:101) w ) .
In this setting, we can compute the full gradient ∇ F ( (cid:101) w ) also in a parallel way.
In order to reduce thread interference and fully utilize the sparsity of datasets, we restrict the sparse gradient ∇ g ([ (cid:98) w ] T it ) to the intersection set of S and T i t and obtain ∇ S g ([ (cid:98) w ] T it ) .
Speciﬁcally, each thread of ASBCD-HTP computes sparse approximate gradient independently and then atomically write this gradient to the shared variable (cid:101) w r .
Note that Sparse variant of SBCDHTP (S 2 BCD-HTP) can be viewed as the single-thread version of ASBCD-HTP, which has linear convergence and the same oracle complexity as ASBCD-HTP.
The detailed description and analysis of S 2 BCD-HTP is listed in Section D. Next we give the main theoretical result of ASBCD-HTP.
Theorem 2 Suppose F ( w ) is ρ − (cid:98) s -strongly convex and each function f
i ( w ) is ρ + (cid:98) s -strongly smooth with parameter (cid:98) s
= 2 s + s ∗ .
Let κ (cid:98) s
= ρ + (cid:98) s ρ − (cid:98) s and
the sparsity level s ≥ Ω( κ (cid:98) s s ∗ ) .
We assume that the step size η = 1 60 ρ + (cid:98) s , the number of inner-loops m ≥ 120 κ (cid:98) s , the number of blocks k = 10 and τ ≤ min { 3 5 √ ∆ , 2 κ (cid:98) s , (cid:113) 2 κ (cid:98) s √ ∆ } (i.e., the linear speedup condition).
Then for the sparsity-constrained problem (1), the gradient oracle and hard thresholding complexities of Algorithm 2 are
[Equation]
where τ denotes the maximum number of concurrent threads (Mania et al., 2017) and ∆ = max j =1 ··· d p j , which is a indicator to measure the sparsity of datasets (Leblond et al., 2017).
Theorem 2 implies that ASBCD-HTP (or S 2 BCD-HTP) can still obtain similar gradient oracle complexity and hard thresholding complexity as SBCD-HTP while obtaining a linear speedup ratio.
Compared with ASVRG-HT (Li et al., 2016a), ASBCD-HTP has a much lower gradient complexity, i.e., O (( n + κ (cid:98) s k ) log 1 (cid:15) ) for ASBCD-HTP vs. O (( n + κ (cid:101) s ) log 1 (cid:15) ) for ASVRG-HT.
For hard thresholding complexity, ASBCD-HTP is κ (cid:101) s times lower than ASVRG-HT, which is a signiﬁcant improvement in the asynchronous setting.
Since ASVRG-HT has to compute hard thresholding operations at each inner-loop, the practical performance may be worse, especially for high-dimensional data.
We also show this improvement by our empirical evaluations.
E XPERIMENTS
In this section, we evaluate the performance of SBCD-HTP, S 2 BCD-HTP and ASBCD-HTP for solving sparse linear regression and sparse logistic regression on real-world datasets.
All the dense algorithms were implemented in MATLAB, while the sparse and asynchronous algorithms were implemented in C++ and executed through MATLAB interface for a fair comparison.
We also test the performance for face recognition tasks.
The detailed information is described in Section
F. Since there is no ground truth on real-world datasets, we run all these baselines sufﬁciently long until (cid:107) w r − w r +1 (cid:107) / (cid:107) w r (cid:107) ≤ 10 − 6 , and then we use the minimum F ( w r ) as the approximate optimal value F ∗ for sub-optimality estimation in our experimental results Zhou et al.
(2018b).
6. D ENSE A LGORITHMS
We ran the experiments of all the dense algorithms on a PC with an Intel i7-7700 CPU and 32GB
RAM.
We compare SBCD-HTP with several state-of-the-art sparsity-constrained algorithms, including FG-HT
(Yuan et al., 2014),
SG-HT (Nguyen et al., 2017), SVRG-HT
(Li et al., 2016b),
ASBCDHT
(Chen & Gu, 2016) and FNHTP (Chen & Gu, 2017).
The detailed descriptions of the [Figure] [Figure] Log O b
j e c t i v e
G ap
FG-HTSG-HTSVRG-HT
ASBCDHTFNHTPSBCD-HTP   600 CPU Time (s) - - - -
Log O b
j e c t i v e
G ap
FG-HTSG-HTSVRG-HT
ASBCDHTFNHTP
[Figure]
HTP
[Figure]
algorithms and datasets are provided in Section F.
We set the sparsity level s = 200 for real-word datasets.
For the number of inner-loops, we set m = 2 n for SBCD-HTP and ASBCDHT, m = n for SVRG-HT as suggested in (Li et al., 2016b; Chen & Gu, 2016) and 1000 (sparse linear regression)/3000 (sparse logistic regression) for FNHTP.
We also set the batch-size of FNHTP to s , and that of ASBCDHT and SBCD-HTP to 5.
For the block coordinate descent methods, we set the number of coordinate blocks k to 10.
All the algorithms are tuned to their best performance.
We report the performance of all the algorithms in terms of both effective passes and CPU time.
More experimental results are presented in Section F. Figures 1 and 2 show that for both sparse linear regression and sparse logistic regression problems, our algorithm converges signiﬁcantly faster than the baseline algorithms in terms of both effective passes and CPU time.
The main reason is that we use all the information of gradients, while the related algorithms use less information caused by too frequent and premature hard thresholding operation in each inner-loop.
6. A SYNCHRONOUS A LGORITHMS
We ran the experiments of all the asynchronous algorithms on a PC with an Intel Xeon(R) Gold 5120 CPU and 64GB RAM.
We compare our sparse and asynchronous algorithms with several stateof-the-art algorithms, including ASG-HT, ASVRG-HT and A 2 SBCD-HT (asynchronous variant of
ABSCDHT).
For the algorithms, we set the number of threads to 20.
We only report the performance of all the algorithms in terms of CPU time since the performance in terms of effective passes is similar to the dense case.
More experimental results are reported in
Section F. Figure 3 shows that ASBCD-HTP and S 2 BCD-HTP signiﬁcantly outperform than other algorithms.
Moreover, ASBCD-HTP has a faster convergence performance than S 2 BCD-HTP because of asynchronous parallel acceleration.
There are three main reasons for the advantages of our
ASBCD-HTP: a) We get the set S from the support set of snapshot point (cid:101) w and G j t .
In this way, the coordinate of the support set which appears as the possible optimal solution can be updated, and thus reducing the error caused by randomness.
b) Because we put the hard thresholding operation in each outer-loop, the computation cost per thread of our algorithm at each inner-loop is less, while each thread of the baseline algorithms has a hard thresholding operation, which is expensive especially for high-dimensional datasets.
c)
Our algorithm updates (cid:101) w r with all information of gradients, while the baseline algorithms update (cid:101) w r with less information caused by too frequent and premature hard thresholding operation in each inner-loop.
We also evaluate the improvement of asynchronous parallel by running the same passes with different numbers of threads. We calculate the speed-up [Figure] [Figure]
peed
[Figure] R a t i o [Figure] 0.51.52.5 3 CPU Time (s) - - - -
Log O b
j e c t i v e
G ap thread = 1 thread = 4 thread = 8 thread =     25 Number of Threads
S peed - up
R a t i o
[Figure]
ratio based on the running time of a single thread.
From Figure 4, we can see that our ASBCD-HTP is accelerated by a nearly linear ratio.
C ONCLUSION
In this paper, we proposed an efﬁcient semi-stochastic block coordinate descent hard thresholding pursuit (SBCD-HTP) method for sparse representation learning.
We proved that SBCD-HTP attains the gradient oracle complexity of O (( n + κ (cid:98) s |B| k ) log( 1 (cid:15) )) , and the hard thresholding complexity of
O (log( 1 (cid:15) )) , respectively, which is an improvement over the existing algorithms.
Moreover, we also presented the sparse and asynchronous parallel variants of SBCD-HTP.
As far as we know, our algorithms are the ﬁrst to use the full information of gradient to update.
This is an improvement on sparse representation learning because it can be easily extended to other sparse learning algorithms.
A PPENDIX A K EY L EMMAS
Lemma 1 ((Jain et al., 2014))
For any index set I , any w ∈ R I and HT ( · , s ) :
R d → R d be the hard thresholding operator, which keeps the largest s entries (in magnitude) and sets the other entries equal to zero.
Then for any w ∗ ∈ R I such that (cid:107) w ∗ (cid:107)
≤ s ∗ , we have
[Equation]
Lemma 2 ((Li et al., 2016b)) Let w ∗ ∈
R
d be the optimal sparse vector such that (cid:107) w ∗ (cid:107)
≤ s ∗ and HT ( · , s ) :
R d → R d be the hard thresholding operator, which keeps the largest s entries (in magnitude) and sets the other entries equal to zero.
Given s > s ∗ for any vector w ∈ R d , we have
[Equation]
Lemma 3 ((Nesterov, 2004))
For any given vector w, w (cid:48) ∈ R d , suppose F ( · ) also satisﬁes RSC and RSS conditions, then the following inequality holds
[Equation]
Lemma 4 Suppose F ( w ) is ρ − (cid:98) s -strongly convex and each function f
i ( w ) is ρ + (cid:98) s -strongly smooth with parameter (cid:98) s
= 2 s + s ∗ .
I =
supp( w ∗ ) ∪ supp( (cid:101) w r ) ∪ supp( (cid:101) w r +1 ) , |B| is the batch size.
For any w t ∈ R d and the sample i t ∈ B , denote ν i t
= ( ∇ I f i t ( w t ) −∇
I f i t ( w ∗ )) − ( ∇ I F ( w t ) −∇
I F ( w ∗ )) , we have
[Equation]
Proof.
See Section B.1 for the proof of Lemma 4.
[Equation]
Lemma
Suppose F ( w ) is ρ − (cid:98) s -strongly convex and each function f
i ( w ) is ρ + (cid:98) s -strongly smooth with parameter (cid:98) s
= 2 s + s ∗ .
Let w ∗ ∈
R
d be the optimal sparse vector with (cid:107) w ∗ (cid:107) 0 ≤ s ∗ .
I = supp( w ∗ ) ∪ supp( (cid:101) w r ) ∪ supp( (cid:101) w r +1 ) , δ > 1 is a uniform constant factor, and |B| is the batch size.
For any w t , (cid:101) w ∈ R d and the sample i t , denote ∇ S g ( w t )
= 1 |B| (cid:80)
i t ∈B ∇ S
f i t ( w t ) −∇
S f i t ( (cid:101) w )+ ∇ S F ( (cid:101) w ) , then we can bound E (cid:107)∇ S
g
I
[Equation]
w t )
[Equation]
2 as follows:
[Equation]
Proof.
See Section B.2 for the detailed proof of Lemma 5.
(cid:3)
Lemma 6 Suppose F ( w ) is ρ − (cid:98) s -strongly convex and each function f
i ( w ) is ρ + (cid:98) s -strongly smooth with parameter (cid:98) s
= 2 s + s ∗ .
Let w ∗ ∈
R
d be the optimal sparse vector with (cid:107) w ∗ (cid:107) 0 ≤ s ∗ .
I =
supp( w ∗ ) ∪ supp( (cid:101) w r ) ∪ supp( (cid:101) w r +1 ) , δ > 1 is a uniform constant factor, and D m = max j =1 , ··· ,d 1 p j .
For any w t , (cid:101) w ∈ R d and the sample i t , denote ∇ S g ( w t ) =
∇ S
f
i t ( w t ) − ∇ S f i t ( (cid:101) w ) + D i t ∇ S F ( (cid:101) w ) , we can bound E (cid:107)∇ S g I ( w t ) (cid:107) 2 as follows:
[Equation]
where p j is the probability that dimension j belonging to the support set of randomly sampling sample i t .
Proof.
See Section B.3 for the proof of Lemma 6.
(cid:3)
This lemma is different from other variance reduced results for sparsity-constrained problem (Mania et al., 2017; Johnson & Zhang, 2013; Chen & Gu, 2016; Shang et al., 2018).
Since this variance reduced approximate gradients introduce an approximate full gradient to make fully sparse update in cardinality constraint problem.
B P ROOFS OF K EY L EMMAS
In this section, we prove three key lemmas.
B. P ROOF OF L EMMA
Proof.
Let ν i t
= ( ∇ I f i t ( w t ) − ∇
I f i t ( w ∗ )) − ( ∇ I F ( w t ) − ∇ I F ( w ∗ )) .
We have
E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)|B| (cid:88) i t ∈B ν i t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =|B| 2 E (cid:88)
i t
[Equation]
i (cid:48)
t ∈B ν (cid:62)
[Equation]
t ν i (cid:48) t =
[Equation]
(cid:88)
[Equation]
(cid:54)
[Equation]
∈B ν (cid:62)
[Equation]
ν
[Equation]
(cid:48)
[Equation]
|
[Equation]
| E (cid:107) ν
i (cid:107)= |B|
[Equation]
|B|
[Equation]
( n
[Equation]
1) (cid:88) i (cid:54)
=
i (cid:48) ν (cid:62) i ν i (cid:48)
[Equation]
|
[Equation]
|
[Equation]
(cid:107) ν
i (cid:107)= |B| −|B| n ( n − 1) (cid:88)
i,i (cid:48)
ν (cid:62) i ν i (cid:48) − |B| −|B
[Equation]
This proof is similar to that of (Zheng & Kwok, 2016).
(cid:3)
B. P ROOF OF L EMMA
Proof.
It is obvious that the stochastic variance reduced gradient satisﬁes
[Equation]
Thus
[Equation]
∇ g ( w t ) is an unbiased estimator of the full gradient ∇ F ( w t ) .
Now
[Equation]
we bound E (cid:107)∇ g I ( w t ) (cid:107) 2 .
For any i ∈ { 1 , 2 , · · · , n }
[Equation]
we deﬁne the following function
[Equation]
It is easy to ﬁnd that ∇ h
i ( w ∗ ) = 0 , which implies that h i ( w ∗ ) =
min w h i ( w ) .
For any vector w , we have
[Equation]
where the second inequality follows from the RSS condition, and the second equality holds due to the fact that (cid:104)∇ h i ( w ) , ∇ I
h i ( w )
[Equation]
= (cid:107)∇ I h i ( w ) (cid:107) 2 , and the last equality holds due to the fact that η = 1 ρ + (cid:98) s minimizes the function.
Then, we have
[Equation]
Since the sampling i is chosen uniformly from { 1 , 2 , · · ·
[Equation]
n }
[Equation]
we have
[Equation]
where the last inequality follows from the restricted strong convexity of F ( w )
[Equation]
And we use a simple underlying consensus that there must exist a uniform factor δ > 1 which makes the following inequality true:
[Equation]
This property is called the extended restricted strong convexity used in the following proof.
Therefore, we have
[Equation]
where the ﬁrst inequality follows from (cid:107) a + b + c + d (cid:107) 2 ≤ 4 (cid:107) a (cid:107) 2 +4 (cid:107) b (cid:107) 2 +4 (cid:107) c (cid:107) 2 +4 (cid:107) d (cid:107) 2 , the third equality follows from Lemma 4, the second inequality follows from E (cid:107) x − E x
and the last inequality follows from (16).
Due to the fact that E (cid:107)∇ S g
I ( w t
[Equation]
(cid:107)
[Equation]
≤ 1 /k E
(cid:107)∇ g I ( w t ) (cid:107)where I = supp ( w ∗ ) ∪ supp ( (cid:101) w r +1 ) ∪ supp ( (cid:101) w r ) and S =
G j t ∪ supp ( (cid:101) w r ) , we have
[Equation]
where k is the number of blocks.
Note that we take expectation with respect to randomized block in the above equation.
This completes the proof.
[Equation]
B. P ROOF OF L EMMA
Proof.
By the deﬁnition ∇ g ( w t ) = ∇ f i t ( w t ) −∇
f i t ( (cid:101) w )
+ D i t ∇ F ( (cid:101) w ) , we have E [ D i t ∇ F ( (cid:101) w )]
= ∇ F ( (cid:101) w ) .
Therefore, ∇ g ( w t ) is an unbiased estimator of ∇ F ( w t ) .
Based on (16), we have the following result:
[Equation]
where the ﬁrst inequality follows from (cid:107) a + b + c (cid:107) 2 ≤ 3 (cid:107) a (cid:107) 2 +3 (cid:107) b (cid:107) 2 +3 (cid:107) c (cid:107) 2 , the second inequality follows from (cid:107)
a + b
[Equation]
≤
[Equation]
a
[Equation]
b
[Equation]
2 , the third inequality follows from E (cid:107) w − E w (cid:107) 2 ≤ E (cid:107) w (cid:107)with E [ ∇ I f i t ( (cid:101) w ) − ∇
I f i t ( w ∗ )]
= ∇ I F ( (cid:101) w ) − ∇
I F ( w ∗ ) , and E (cid:107)∇ I f i ( w t ) − ∇ I f i ( w ∗ ) (cid:107) 2 ≤ 4 δρ + (cid:98) s [ F ( w t ) − F ( w ∗ )] with δ > 1 , which is also satisﬁed for (cid:101) w with δ > 1 .
Since D m = max v =1 , ··· ,d 1 p v , we bound the (cid:107)∇ I F ( (cid:101) w ) − D ∇ I F ( (cid:101) w ) (cid:107) 2 term as follows:
[Equation]
[Equation]
Due to the fact that E (cid:107)∇ S g I ( w t ) (cid:107) 2 ≤ 1 k E (cid:107)∇ g I ( w t ) (cid:107) 2 , we have
[Equation]
where k is the number of blocks.
Note that we take expectation with respect to randomized block in the above equation.
This completes the proof.
[Equation]
C P ROOFS OF T HEOREM 1 AND C OROLLARIES 1 AND
In this section, we give the detailed proofs of Theorem 1 and Corollaries 1 and 2, which can guarantee the convergence properties of our SBCD-HTP.
C. P ROOF OF T HEOREM
Proof.
In this subsection
[Equation]
we provide the proof of Theorem 1.
Let w t
+
=
w t − η ∇ S g I ( w t ) and
I =
I ∗ ∪I r ∪I r +1 , where I ∗ =
supp ( w ∗ ) , I r = supp ( (cid:101) w r ) and I r +1 = supp ( (cid:101) w r +1 ) .
Conditioning on w t
[Equation]
we have the following expectation
[Equation]
where the ﬁrst inequality follows from the expectation with respect to randomized block
[Equation]
the second inequality follows from our extended restricted strong convexity in the proof of Lemma
[Equation]
and the third inequality holds by Lemma 5.
Notice that (cid:101) w = w 0 = (cid:101) w r − 1 .
Summing (22) over t = 0 , · · · , m − 1 and taking expectation with respect to all randomness
[Equation]
we have
[Equation]
Notice that the convergence property used here is similar to the proofs of the convergence results for many hard thresholding algorithms (Chen & Gu, 2016; 2017; Li et al., 2016b).
Moreover, we can obtain
[Equation]
where the second inequality holds due to the fact that we have the following results: When F ( (cid:101) w r ) −
F ( w m ) > 0 , by using Lemma 3 on w m and (cid:101) w r , we have
[Equation]
[Equation]
where the last inequality holds due to the fact that we assume that there exists a constant factor σ>making this inequality true.
Then by using the RSS condition, we have
[Equation]
After simpliﬁcation
[Equation]
we have F ( (cid:101) w r ) − F ( w m ) ≤ ρ +
[Equation]
s 2 σ (cid:107) (cid:101) w r − w m
(cid:107) 2 .
Using Lemma
[Equation]
we have
[Equation]
27) and
[Equation]
When F ( (cid:101) w r ) − F ( w m ) < 0 , we can omit − (16 δρ + (cid:98) s η 2 m (1+ n −|B| |B| ( n − 1) )
− 2 mηδ )
E
[ F
[Equation]
w r ) − F ( w m )] directly since 16 δρ + (cid:98) s η 2 m (1+ n −|B| |B| ( n − 1) )
− 2 mηδ is less than zero.
Because this is the simple situation, we only analyze the complex one in our whole proof.
Therefore, we have the following inequality:
[Equation]
Since (cid:101) w r
=
H k ( w m ) , i.e., (cid:101) w r is the best s -sparse approximation of w m , then we have the following result due to Lemma 2,
[Equation]
Let α = 1 + 2 √ s ∗ √ s − s ∗
[Equation]
and by combining (29) and (30)
[Equation]
we have
[Equation]
(31) where the last inequality follows from the RSC condition and the deﬁnition of ˜ I , i.e., ˜
I
=
supp ( HT ( ∇ F ( w ∗ ) , 2 s )) ∪ supp ( w ∗ ) .
Through proper simpliﬁcation, we have the following result:
[Equation]
32) where the ﬁrst term of the LHS of (31) is omitted due to the fact that the coefﬁcient of E (cid:107) (cid:101) w r − w ∗
(cid:107)is larger than zero, when 8 ρ + (cid:98) s mη
[Equation]
s )
> 0 , which can be naturally satisﬁed under our parameter setting with s ≥ κ (cid:98) s s ∗ .
Following from (32), we have
[Equation]
where β =
[Equation]
and ω = 8 ρ + (cid:98) s η (1+ n −|B| |B|
[Equation]
By applying (33) recursively, then we have the following desired bound with β ≤ 12 < 1 :
[Equation]
This completes the proof.
[Equation]
C. P ROOF OF C OROLLARY
Proof.
We then give the proof for the statistical estimation analysis.
Following from the RSC condition of F ( · ) , we have
[Equation]
Let φ = ( 12 ) r E
[ F ( (cid:101) w 0 )
− F
[Equation]
w ∗ )]
+ η 1 − ω
[Equation]
˜ I F ( w ∗ ) (cid:107) 2 .
Combining (34) and (35)
[Equation]
we have
[Equation]
Using the duality of norms
[Equation]
we have
[Equation]
Combining (36), (37) and (cid:107) E w (cid:107) 2 ≤ E (cid:107) w (cid:107) 2 , we have
[Equation]
Let a = E (cid:107) (cid:101) w r − w ∗
cid:107) .
From the above inequality
[Equation]
we solve the following quadratic function with respect to a ,
[Equation]
which yields the bound
[Equation]
This completes the proof.
[Equation] [Equation]
Now we show that we can guarantee that β ≤ 1 by providing the appropriate choices of constants with η , k and m used in the theorem.
More speciﬁcally, let η ≤ C 3 ρ
+ (cid:98) s ≤ 1 48 ρ
+ (cid:98) s , and |B| = 1 , then we have
[Equation]
If η ≥ C 2 ρ + (cid:98) s with C 2 ≤ C 3 , then we have
[Equation] [Equation]
If we guarantee
[Equation]
, then we have
[Equation]
43) where δ > 1 .
If we choose C 2 = 160 , C 3 = 148 , C 4 = 1800 , k = 10 and δ = 2 , then we have β ≤ 12 .
Note that the convergence rate β does not include α , therefore, the required value of k comes from the restricted constraint of the coefﬁcient of (cid:107) (cid:101) w r − w ∗ (cid:107) 2 , which implys that s =
Ω( κ (cid:98) s s ∗ ) .
C. P ROOF OF C OROLLARY
Proof.
[Equation] [Equation]
Let (cid:37) 1 , (cid:37) 2 , · · · be a non-negative sequence of random variables which is deﬁned as
[Equation]
For a ﬁxed (cid:15) > 0 , it follows from (34) and the Markov inequality,
[Equation]
[Equation]
For a given ζ ∈ (0 , 1) , let the right side of (46) be not greater than ζ , which requires
[Equation] [Equation]
Therefore, the result in (44) holds with probability at least 1 − ζ .
Thus, we need O (log( 1 (cid:15) ))
outer iterations to get (cid:101) w r satisfying (44).
Since within each outer-iteration, we need to calculate a full gradient and m stochastic variance reduced gradients with mini-batch size |B| , the overall oracle complexity is O (( n + κ (cid:98) s |B| k ) log 1 (cid:15) ) .
Note that in high-dimensional region, the effect of support of snapshot point can be ignored.
On the other hand, we only perform a hard thresholding operation at each outer iteration.
Then we have
[Equation] [Equation]
Thus, the hard thresholding oracle complexity is O (log( 1 (cid:15) )) .
This completes the proof.
[Equation]
D P ROOF OF T HEOREM OF S PARSE VARIANT
In this section, we theoretically analyze the convergence properties of our Serial Sparse SBCD-HTP (S 2 BCD-HTP).
Input: number of outer-loops R , number of inner-loops m , step size η , sparsity level s ;
Initialize: (cid:101) w 0 ; 1: for r = 0 , 1 , . . .
, R − 1 do 2: w 0 = (cid:101)
w = (cid:101) w r ; 3: ∇ F ( (cid:101) w ) = 1 n (cid:80) ni
=1 ∇ f i ( (cid:101) w ) ; 4: (cid:101) G = supp ( (cid:101) w ) ; 5: for t = 0 , 1 , . . .
, m − 1 do 6:
Randomly sample i t from { 1 , 2 , . .
. , n } uniformly; 7:
Randomly sample j t from { 1 , 2 , . .
. , k } uniformly; 8:
T i t := support of sample i t ; 9:
S = (cid:101) G ∪ G j t ; 10: ∇ S g ([ w t ] T it ) = ∇ S
f i t ([ w t ] T it ) − ∇
S f i t ([ (cid:101) w ] T it )
+ D i t ∇ S F ( (cid:101) w ) ; 11: w t +1 =
w t − η ∇ S g ([ w t ] T it ) ; 12: end for 13: (cid:101) w r
+
=
HT ( w m , s ) ; 14: end for
Output: (cid:101) w R .
D.
P ROOF
Proof.
We start with the iterate difference between w t +1 and w ∗ .
By expanding iterate difference and taking expectation with respect to all randomness, we get
E (cid:107) w t +1 − w ∗ (cid:107) 2 = E (cid:107) w t − η ∇ S g I ( w t ) − w ∗ (cid:107)= (cid:107) w t − w ∗
cid:107)
+ η 2 (cid:107)∇
[Equation]
where the last inequality holds by the unbiasedness of the sparse gradient estimator
∇ g I ( w t ) and the expectation with respect to randomized block.
By using the ρ − (cid:98) s -strongly convex condition, we get the bound for (cid:104)∇ F ( w t ) , w ∗ − w t (cid:105) as follows:
[Equation]
Due to the support set I , we need the extended RSC condition with δ > 1 , which has been introduced in the proof of Lemma 5.
Based on this underlying consensus, we can obtain the bound for (cid:104)∇ F I ( w t ) , w ∗
[Equation]
w t (cid:105) .
[Equation]
Using Lemma
[Equation]
we have the following sparse variance gradient bound,
[Equation]
By combining the above inequalities
[Equation]
we have
[Equation]
where the second inequality holds by Lemma 6.
Summing the above inequality over t = 0 , · · · , m − 1 and taking expectation with respect to all randomness in this epoch, we have
[Equation]
Using (28)
[Equation]
we obtain
[Equation]
Since (cid:101) w r
=
HT ( w m
[Equation]
s ) , i.e., (cid:101) w r is the best s -sparse approximation of w m , and due to Lemma 2, then we have following result,
[Equation]
[Equation]
where α = 1 + 2 √ s ∗ √ s − s ∗
[Equation]
and we use ρ − (cid:98) s -strongly convex of F ( w ) to bound (cid:107) (cid:101) w r − 1 − w ∗ (cid:107) 2 in the last inequality.
Through proper simpliﬁcation
[Equation]
we can obtain
[Equation]
where the second term of LHS of (56) is omitted due to the fact that the coefﬁcient of E (cid:107) (cid:101) w r − w ∗
(cid:107)is larger than zero, when 6 ρ + (cid:98) s mη 2 − mη +
[Equation]
( d − s ∗ ) δρ + (cid:98) s ( d − s ) > 0 , which can be naturally satisﬁed under our parameter setting with s ≥ κ (cid:98) s s ∗ .
Therefore, we can obtain the following result,
[Equation]
where ˜ I = supp( w ∗ ) ∪ supp( HT ( ∇ F ( w ∗ ) , 2 s ))
We deﬁne ξ :=
[3+12( D 2 m − 1)]
η
[Equation]
+
[Equation]
In order to obtain a linear convergence rate, we have to bound β :=
[Equation]
If we choose η ≤
[Equation]
m ρ + (cid:98) s , then the second term of β will be less than 12 , i.e., (12+24( D 2 m −
[Equation]
ρ +
[Equation]
As for the ﬁrst term, when m ≥ 500 D 4 m κ (cid:98) s D 2 m − 1 with η ≥
[Equation]
+
[Equation]
s , δ = 4 and k = 10 , we have 1 (
[Equation]
−
[Equation]
+
[Equation]
s ≤ 14 .
Thus, we get the linear convergence for our serial sparse SBCD-HTP (S 2 BCDHTP).
[Equation]
which means that the total gradient evaluation oracle complexity is O (( n
[Equation]
κ
[Equation]
k log 1 (cid:15) ) , and the hard thresholding oracle complexity is O (log( 1 (cid:15) )) .
This completes the proof.
(cid:3)
E P ROOF OF T HEOREM
In this section, we analyze the convergence properties of the proposed Asynchronous Sparse SBCDHTP (ASBCD-HTP).
We ﬁrst have to specify the iterates labeling order used in our asynchronous analysis.
We use “After Read” labeling order (Leblond et al., 2017) in our proof, which enjoys a simpler analysis but requires the order of randomly sampling step to be an uniform distributed sample.
In order to give a clear proof, we adopt the “After Read” labeling order and make the following assumptions: “After Read” labeling order:
Inconsistently read the iterate ˆ w t ; 2.
Increase iterates counter t + 1 and sample a random sample i t
+1 ; 3.
Compute the update − η ∇ g ( ˆ w t ) ; 4.
Atomic write the update to shared variable coordinately.
Assumption 4 The labeling order increases after Step 8 in Algorithm 2 ﬁnished, and thus the future perturbation is not considered in the effect of asynchrony in the current step.
Assumption 5 We assume that uniform distributed samples and the independence of the sample i t
+1 with ˆ w t .
Following (Leblond et al., 2017), we can explicit the effect of asynchrony as follows:
[Equation] [Equation]
where w t is the t -th inner iteration solution.
In the proof of Theorem 2, we use w t to represent the temporary solution of the t -th iteration for a clean representation.
And G u is a diagonal matrix with entries in { 0 , +1 } .
This explicitly deﬁnes the coordinate perturbation from the past updates.
Here, τ denotes the maximum number of overlaps between concurrent threads.
We also denote ∆ = max j =1 , ··· ,d p j , which provides a measure of sparsity following (Leblond et al., 2017).
E. P ROOF
Proof.
We analyze our asynchronous algorithm based on the “perturbed iterate analysis” framework (Mania et al., 2017) with “After Read” labeling order.
By expanding the iterate difference and taking expectation with respect to the sample i t
+1 , we get
[Equation]
where ˆ w t is the “perturbed” iterate with perturbation ψ and the last inequality follows from the expectation with respect to randomized block.
We need to bound the term, 2 η (cid:104)
[Equation]
Since i t +1 is independent to ˆ w t
[Equation]
we can write:
[Equation]
We can use ρ − (cid:98) s -strong convexity bound of F ( ˆ w ) with constant factor δ > 1 as well as a squared triangle inequality to get:
[Equation]
Then we make an identical deformation
[Equation]
and we have
[Equation]
Putting it all together
[Equation]
we get the initial recursive inequity
[Equation]
written here explicitly:
[Equation]
where a t := E i t +1 (cid:107) w t − w ∗ (cid:107) 2 and ˆ e t :=
E i t +
[ F ( ˆ w t ) − F ( w ∗ )] .
From Lemma 1 in (Leblond et al., 2017), we can bound the asynchronous terms (cid:107) ˆ w t − w t (cid:107) 2 , (cid:104)
ˆ w t − w t and ∇ g
I ( ˆ w t ) (cid:105) as follows.
[Equation] [Equation]
) where G u are d × d diagonal matrices whose entries in { 0 , +1 } .
Each update in ˆ w t is already in w t – this is the case of 0 .
On the contrary, some updates might be late: this is the case of +1 .
Then by combining (68), (69) and (70), we get
[Equation]
where C 1 := 1 + √ ∆ τ and C 2 := √ ∆ + ηρ − (cid:98) s C 1 .
τ represents the mawimum number of overlaps between concurrent threads.
And ∆ = max j =1 , ··· ,d p j , which is a measure of sparsity with 1 /n ≤ ∆ ≤ 1 .
Using Lemma 6 to bound the sparse variance term, the above inequality becomes
[Equation]
where ˜ e := E [ F ( (cid:101) w ) − F ( w ∗ )] , ˆ e t :=
E [ F ( ˆ w t ) − F ( w ∗ )] .
Note that we have (cid:101) w = w 0 = (cid:101) w r − 1 .
Summing (72) over t = 0 , · · · , m − 1 and taking expectation with all randomness in this epoch, we get
[Equation]
where a m = E (cid:107) w m − w ∗ (cid:107) 2 , and a 0 = E (cid:107) w 0 − w ∗ (cid:107) 2 .
Then we focus on upper bounding the third term on the RHS of (73),
[Equation]
Substituting the above inequality into (73)
[Equation]
we obtain
[Equation]
Then we bring (28) into the above inequality
[Equation]
we have
[Equation]
where the second inequality holds since we use (28) and make a proper simpliﬁcation
[Equation]
Using Lemma
[Equation]
we have
[Equation]
77) where α = 1 + 2 √ s ∗ √ s − s ∗ , ˜ a r = (cid:107) (cid:101)
w r − w ∗ (cid:107) 2 , and ˜ I
= supp( w ∗ ) ∪ supp( HT ( ∇ F ( w ∗ ) , 2 s )) .
The ﬁrst term of LHS of (77) is omitted due to the coefﬁcient of ˜ a r is larger than zero, when 6 ρ +
[Equation]
s m
[Equation]
τC
[Equation]
+ (cid:98) s ( d − s ) > 0 , which can be naturally satisﬁed under our parameter settings with s ≥ κ (cid:98) s s ∗ .
Therefore, we obtain the following recursive inequality,
[Equation]
We assume that the following settings has satisﬁed with the above constraint and then by choosing m = 120 κ
[Equation]
s , η =
[Equation]
s , k = 10 , we get
[Equation]
where ξ = η (3+
( D
[Equation]
[Equation] [Equation]
(1 −
[Equation]
(cid:98) s η ( τC 2 + C 1 )) .
In order to ensure linear speed up, τ needs to satisfy the following constraint,
[Equation]
We assume that 37( ρ + (cid:98) s ) 2 ≥ D 2 m and δ = 10 , then by simply setting τ ≤ min { 3 5 √ ∆ , 2 κ (cid:98) s , (cid:113) 2 κ (cid:98) s √ ∆ } , the above constraint is satisﬁed with θ ≤ 0 .
7538 , which implies that the oracle gradient evaluation and hard thresholding oracle complexities are O (( n + κ (cid:98) s k ) log 1 (cid:15) ) and O (log( 1 (cid:15) )) , respectively.
We can simply infer the statistical estimation result as before.
Here, we will omit this part.
This completes the proof.
[Equation]
E. D ISCUSSION ABOUT S PARSE V ARIANCE B OUND
In the dense update case (i.e., D m = 1 ), Lemma 6 is very similar to Lemma 5 when |B| = 1 .
In the sparse update case, Lemma 6 highly correlates with the sparsity of datasets ( ∝ D 2 m ), which could be much loose in some extreme cases.
If D m is very large (imaging a dataset with some dimensions contain only one entry among the n samples, so D m = n ), the step size of sparse (asynchronous)
SBCD-HTP will be much smaller than that of SBCD-HTP.
Our algorithms can still converge to the optimal solution up to the statistical error.
Actually, it is still an open problem whether we can have a tighter variance bound in the sparse update setting that is uncorrelated with D m .
F M ORE E XPERIMENTAL R ESULTS
[Table]
F. R EAL -W ORLD D ATASETS AND E QUIPMENTS
We summarize the detailed information of all the real-world datasets in Table 2, including rcv1train, rcv1-test, real-sim, news20 and E2006-TFIDF.
All these datasets are provided in the LibSVM website 1 .
The E2006-TFIDF dataset includes 16,087 training data points and 150,360 features.
Instead of using all features, we randomly select 80,000 features for training.
As for the news dataset, we randomly choose 30,000 features for training.
F. B ASELINE M ETHODS
We compare our serial dense SBCD-HTP method with the state-of-the-art methods:
• Fast Gradient with Hard Thresholding (FG-HT)
(Yuan et al., 2014):
This method is based on a standard gradient descent step and combined with hard thresholding to solve sparsity-constrained problems.
• Stochastic Gradient with Hard Thresholding (SG-HT)
(Nguyen et al., 2017):
This method combines stochastic gradient descent with hard thresholding for solving large-scale sparsity-constrained problems.
• Stochastic Variance Reduced Gradient with Hard Thresholding (SVRG-HT) (Li et al., 2016b):
This method incorporates the variance reduction technique in (Johnson & Zhang, 2013) into the hard thresholding algorithm to improve the efﬁciency of stochastic optimization.
• Accelerated Stochastic Block Coordinate Gradient Descent with Hard Thresholding (ASBCDHT) (Chen & Gu, 2016):
Block coordinate descent (BCD) is used for solving sparsity-constrained problems.
• Fast Newton Hard Thresholding Pursuit (FNHTP) (Chen & Gu, 2017): This method tries to iteratively approximate the inverse Hessian matrix and combines a Newton algorithm with hard thresholding.
For the proposed Serial Sparse SBCD-HTP (S 2 BCD-HTP) and Asynchronous sparse parallel
SBCD-HTP (ASBCD-HTP) methods, we compare them with the following asynchronous parallel sparsity-constrained methods:
• Asynchronous Accelerated Stochastic Block Coordinate Descent Hard Thresholding
(A 2 SBCD-HT):
This algorithm is the asynchronous variant of ASBCDHT by using our parallel framework.
• Asynchronous Stochastic Gradient with Hard Thresholding (ASG-HT):
This method is an extension of SG-HT from serial dense setting to asynchronous sparse setting.
• Asynchronous Stochastic Variance Reduced Gradient Hard Thresholding (ASVRGHT)
(Li et al., 2016a):
This method incorporates multicore structure to compute stochastic variance reduced gradient in a parallel way.
F. F ACE R ECOGNITION D ATASETS
Although there are many datasets available for face recognition, we choose a common dataset (i.e. the Extended Yale B database (Georghiades et al., 2001)).
The Extended Yale B database contains 2,414 frontal-face images of 38 people under different controlled lighting conditions (Georghiades et al., 2001).
For each individual, we randomly choose 26 images for training and 15 images for testing.
F. E XPERIMENTAL S ETUP
For each sparsity-constrained algorithm
, the sparse level makes great difference to the solution of sparse representation, especially at different noise levels.
Therefore, in order to approach the best performance of all these algorithms, we change the sparsity parameter within a certain range for each algorithm.
Thus, we can make sure that all these algorithms achieve the best recognition rates in the parameter setting.
In all settings of the experiments, the images are down-sampled to 32 × 32 pixels.
[Figure] [Figure]
Based on the sparse representation-based classiﬁcation algorithm (SRC) (Wright et al., 2008)
[Figure]
a series of processing operations are made to the above dataset.
We ﬁrst rescale the training matrix into [0,1] for the convenience of adding noise, and then add Gaussian noise with zero mean and standard deviation ϑ .
Finally, we normalize the columns of the training matrix to have unit (cid:96) 2 -norm.
The parameter settings of comparison algorithms for face recognition task are the same as those described in Section 6.1.
F. R ESULTS
We ﬁrst measure the effect of mini-batch size |B| in the convergence of SBCD-HTP on rcv1-train, as shown in Figure 6.
It is obvious that the large mini-batch size can accelerate the convergence rate and make SBCD-HTP to obtain a better solution.
This result is exactly in line with our theoretical analysis.
However, when the mini-batch size is relatively large, it will deteriorate the performance of running time.
One can simply choose the mini-batch size to 3 or 5 as suggested in Figure 6.
Then, we compare the performance of some hard thresholding algorithms, including SG-HT, SVRGHT and ASBCDHT, with our SBCD-HTP for solving face recognition tasks on the Extended Yale B dataset.
As shown in Figure 5, SBCD-HTP can achieve the best performance among these stochastic hard thresholding methods in a short time.
It is clear that SBCD-HTP can not only optimize the objective function efﬁciently but also obtain a better representative solution on real applications, which is consistent with our theoretical results as described in Section 4.2. This result also shows that our SBCD-HTP has a better generalization accuracy than other hard thresholding algorithms
[Figure]
F. L INEAR / LOGISTIC REGISTRATION
We show more practical evaluation results for sparse linear regression and sparse logistic regression problems.
All hardware environment and parameter settings are the same as in Section 6.
For
SBCD-HTP, from Figures 7 and 8, we can come to the same conclusion as we did before, that is in both sparse linear regression and sparse logistic regression, our algorithm converges faster than the baseline algorithms in terms of effective passes and running time.
However, we ﬁnd an interesting result, which is that for the E2006-TFIDF dataset, all the algorithms converge with high precision solutions, so that we have to use some methods to magnify their differences.
Of course, our SBCDHTP has the best performance, but the gap is not that big.
We think that this is due to ease with which E2006-TFIDF dataset can be optimized, since this phenomenon does not occur on other datasets.
For S 2 BCD-HTP and ASBCD-HTP, from Figure 9, our S 2 BCD-HTP and ASBCD-HTP far exceed the baseline algorithms in terms of running time.
Therefore, we can conﬁrmedly obtain the conclusion that our Sparse and Asynchronous algorithms outperform the state-of-the-art hard thresholding algorithms for sparse datasets.
This is a signiﬁcant improvement for hard thresholding algorithms to solve sparse learning problems in the high-dimensional asynchronous setting. [Figure] j [Figure]
c t i v
[Figure]
G
[Figure]
j
[Figure]
c t i
[Figure]
F.7. U PPER BOUND
In order to conduct an experiment on the upper bound of Theorem 1.
We ﬁrst generate some n × d synthetic matrices X , each row of which is drawn independently from a d -dimensional Gaussian distribution with mean 0 and covariance matrix Σ ∈ R d × d .
The response vector is generated from the model y
=
Xw ∗ + (cid:15) , where w ∗ ∈ R
d is the s ∗ -sparse coefﬁcient vector, and we need to generate the noise (cid:15) drawn from a multivariate normal distribution N (0 , σ 2 I ) with σ 2 = 0 .
The nonzero entries in w ∗ are sampled independently from a uniform distribution over the interval [ − 1 , 1] .
For the experiments, we construct the following synthetic data set: n = 5000 , d = 10000 , s ∗ = 500 , and the diagonal entries of the covariance matrix Σ are set to 1, and the other entries are set to 0 .
[Figure]
We set the step-size η = 1 48 ρ (cid:98) s and batch size |B| = 1 during our experiments.
The upper bound is then computed according to Theorem 1.
Under this condition, we obtain the result, as is shown in
Figure 10.
As we can see that the upper bound converges faster at ﬁrst and then it becomes slow since the statistical error becomes the main factor.
The convergence performance of SBCD-HTP is always bounded below the black line, which demonstrates the correctness of our theorem.
F. L INEAR SVM
In this subsection, we consider the sparsity-constrained
L 2 -SVM problem (i.e., min w
[Equation]
n
[Equation]
n i =
[Equation]
max
[Equation]
[Equation]
− y
[Equation]
i )
w
[Equation]
x
[Equation]
i ) } (cid:1) 2 +
[Equation]
[Equation]
w (cid:107) 2 , subject to (cid:107) w (cid:107) 0 ≤ s ), where y ( i ) is the known label and x ( i ) is the given sample.
We ﬁx the regularization parameter λ
= 10 − 4 .
The other parameters are also same to the above experiments.
The objective value gap and running time of the baselines are provided in Figure 11.
We can see that we also make a similar observation as from the previous results.
SBCD-HT performs the best among these considered baselines.
[Figure]
[Figure]