NOODL: P ROVABLE O NLINE D ICTIONARY L EARNING AND S PARSE C ODING
A BSTRACT
We consider the dictionary learning problem, where the aim is to model the given data as a linear combination of a few columns of a matrix known as a dictionary , where the sparse weights forming the linear combination are known as coefﬁcients .
Since the dictionary and coefﬁcients, parameterizing the linear model are unknown, the corresponding optimization is inherently non-convex.
This was a major challenge until recently, when provable algorithms for dictionary learning were proposed.
Yet, these provide guarantees only on the recovery of the dictionary, without explicit recovery guarantees on the coefﬁcients.
Moreover, any estimation error in the dictionary adversely impacts the ability to successfully localize and estimate the coefﬁcients.
This potentially limits the utility of existing provable dictionary learning methods in applications where coefﬁcient recovery is of interest.
To this end, we develop NOODL: a simple Neurally plausible alternating Optimization-based Online Dictionary Learning algorithm, which recovers both the dictionary and coefﬁcients exactly at a geometric rate, when initialized appropriately.
Our algorithm, NOODL, is also scalable and amenable for large scale distributed implementations in neural architectures, by which we mean that it only involves simple linear and non-linear operations.
Finally, we corroborate these theoretical results via experimental evaluation of the proposed algorithm with the current state-of-the-art techniques.
I NTRODUCTION
Sparse models avoid overﬁtting by favoring simple yet highly expressive representations.
Since signals of interest may not be inherently sparse, expressing them as a sparse linear combination of a few columns of a dictionary is used to exploit the sparsity properties.
Of speciﬁc interest are overcomplete dictionaries, since they provide a ﬂexible way of capturing the richness of a dataset, while yielding sparse representations that are robust to noise; see Mallat and Zhang (1993); Chen et al.
Donoho et al.
In practice however, these dictionaries may not be known, warranting a need to learn such representations – known as dictionary learning (DL) or sparse coding (Olshausen and Field, 1997).
Formally, this entails learning an a priori unknown dictionary
A ∈ R n × m and sparse coefﬁcients x ∗ ( j ) ∈ R m from data samples y ( j ) ∈ R n generated as
[Equation]
This particular model can also be viewed as an extension of the low-rank model (Pearson, 1901).
Here, instead of sharing a low-dimensional structure, each data vector can now reside in a separate low-dimensional subspace.
Therefore, together the data matrix admits a union-of-subspace model.
As a result of this additional ﬂexibility, DL ﬁnds applications in a wide range of signal processing and machine learning tasks, such as denoising (Elad and Aharon, 2006), image inpainting (Mairal et al., 2009), clustering and classiﬁcation (Ramirez et al., 2010; Rambhatla and Haupt, 2013; Rambhatla et al., 2016; 2017; 2019b;a), and analysis of deep learning primitives (Ranzato et al., 2008;
Gregor and LeCun, 2010); see also Elad (2010), and references therein.
Notwithstanding the non-convexity of the associated optimization problems (since both factors are unknown), alternating minimization-based dictionary learning techniques have enjoyed signiﬁcant success in practice.
Popular heuristics include regularized least squares-based (Olshausen and Field, 1997; Lee et al., 2007; Mairal et al., 2009; Lewicki and Sejnowski, 2000; Kreutz-Delgado et al., 2003), and greedy approaches such as the method of optimal directions (MOD) (Engan et al., 1999) and k-SVD (Aharon et al., 2006).
However, dictionary learning, and matrix factorization models in general, are difﬁcult to analyze in theory; see also Li et al.
(2016a).
To this end, motivated from a string of recent theoretical works (Gribonval and Schnass, 2010; Jenatton et al., 2012; Geng and Wright, 2014), provable algorithms for DL have been proposed recently to explain the success of aforementioned alternating minimization-based algorithms (Agarwal et al., 2014; Arora et al., 2014; 2015).
However, these works exclusively focus on guarantees for dictionary recovery.
On the other hand, for applications of DL in tasks such as classiﬁcation and clustering – which rely on coefﬁcient recovery – it is crucial to have guarantees on coefﬁcients recovery as well.
Contrary to conventional prescription, a sparse approximation step after recovery of the dictionary does not help; since any error in the dictionary – which leads to an error-in-variables (EIV) (Fuller, 2009) model for the dictionary – degrades our ability to even recover the support of the coefﬁcients (Wainwright, 2009).
Further, when this error is non-negligible, the existing results guarantee recovery of the sparse coefﬁcients only in (cid:96) 2 -norm sense (Donoho et al., 2006).
As a result, there is a need for scalable dictionary learning techniques with guaranteed recovery of both factors.
1. S UMMARY OF O UR C ONTRIBUTIONS
In this work, we present a simple online DL algorithm motivated from the following regularized least squares-based problem, where S ( · ) is a nonlinear function that promotes sparsity.
[Equation]
Although our algorithm does not optimize this objective, it leverages the fact that the problem (P1) is convex w.r.t A , given the sparse coefﬁcients { x ( j ) } .
Following this, we recover the dictionary by choosing an appropriate gradient descent-based strategy (Arora et al., 2015; Engan et al., 1999).
To recover the coefﬁcients, we develop an iterative hard thresholding (IHT)-based update step (Haupt and Nowak, 2006; Blumensath and Davies, 2009), and show that – given an appropriate initial estimate of the dictionary and a mini-batch of p data samples at each iteration t of the online algorithm – alternating between this IHT-based update for coefﬁcients, and a gradient descent-based step for the dictionary leads to geometric convergence to the true factors, i.e., x ( j ) → x ∗ ( j ) and A ( t ) i
→ A ∗ i as t →∞ .
In addition to achieving exact recovery of both factors, our algorithm – Neurally plausible alternating Optimization-based Online Dictionary Learning (NOODL) – has linear convergence properties.
Furthermore, it is scalable, and involves simple operations, making it an attractive choice for practical DL applications.
Our major contributions are summarized as follows:
• Provable coefﬁcient recovery:
To the best of our knowledge, this is the ﬁrst result on exact recovery of the sparse coefﬁcients { x ∗ ( j ) } , including their support recovery, for the DL problem.
The proposed IHT-based strategy to update coefﬁcient under the EIV model, is of independent interest for recovery of the sparse coefﬁcients via IHT, which is challenging even when the dictionary is known; see also Yuan et al.
(2016) and Li et al.
(2016b).
• Unbiased estimation of factors and linear convergence: The recovery guarantees on the coefﬁcients also helps us to get rid of the bias incurred by the prior-art in dictionary estimation.
Furthermore, our technique geometrically converges to the true factors.
• Online nature and neural implementation: The online nature of algorithm, makes it suitable for machine learning applications with streaming data.
In addition, the separability of the coefﬁcient update allows for distributed implementations in neural architectures (only involves simple linear and non-linear operations) to solve large-scale problems.
To showcase this, we also present a prototype neural implementation of NOODL.
In addition, we also verify these theoretical properties of NOODL through experimental evaluations on synthetic data, and compare its performance with state-of-the-art provable DL techniques.
1. R ELATED W ORKS
With the success of the alternating minimization-based techniques in practice, a push to study the
DL problem began when Gribonval and Schnass (2010) showed that for m = n , the solution pair ( A ∗ , X ∗ ) lies at a local minima of the following non-convex optimization program, where X
= [ x (1) , x (2) , . . .
, x ( p ) ] and Y =
[ y (1) , y (2) , . .
, y ( p ) ] , with high probability over the randomness of the coefﬁcients,
[Equation]
[Table]
Following this
[Table]
Geng and Wright (2014) and Jenatton et al.
(2012) extended these results to the overcomplete case ( n < m )
[Table]
and the noisy case, respectively.
Concurrently, Jung et al.
(2014; 2016) studied the nature of the DL problem for S ( · ) = (cid:107) · (cid:107) 1 (in (P1))
[Table]
and derived a lower-bound on the minimax risk of the DL problem.
However, these works do not provide any algorithms for DL.
Motivated from these theoretical advances, Spielman et al.
(2012) proposed an algorithm for the under-complete case n ≥ m that works up-to a sparsity of k
= O ( √ n ) .
Later, Agarwal et al.
(2014) and Arora et al.
(2014) proposed clustering-based provable algorithms for the overcomplete setting, motivated from MOD (Engan et al., 1999) and k-SVD (Aharon et al., 2006), respectively.
Here, in addition to requiring stringent conditions on dictionary initialization, Agarwal et al.
(2014) alternates between solving a quadratic program for coefﬁcients and an MOD-like (Engan et al., 1999) update for the dictionary, which is too expensive in practice.
Recently, a DL algorithm that works for almost linear sparsity was proposed by Barak et al.
(2015); however, as shown in Table 1, this algorithm may result in exponential running time.
Finally, Arora et al. (2015) proposed a provable online DL algorithm, which provided improvements on initialization, sparsity, and sample complexity, and is closely related to our work.
We summarize the relevant provable DL techniques in Table 1.
The algorithms discussed above implicitly assume that the coefﬁcients can be recovered, after dictionary recovery, via some sparse approximation technique.
However, as alluded to earlier, the guarantees for coefﬁcient recovery – when the dictionary is known approximately – may be limited to some (cid:96) 2 norm bounds (Donoho et al., 2006).
This means that, the resulting coefﬁcient estimates may not even be sparse.
Therefore, for practical applications, there is a need for efﬁcient online algorithms with guarantees, which serves as the primary motivation for our work.
Notation.
Given an integer n , we denote [ n ] = { 1 , 2 , . . . , n } .
The bold upper-case and lowercase letters are used to denote matrices M and vectors v , respectively.
M i , M ( i, :) , M ij , and v i (and v ( i ) ) denote the i -th column, i -th row, ( i, j ) element of a matrix, and i -th element of a vector, respectively.
The superscript ( · ) ( n ) denotes the n -th iterate, while the subscript ( · ) ( n ) is reserved for the n -th data sample.
Given a matrix M , we use (cid:107) M (cid:107) and (cid:107) M (cid:107) F as the spectral norm and Frobenius norm.
Given a vector v , we use (cid:107) v (cid:107) , (cid:107) v (cid:107) 0 , and (cid:107) v (cid:107) 1 to denote the (cid:96) 2 norm, (cid:96)
(number of nonzero entries), and (cid:96) 1 norm, respectively.
We also use standard notations O ( · ) , Ω( · ) ( (cid:101) O ( · ) , (cid:101) Ω( · ) ) to indicate the asymptotic behavior (ignoring logarithmic factors).
Further, we use g ( n ) =
O ∗ ( f ( n )) to indicate that g ( n ) ≤ Lf ( n ) for a small enough constant L , which is independent of n .
We use c ( · ) for constants parameterized by the quantities in ( · ) .
T τ ( z ) :
= z · 1 | z |≥ τ denotes the hardthresholding operator, where “ 1 ” is the indicator function.
We use supp( · ) for the support (the set of non-zero elements) and sign( · ) for the element-wise sign.
A LGORITHM
We now detail the speciﬁcs of our algorithm – NOODL, outlined in Algorithm 1.
NOODL recovers both the dictionary and the coefﬁcients exactly given an appropriate initial estimate A (0) of the dictionary.
Speciﬁcally, it requires A (0) to be ( (cid:15) 0 , 2) -close to A ∗ for (cid:15)
= O ∗ (1 / log( n )) , where ( (cid:15), κ ) -closeness is deﬁned as follows.
This implies that, the initial dictionary estimate needs to be column-wise, and in spectral norm sense, close to A ∗ , which can be achieved via certain initialization algorithms, such as those presented in Arora et al.
Deﬁnition 1 ( ( (cid:15), κ ) -closeness) .
A dictionary A is ( (cid:15), κ ) -close to A ∗ if (cid:107) A − A ∗ (cid:107) ≤ κ (cid:107)
A ∗ (cid:107) , and if there is a permutation π : [ m ] → [ m ] and a collection of signs σ : [ m ] → {± 1 } such that
(cid:107) σ ( i )
A π ( i ) −
A ∗ i (cid:107) ≤ (cid:15)
∀ i ∈
m
Initialize:
)( j ) =
T C/
A ( t ) (cid:62) y
( j ) )
Update: x ( r +1)
( j ) =
[Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation]
(cid:62) (
[Equation]
[Equation]
( r
[Equation]
[Equation]
( j ) ))
(4) end
Form empirical gradient estimate: (cid:98) g ( t ) =
1 p (cid:80) pj =
( A ( t ) (cid:98) x ( t ) ( j ) − y
( j ) )sign( (cid:98) x
( t ) ( j ) ) (cid:62) (5)
Take a gradient descent step: A ( t +1) =
A ( t ) − η A
[Equation]
Normalize: A ( t +1)
[Equation]
end
Due to the streaming nature of the incoming data, NOODL takes a mini-batch of p data samples at the t -th iteration of the algorithm, as shown in Algorithm 1.
It then proceeds by alternating between two update stages: coefﬁcient estimation (“Predict”) and dictionary update (“Learn”) as follows.
Predict Stage : For a general data sample
y
=
A ∗ x ∗ , the algorithm begins by forming an initial coefﬁcient estimate x (0) based on a hard thresholding (HT) step as shown in (3), where T τ ( z ) :
= z · 1 | z |≥ τ for a vector z .
Given this initial estimate x (0) , the algorithm iterates over R = Ω(log(1 /δ R ))
IHT-based steps (4) to achieve a target tolerance of δ R , such that (1 − η x ) R ≤ δ R .
Here, η ( r ) x is the learning rate, and τ ( r ) is the threshold at the r -th iterate of the IHT.
In practice, these can be ﬁxed to some constants for all iterations; see A.6 for details.
Finally at the end of this stage, we have estimate (cid:98) x ( t ) :
= x ( R ) of x ∗ .
Learn Stage: Using this estimate of the coefﬁcients, we update the dictionary at t -th iteration A ( t ) by an approximate gradient descent step (6), using the empirical gradient estimate (5) and the learning rate η A =
Θ( m/k ) ; see also A.5 .
Finally, we normalize the columns of the dictionary and continue to the next batch.
The running time of each step t of NOODL is therefore O ( mnp log(1 /δ R )) .
For a target tolerance of (cid:15) T and δ T , such that (cid:107) A ( T ) i
[Equation]
(cid:15) T , ∀ i ∈ [ m ] and | (cid:98) x ( T )
i − x ∗
i | ≤ δ T we choose T =
max(Ω(log(1 /(cid:15) T )) , Ω(log( √ k/δ T ))) .
NOODL uses an initial HT step and an approximate gradient descent-based strategy as in Arora et al.
Following which, our IHT-based coefﬁcient update step yields an estimate of the coefﬁcients at each iteration of the online algorithm.
Coupled with the guaranteed progress made on the dictionary, this also removes the bias in dictionary estimation.
Further, the simultaneous recovery of both factors also avoids an often expensive post-processing step for recovery of the coefﬁcients.
M AIN R ESULT
We start by introducing a few important deﬁnitions.
First, as discussed in the previous section we require that the initial estimate A (0) of the dictionary is ( (cid:15) 0 , 2) -close to A ∗ .
In fact, we require this closeness property to hold at each subsequent iteration t , which is a key ingredient in our analysis.
This initialization achieves two goals.
First, the (cid:107) σ ( i )
A π ( i ) −
A ∗ i (cid:107) ≤ (cid:15)
condition ensures that the signed-support of the coefﬁcients are recovered correctly (with high probability) by the hard thresholding-based coefﬁcient initialization step, where signed-support is deﬁned as follows.
Deﬁnition 2.
The signed-support of a vector x is deﬁned as sign( x ) · x (supp( x )) .
Next, the (cid:107) A −
A ∗ (cid:107) ≤ 2 (cid:107) A ∗ (cid:107) condition keeps the dictionary estimates close to A ∗ and is used in our analysis to ensure that the gradient direction (5) makes progress.
Further, in our analysis, we ensure (cid:15) t (deﬁned as (cid:107) A ( t )
i − A ∗
i (cid:107) ≤ (cid:15) t ) contracts at every iteration, and assume (cid:15) 0 , (cid:15) t
= O ∗ (1 / log( n )) .
Also, we assume that the dictionary A is ﬁxed (deterministic) and µ -incoherent, deﬁned as follows.
Deﬁnition 3.
A matrix A ∈ R n × m with unit-norm columns is µ -incoherent if for all i (cid:54) =
j the inner-product between the columns of the matrix follow |(cid:104)
A i , A j (cid:105)| ≤ µ/ √ n .
The incoherence parameter measures the degree of closeness of the dictionary elements.
Smaller values (i.e., close to 0 ) of µ are preferred, since they indicate that the dictionary elements do not resemble each other.
This helps us to effectively tell dictionary elements apart (Donoho and Huo, 2001; Cand`es and Romberg, 2007).
We assume that µ =
O (log( n ))
(Donoho and Huo, 2001).
Next, we assume that the coefﬁcients are drawn from a distribution class D deﬁned as follows.
Deﬁnition 4 (Distribution class D ) .
The coefﬁcient vector x ∗ belongs to an unknown distribution
D , where the support S = supp( x ∗ ) is at most of size k , Pr [ i ∈ S ]
=
Θ( k/m ) and Pr [ i, j ∈ S ]
= Θ( k 2 /m 2 ) .
Moreover, the distribution is normalized such that E [ x ∗ i | i ∈ S ] =
and E [ x ∗
i | i ∈
S ] = 1 , and when i ∈ S , | x ∗ i | ≥ C for some constant C ≤ 1 .
In addition, the non-zero entries are sub-Gaussian and pairwise independent conditioned on the support.
The randomness of the coefﬁcient is necessary for our ﬁnite sample analysis of the convergence.
Here, there are two sources of randomness.
The ﬁrst is the randomness of the support, where the non-zero elements are assumed to pair-wise independent.
The second is the value an element in the support takes, which is assumed to be zero mean with variance one, and bounded in magnitude.
Similar conditions are also required for support recovery of sparse coefﬁcients, even when the dictionary is known (Wainwright, 2009; Yuan et al., 2016).
Note that, although we only consider the case | x ∗ i | ≥ C for ease of discussion, analogous results may hold more generally for x ∗ i s drawn from a distribution with sufﬁciently (exponentially) small probability of taking values in [ − C, C ] .
Recall that, given the coefﬁcients, we recover the dictionary by making progress on the least squares objective (P1) (ignoring the term penalizing S ( · ) ).
Note that, our algorithm is based on ﬁnding an appropriate direction to ensure descent than the geometry of the objective.
To this end, we adopt a gradient descent-based strategy for dictionary update.
However, since the coefﬁcients are not exactly known, this results in an approximate gradient descent-based approach, where the empirical gradient estimate is formed as (5).
In our analysis, we establish the conditions under which both the empirical gradient vector (corresponding to each dictionary element) and the gradient matrix concentrate around their means.
To ensure progress at each iterate t , we show that the expected gradient vector is (Ω( k/m ) , Ω( m/k ) , 0)
-correlated with the descent direction, deﬁned as follows.
Deﬁnition 5.
A vector g ( t ) is ( ρ − , ρ + , ζ t ) -correlated with a vector z ∗ if
[Equation]
[Table]
This can be viewed as a local descent condition which leads to the true dictionary columns; see also
Cand`es et al.
(2015), Chen and Wainwright (2015) and Arora et al.
In convex optimization literature, this condition is implied by the 2 ρ − -strong convexity, and 1 / 2 ρ
+ -smoothness of the objective.
We show that for NOODL, ζ t = 0 , which facilitates linear convergence to A ∗ without incurring any bias.
Overall our speciﬁc model assumptions for the analysis can be formalized as:
A.1 A ∗ is µ -incoherent (Def. 3), where µ = O (log( n )) , (cid:107) A ∗ (cid:107) =
O ( (cid:112) m/n ) and m =
O ( n ) ; A.
The coefﬁcients are drawn from the distribution class D , as per Def. 4; A.3 The sparsity k satisﬁes k = O ( √ n/µ log( n )) ;
A.4 A (0) is ( (cid:15) 0 , 2) -close to A ∗ as per Def. 1, and (cid:15)
= O ∗ (1 / log( n )) ; A.5 The step-size for dictionary update satisﬁes η A = Θ( m/k ) ; A.6 The step-size and threshold for coefﬁcient estimation satisﬁes η ( r ) x < c 1 ( (cid:15) t , µ, n, k )
= (cid:101) Ω( k/ √ n ) < 1 and τ ( r ) =
c 2 ( (cid:15) t , µ, k, n ) = (cid:101) Ω( k 2 /n )
for small constants c 1 and c 2 .
We are now ready to state our main result.
A summary of the notation followed by a details of the analysis is provided in Appendix A and Appendix B, respectively.
Theorem 1 (Main Result) .
Suppose that assumptions A.1 - A.6 hold, and Algorithm 1 is provided with p
= (cid:101) Ω( mk 2 )
new samples generated according to model (1) at each iteration t .
Then for some 0 < ω < 1 / 2 , the estimate A ( t ) at ( t ) -th iteration satisﬁes
[Equation]
−
[Equation]
for all t = 1 , 2 , . . . .
Furthermore, given R = log( n ) , with probability at least (1 − δ alg ) for some small constant δ alg , the coefﬁcient estimate (cid:98) x ( t ) i at t -th iteration has the correct signed-support and satisﬁes
[Equation]
all i ∈
[Equation]
Our main result establishes that when the model satisﬁes A.1 ∼ A.3 , the errors corresponding to the dictionary and coefﬁcients geometrically decrease to the true model parameters, given appropriate dictionary initialization and learning parameters (step sizes and threshold); see A.4 ∼ A.6 .
In other words, to attain a target tolerance of (cid:15) T and δ T , where (cid:107) A ( T )
i
[Equation]
we require T =
max(Ω(log(1 /(cid:15) T )) , Ω(log( √ k/δ T )))
outer iterations and R =
Ω(log(1 /δ R ))
IHT steps per outer iteration.
Here, δ R ≥ (1 − η x ) R is the target decay tolerance for the IHT steps.
An appropriate number of IHT steps, R , remove the dependence of ﬁnal coefﬁcient error (per outer iteration) on the initial x (0) .
In Arora et al.
(2015), this dependence in fact results in an irreducible error, which is the source of bias in dictionary estimation.
As a result, since (for NOODL)
the error in the coefﬁcients only depends on the error in the dictionary, it can be made arbitrarily small, at a geometric rate, by the choice of (cid:15) T , δ T , and δ R .
Also, note that, NOODL can tolerate i.i.d.
noise, as long as the noise variance is controlled to enable the concentration results to hold; we consider the noiseless case here for ease of discussion, which is already highly involved.
Intuitively, Theorem 1 highlights the symbiotic relationship between the two factors.
It shows that, to make progress on one, it is imperative to make progress on the other.
The primary condition that allows us to make progress on both factors is the signed-support recovery (Def. 2).
However, the introduction of IHT step adds complexity in the analysis of both the dictionary and coefﬁcients.
To analyze the coefﬁcients, in addition to deriving conditions on the parameters to preserve the correct signed-support, we analyze the recursive IHT update step, and decompose the noise term into a component that depends on the error in the dictionary, and the other that depends on the initial coefﬁcient estimate.
For the dictionary update, we analyze the interactions between elements of the coefﬁcient vector (introduces by the IHT-based update step) and show that the gradient vector for the dictionary update is (Ω( k/m ) , Ω( m/k ) , 0) -correlated with the descent direction.
In the end, this leads to exact recovery of the coefﬁcients and removal of bias in the dictionary estimation.
Note that our analysis pipeline is standard for the convergence analysis for iterative algorithms.
However, the introduction of the IHT-based strategy for coefﬁcient update makes the analysis highly involved as compared to existing results, e.g., the simple HT-based coefﬁcient estimate in Arora et al.
NOODL has an overall running time of O ( mnp log(1 /δ R ) max(log(1 /(cid:15) T ) , log( √ k/δ T )) to achieve target tolerances (cid:15) T and δ T , with a total sample complexity of p · T = (cid:101) Ω( mk 2 ) .
Thus to remove bias, the IHT-based coefﬁcient update introduces a factor of log(1 /δ R ) in the computational complexity as compared to Arora et al.
(2015) (has a total sample complexity of p · T = (cid:101) Ω( mk ) ), and also does not have the exponential running time and sample complexity as Barak et al.
; see Table 1.
N EURAL IMPLEMENTATION OF NOODL
The neural plausibility of our algorithm implies that it can be implemented as a neural network.
This is because, NOODL employs simple linear and non-linear operations (such as inner-product and hard-thresholding) and the coefﬁcient updates are separable across data samples, as shown in (4) of Algorithm 1.
To this end, we present a neural implementation of our algorithm in Fig.
1, which showcases the applicability of NOODL in large-scale distributed learning tasks, motivated from the implementations described in (Olshausen and Field, 1997) and (Arora et al., 2015).
The neural architecture shown in Fig.
1(a) has three layers – input layer, weighted residual evaluation layer, and the output layer.
The input to the network is a data and step-size pair ( y ( j ) , η x ) to each input node.
Given an input, the second layer evaluates the weighted residuals as shown in Fig.
Finally, the output layer neurons evaluate the IHT iterates x ( r +1)
( j ) (4).
We illustrate the operation of this architecture using the timing diagram in Fig. 1(b).
The main stages of operation are as follows.
[Figure] [Figure]
[Figure]
Initial Hard Thresholding Phase :
The coefﬁcients initialized to zero, and an input ( y ( j ) , 1) is provided to the input layer at a time instant (cid:96) = 0 , which communicates these to the second layer.
Therefore, the residual at the output of the weighted residual evaluation layer evaluates to y ( j ) at (cid:96) = 1 .
Next, at (cid:96) = 2 , this residual is communicated to the output layer, which results in evaluation of the initialization x (0)( j ) as per (3).
This iterate is communicated to the second layer for the next residual evaluation.
Also, at this time, the input layer is injected with ( y ( j ) , η x ) to set the step size parameter η x for the IHT phase, as shown in Fig. 1(b).
Iterative Hard Thresholding (IHT) Phase :
Beginning (cid:96) = 3 , the timing sequence enters the
IHT phase.
Here, the output layer neurons communicate the iterates x ( r +1)
( j ) to the second layer for evaluation of subsequent iterates as shown in Fig. 1(b).
The process then continues till the time instance (cid:96) = 2 R + 1 , for R = Ω(log(1 /δ R )) to generate the ﬁnal coefﬁcient estimate (cid:98) x ( t ) ( j ) :
= x ( R ) ( j ) for the current batch of data.
At this time, the input layer is again injected with ( y ( j ) , 1) to prepare the network for residual sharing and gradient evaluation for dictionary update.
Dictionary Update Phase: The procedure now enters the dictionary update phase, denoted as “Hebbian Learning” in the timing sequence.
In this phase, each output layer neuron communicates the ﬁnal coefﬁcient estimate (cid:98) x ( t ) ( j ) =
x ( R ) ( j ) to the second layer, which evaluates the residual for one last time (with η x = 1 ), and shares it across all second layer neurons (“Hebbian learning”).
This allows each second layer neuron to evaluate the empirical gradient estimate (5), which is used to update the current dictionary estimate (stored as weights) via an approximate gradient descent step.
This completes one outer iteration of Algorithm 1, and the process continues for T iterations to achieve target tolerances (cid:15) T and δ T , with each step receiving a new mini-batch of data.
E XPERIMENTS
We now analyze the convergence properties and sample complexity of NOODL via experimental evaluations.
The experimental data generation set-up, additional results, including analysis of computational time, are shown in Appendix
E. The associated code is made available at https://github.com/srambhatla/NOODL .
5. C ONVERGENCE A NALYSIS
We compare the performance of our algorithm NOODL with the current state-of-the-art alternating optimization-based online algorithms presented in Arora et al.
and the popular algorithm presented in Mairal et al.
(2009) (denoted as Mairal ‘ 09 ).
First of these, Arora15(‘‘biased’’) , is a simple neurally plausible method which incurs a bias and has a sample complexity of Ω( mk ) .
The other, referred to as Arora15(‘‘unbiased’’) , incurs no bias as per Arora et al.
, but the sample complexity results were not established.
Discussion: Fig.
2 panels (a-i), (b-i), (c-i), and (d-i) show the performance of the aforementioned methods for k = 10 , 20 , 50 , and 100 , respectively.
Here, for all experiments we set η x = 0 .
2 and τ = 0 .
We terminate NOODL when the error in dictionary is less than 10 − 10 .
Also, for coefﬁcient update, we terminate when change in the iterates is below 10 − 12 .
For k = 10 , 20 and k = 50 , we note that Arora15(‘‘biased’’) and Arora15(‘‘unbiased’’) incur signiﬁcant bias,
[Figure]
i
[Figure]
n a
[Figure]
y
[Figure]
ec ov er y
[Figure]
2 p=m0.2 0.
S u cc e ss
P r obab ili t y m=nm=2nm=4n (a-ii) (b-ii) (c-ii) (d-ii) (e-ii) Coefﬁcients while NOODL converges to A ∗ linearly .
NOODL also converges for signiﬁcantly higher choices of sparsity k , i e., for k = 100 as shown in panel (d), beyond k
= O ( √ n ) , indicating a potential for improving this bound.
Further, we observe that Mairal ‘ 09 exhibits signiﬁcantly slow convergence as compared to NOODL.
Also, in panels (a-ii), (b-ii), (c-ii) and (d-ii) we show the corresponding performance of NOODL in terms of the error in the overall ﬁt ( (cid:107) Y − AX (cid:107) F / (cid:107) Y (cid:107) F ), and the error in the coefﬁcients and the dictionary, in terms of relative Frobenius error metric discussed above.
We observe that the error in dictionary and coefﬁcients drops linearly as indicated by our main result.
5. P HASE TRANSITIONS
Fig.
2 panels (e-i) and (e-ii), shows the phase transition in number of samples with respect to the size of the dictionary m .
We observe a sharp phase transition at pm = 1 for the dictionary, and at pm = 0 .
75 for the coefﬁcients.
This phenomenon is similar one observed by Agarwal et al.
(however, theoretically they required p =
O ( m 2 ) ).
Here, we conﬁrm number of samples required by NOODL are linearly dependent on the dictionary elements m .
F UTURE W ORK
We consider the online DL setting in this work.
We note that, empirically NOODL works for the batch setting also.
However, analysis for this case will require more sophisticated concentration results, which can address the resulting dependence between iterations of the algorithm.
In addition, our experiments indicate that NOODL works beyond the sparsity ranges prescribed by our theoretical results.
Arguably, the bounds on sparsity can potentially be improved by moving away from the incoherence-based analysis.
We also note that in our experiments, NOODL converges even when initialized outside the prescribed initialization region, albeit it achieves the linear rate once it satisﬁes the closeness condition A.4 .
These potential directions may signiﬁcantly impact the analysis and development of provable algorithms for other factorization problems as well.
We leave these research directions, and a precise analysis under the noisy setting, for future explorations.
C ONCLUSIONS
We present NOODL, to the best of our knowledge, the ﬁrst neurally plausible provable online algorithm for exact recovery of both factors of the dictionary learning (DL) model.
NOODL alternates between: (a) an iterative hard thresholding (IHT)
-based step for coefﬁcient recovery, and (b) a gradient descent-based update for the dictionary, resulting in a simple and scalable algorithm, suitable for large-scale distributed implementations.
We show that once initialized appropriately, the sequence of estimates produced by NOODL converge linearly to the true dictionary and coefﬁcients without incurring any bias in the estimation.
Complementary to our theoretical and numerical results, we also design an implementation of NOODL in a neural architecture for use in practical applications.
In essence the analysis of this inherently non-convex problem impacts other matrix and tensor factorization tasks arising in signal processing, collaborative ﬁltering and machine learning.
A CKNOWLEDGMENT
The authors would like to graciously acknowledge support from DARPA Young Faculty Award,
Grant No. N66001-14-1-4047.
A S UMMARY OF N OTATION
We summarizes the deﬁnitions of some frequently used symbols in our analysis in Table 2.
In addition, we use D ( v ) as a diagonal matrix with elements of a vector v on the diagonal.
Given a matrix M , we use M − i to denote a resulting matrix without i -th column.
Also note that, since we show that (cid:107) A ( t ) i − A ∗ i (cid:107) ≤ (cid:15) t contracts in every step, therefore we ﬁx (cid:15) t , (cid:15)
= O ∗ (1 / log( n )) in our analysis.
B P ROOF OF T HEOREM
We now prove our main result.
The detailed proofs of intermediate lemmas and claims are organized in Appendix C and Appendix D, respectively.
Furthermore, the standard concentration results are stated in Appendix F for completeness.
Also, see Table 3 for a map of dependence between the results.
O VERVIEW
Given an ( (cid:15) 0 , 2) -close estimate of the dictionary, the main property that allows us to make progress on the dictionary is the recovery of the correct sign and support of the coefﬁcients.
Therefore, we ﬁrst show that the initial coefﬁcient estimate (3) recovers the correct signed-support in Step I.A.
Now, the IHT-based coefﬁcient update step also needs to preserve the correct signed-support.
This is to ensure that the approximate gradient descent-based update for the dictionary makes progress.
Therefore, in Step I.B, we derive the conditions under which the signed-support recovery condition is preserved by the IHT update.
To get a handle on the coefﬁcients, in Step II.A, we derive an upper-bound on the error incurred by each non-zero element of the estimated coefﬁcient vector, i.e., | (cid:98) x
i − x ∗ i | for i ∈ S for a general coefﬁcient vector x ∗ , and show that this error only depends on (cid:15) t (the column-wise error in the dictionary) given enough IHT iterations R as per the chosen decay parameter δ R .
In addition, for analysis of the dictionary update, we develop an expression for the estimated coefﬁcient vector in
Step II.B.
We then use the coefﬁcient estimate to show that the gradient vector satisﬁes the local descent condition (Def. 5).
This ensures that the gradient makes progress after taking the gradient descent-based step (6).
To begin, we ﬁrst develop an expression for the expected gradient vector (corresponding to each dictionary element) in Step III.A. Here, we use the closeness property Def 1 of the dictionary estimate.
Further, since we use an empirical estimate, we show that the empirical gradient vector concentrates around its mean in Step III.B. Now using Lemma 15, we have that descent along this direction makes progress.
Next in Step IV.A and Step IV.B, we show that the updated dictionary estimate maintains the closeness property Def 1.
This sets the stage for the next dictionary update iteration.
As a result, our main result establishes the conditions under which any t -th iteration succeeds.
Our main result is as follows.
Theorem 1 (Main Result)
Suppose that assumptions A.1 - A.6 hold, and Algorithm 1 is provided with p
= (cid:101) Ω( mk 2 )
new samples generated according to model (1) at each iteration t .
Then for some 0 < ω < 1 / 2 , the estimate A ( t ) at ( t ) -th iteration satisﬁes
[Equation]
Furthermore, given R = log( n ) , with probability at least (1 − δ alg ) for some small constant δ alg , the coefﬁcient estimate (cid:98) x ( t ) i at t -th iteration has the correct signed-support and satisﬁes ( (cid:98)
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
(cid:107) A (0)
i −
[Equation]
) , for all i ∈ supp( x ∗ ) .
Here, δ ( t ) alg =
δ ( t ) T + δ ( t ) β
+ δ HW + δ ( t ) g
i + δ ( t ) g , δ ( t ) T = 2 m exp( − C 2 / O ∗ ( (cid:15) 2 t )) , δ ( t ) β
= 2 k exp( − 1 / O ( (cid:15) t )) , δ ( t )HW = exp( − 1 / O
( (cid:15) t )) , δ ( t ) g i =
exp( − Ω( k )) , and δ ( t ) g
= ( n + m ) exp( − Ω( m (cid:112) log( n )) .
S TEP I: C OEFFICIENTS HAVE THE CORRECT SIGNED - SUPPORT
As a ﬁrst step
[Table]
we ensure that our coefﬁcient estimate has the correct signed-support (Def. 2).
To this end
[Table]
we ﬁrst show that the initialization has the correct signed-support
[Table]
and then show that the iterative hard-thresholding (IHT)-based update step preserves the correct signed-support for a suitable choice of parameters
[Table] [Table]
δ T | (cid:98) x
( T ) i − x ∗ i | ≤ δ T ∀ i ∈
[Table]
∗ )
[Table]
( (cid:96) )
i
[Table]
( (cid:96) )
i :=
| x ∗ i − x ( (cid:96) )
i |
[Table]
∗
[Table]
ProbabilitiesSymbolDeﬁnition
Symbol
Deﬁnition q i q i =
Pr [ i ∈ S ]
= Θ( km ) q
i,j q i,j =
Pr [ i, j ∈ S ]
=
Θ( k 2 m 2 ) p i p i
=
E [ x ∗ i sign( x ∗ i ) | x ∗ i (cid:54) = 0] δ ( t ) T δ ( t ) T = 2 m exp( − C 2 / O ∗ ( (cid:15) 2 t ))
δ ( t ) β δ ( t ) β
= 2 k exp( − 1 / O ( (cid:15) t ))
δ ( t )HW δ ( t )HW = exp( − 1 / O
( (cid:15) t ))
δ ( t ) g i δ ( t ) g i =
exp( − Ω( k ))
δ ( t ) g δ ( t ) g
=
( n + m ) exp(
− Ω( m (cid:112) log( n ))
Other terms
Symbol
Deﬁnition ξ ( r +1)
j ξ ( r +1)
j := (cid:80)
[Equation]
x
[Equation]
i β ( t ) j β ( t
[Equation]
(cid:80)
[Equation]
t
[Table]
t β = O ( √ k(cid:15) t ) is an upper-bound on β ( t ) j with probability at least (1 − δ ( t ) β )
[Equation]
r
[Equation] [Equation]
[Table]
[Equation]
below
[Equation]
[Table] [Table]
[Table] [Table]
• Step I.A:
Showing that the initial coefﬁcient estimate has the correct signed-support– Given an ( (cid:15) 0 , 2) -close estimate A (0) of A ∗ , we ﬁrst show that for a general sample y the initialization step (3) recovers the correct signed-support with probability at least (1 − δ ( t ) T ) , where δ ( t ) T = 2 m exp( − C 2 O ∗ ( (cid:15) 2 t ) ) .
This is encapsulated by the following lemma.
Lemma 1 ( Signed-support recovery by coefﬁcient initialization step ) .
Suppose A ( t ) is (cid:15) t -close to A ∗ .
Then, if µ = O (log( n )) , k = √ n/µ log( n ) , and (cid:15) t
= O ∗ (1 / (cid:112) log( m )) , with probability at least (1 − δ ( t ) T ) for each random sample y =
A ∗ x ∗ : sign( T C/ 2 (( A ( t ) ) (cid:62) y )
= sign( x ∗ ) ,
where δ ( t ) T = 2 m exp( − C 2 O ∗ ( (cid:15) 2 t ) ) .
Note that this result only requires the dictionary to be column wise close to the true dictionary and works for less stringent conditions on
the
initial dictionary
estimate,
i
.e.
requires (cid:15)
t =
O ∗ (1 / (cid:112) log( m )) instead of (cid:15) t = O ∗ (1 / log( m )) ;
see
also
(Arora et al., 2015)
• Step I.B: The iterative IHT-type updates preserve the correct signed support– Next, we show that the IHT-type coefﬁcient update step (4) preserves the correct signed-support for an appropriate choice of step-size parameter η ( r ) x and threshold τ ( r ) .
The choice of these parameters arises from the analysis of the IHT-based update step.
Speciﬁcally, we show that at each iterate
r , the step-size η ( r ) x should be chosen to ensure that the component corresponding to the true coefﬁcient value is greater than the “interference” introduced by other non-zero coefﬁcient elements.
Then, if the threshold is chosen to reject this “noise”, each iteration of the IHT-based update step preserves the correct signed-support.
Lemma 2 ( IHT update step preserves the correct signed-support ) .
Suppose A ( t ) is (cid:15) tclose to A ∗ , µ = O (log( n )) , k = √ n/µ log( n ) , and (cid:15) t =
O ∗ (1 / log( m ))
Then, with probability at least (1 − δ ( t ) β − δ ( t ) T ) , each iterate of the IHT-based coefﬁcient update step shown in (4) has the correct signed-support, if for a constant c ( r ) 1 ( (cid:15) t , µ, k, n ) = (cid:101) Ω( k 2 /n ) , the step size is chosen as η ( r ) x ≤ c ( r ) 1 , and the threshold τ ( r ) is chosen as
[Equation]
for some constants c 1 and c 2 .
Here, t β = O ( √ k(cid:15) t ) , δ ( t ) T = 2 m exp( − C
[Equation]
∗ ( (cid:15) 2 t ) ) ,and δ ( t ) β = 2 k exp( − 1 O ( (cid:15) t ) ) .
Note that, although we have a dependence on the iterate r in choice of η ( r ) x and τ ( r ) , these can be set to some constants independent of r .
In practice, this dependence allows for greater ﬂexibility in the choice of these parameters.
S TEP II: A NALYZING THE COEFFICIENT ESTIMATE
We now derive an
upper-bound on the error incurred by each non-zero coefﬁcient element.
Further, we derive an expression for the coefﬁcient estimate at the t -th round of the online algorithm (cid:98) x ( t ) :
= x ( R ) ; we use (cid:98) x instead of (cid:98) x
( t ) for simplicity.
• Step II.A: Derive a bound on the error incurred by the coefﬁcient estimate– Since Lemma 2 ensures that (cid:98) x has the correct signed-support, we now focus on the error incurred by each coefﬁcient element on the support by analyzing (cid:98) x .
To this end, we carefully analyze the effect of the recursive update (4), to decompose the error incurred by each element on the support into two components – one that depends on the initial coefﬁcient estimate x (0) and other that depends on the error in the dictionary.
We show that the effect of the component that depends on the initial coefﬁcient estimate diminishes by a factor of (1 − η x + η x µ t √ n ) at each iteration r .
Therefore, for a decay parameter δ R , we can choose the number of IHT iterations R , to make this component arbitrarily small.
Therefore, the error in the coefﬁcients only depends on the per column error in the dictionary, formalized by the following result.
Lemma 3 ( Upper-bound on the error in coefﬁcient estimation ) .
With probability at least (1 − δ ( t ) β − δ ( t ) T ) the error incurred by each element i 1 ∈ supp( x ∗ ) of the coefﬁcient estimate is upper-bounded as
[Equation]
where t
[Equation]
= O
[Equation] [Equation]
( −
[Equation] [Equation]
t ) ) , and µ t is the incoherence between the columns of A ( t )
; see Claim 1.
This result allows us to show that if the column-wise error in the dictionary decreases at each iteration t , then the corresponding estimates of the coefﬁcients also improve.
• Step II.B:
Developing an expression for the coefﬁcient estimate– Next,
we derive
the expression for the coefﬁcient estimate in the
following lemma.
This
expression is used to analyze the dictionary update.
Lemma 4 ( Expression for the coefﬁcient estimate at the end of R -th IHT iteration ) .
With probability at least (1 − δ ( t ) T − δ ( t ) β ) the i 1 -th element of the coefﬁcient estimate, for each i 1 ∈ supp( x ∗ ) , is given by
[Equation]
Here, ϑ ( R ) i 1 is | ϑ ( R )
i 1 |
=
O ( t β ) , where t β = O ( √ k(cid:15) t ) .
Further, λ ( t ) i 1 = |(cid:104) A ( t )
i 1 − A ∗
i 1 , A ∗ i 1 (cid:105)| ≤ (cid:15) 2 t 2 , δ ( t ) T = 2 m exp( − C 2 O ∗ ( (cid:15) 2 t ) ) and δ ( t ) β
= 2 k exp( − 1 O ( (cid:15) t ) ) .
We again observe that the error in the coefﬁcient estimate depends on the error in the dictionary via λ ( t )
i 1 and ϑ ( R )
i 1 .
S TEP III: A NALYZING THE GRADIENT FOR DICTIONARY UPDATE
Given the coefﬁcient estimate we now show that the choice of the gradient as shown in (5) makes progress at each step.
To this end, we analyze the gradient vector corresponding to each dictionary element to see if it satisﬁes the local descent condition of Def. 5.
Our analysis of the gradient is motivated from Arora et al.
However, as opposed to the simple HT-based coefﬁcient update step used by Arora et al.
, our IHT-based coefﬁcient estimate adds to signiﬁcant overhead in terms of analysis.
Notwithstanding the complexity of the analysis, we show that this allows us to remove the bias in the gradient estimate.
To this end, we ﬁrst develop an expression for each expected gradient vector, show that the empirical gradient estimate concentrates around its mean, and ﬁnally show that the empirical gradient vector is (Ω( k/m ) , Ω( m/k ) , 0)
-correlated with the descent direction, i.e. has no bias.
• Step III.A: Develop an expression for the expected gradient vector corresponding to each dictionary element– The expression for the expected gradient vector g ( t ) j corresponding to j -th dictionary element is given by the following lemma.
Lemma 5 ( Expression for the expected gradient vector ) .
Suppose that A ( t ) is ( (cid:15) t , 2) - near to A ∗ .
Then, the dictionary update step in Algorithm 1 amounts to the following for the j -th dictionary element
[Equation]
t +1)
[Equation]
=
A
[Equation]
t )
[Equation]
g
[Equation]
t
[Equation]
, where g ( t ) j is given by
[Equation]
= |
[Equation]
t
[Equation]
− A ∗ j , A ∗ j (cid:105)| , and ∆ ( t ) j :=
E [ A ( t ) S
ϑ ( R ) S sign( x ∗ j )] , where (cid:107) ∆ ( t ) j (cid:107) =
O (
[Equation]
mq i,j p j (cid:15) t (cid:107) A ( t ) (cid:107) ) .
• Step III.B: Show that the empirical gradient vector concentrates around its expectation– Since we only have access to the empirical gradient vectors, we show that these concentrate around their expected value via the following lemma.
Lemma 6 ( Concentration of the empirical gradient vector ) .
Given p = (cid:101) Ω( mk 2 ) samples, the empirical gradient vector estimate corresponding to the i -th dictionary element,
[Equation]
g ( t ) i concentrates around its expectation, i
e.,
[Equation]
with probability at least (1 − δ ( t ) g
i − δ ( t ) β − δ ( t ) T − δ ( t )HW ) , where δ ( t ) g i =
exp( − Ω( k )) .
• Step III.C:
Show that the empirical gradient vector is correlated with the descent direction– Next, in the following lemma we show that the empirical gradient vector (cid:98) g ( t ) j is correlated with the descent direction.
This is the main result which enables the progress in the dictionary (and coefﬁcients) at each iteration t .
Lemma 7 ( Empirical gradient vector is correlated with the descent direction ) .
Suppose A ( t ) is ( (cid:15) t , 2) -near to A ∗ , k
= O ( √ n ) and η A
= O ( m/k ) .
Then, with probability at least (1 − δ ( t ) T − δ ( t ) β − δ ( t )HW − δ ( t ) g i ) the empirical gradient vector (cid:98) g ( t )
j is (Ω( k/m ) , Ω( m/k ) , 0) -correlated with ( A ( t ) j − A ∗ j ) , and for any t ∈ [ T ] ,
[Equation]
This result ensures for at any t ∈ [ T ] , the gradient descent-based updates made via (5) gets the columns of the dictionary estimate closer to the true dictionary, i.e., (cid:15)
t +1 ≤ (cid:15) t .
Moreover, this step requires closeness between the dictionary estimate A ( t ) and A ∗ , in the spectral norm-sense, as per Def 1.
S TEP IV: S HOW THAT THE DICTIONARY MAINTAINS THE CLOSENESS PROPERTY
As discussed above, the closeness property (Def 1) is crucial to show that the gradient vector is correlated with the descent direction.
Therefore, we now ensure that the updated dictionary A ( t +1) maintains this closeness property.
Lemma 7 already ensures that (cid:15) t +1 ≤ (cid:15) t .
As a result, we show that A ( t +1) maintains closeness in the spectral norm-sense as required by our algorithm, i.e., that it is still ( (cid:15) t +1 , 2) -close to the true dictionary.
Also, since we use the gradient matrix in this analysis, we show that the empirical gradient matrix concentrates around its mean.
• Step IV.A:
The empirical gradient matrix concentrates around its expectation : We ﬁrst show that the empirical gradient matrix concentrates as formalized by the following lemma.
Lemma
8 ( Concentration of
the
empirical
gradient
matrix ) .
With probability at least (1 − δ ( t ) β − δ ( t ) T − δ ( t )HW − δ ( t ) g ) , (cid:107) (cid:98) g
( t ) − g ( t ) (cid:107) is upper-bounded by O ∗ ( km (cid:107)
A ∗ (cid:107) ) , where δ ( t ) g = ( n + m ) exp( − Ω( m (cid:112) log( n )) .
• Step IV.B: The “closeness” property is maintained after the updates made using the empirical gradient estimate : Next, the following lemma shows that the updated dictionary A ( t +1) maintains the closeness property.
Lemma 9 ( A ( t +1) maintains closeness ) .
Suppose A ( t ) is ( (cid:15) t , 2) near to A ∗ with (cid:15) t =
O ∗ (1 / log( n )) , and number of samples used in step t is p = (cid:101) Ω( mk 2 ) , then with probability at least (1 − δ ( t ) T − δ ( t ) β − δ ( t )HW − δ ( t ) g ) , A ( t +1) satisﬁes (cid:107)
A ( t +1)
− A ∗ (cid:107) ≤ 2 (cid:107)
A ∗ (cid:107) .
S TEP V: C OMBINE RESULTS TO SHOW THE MAIN RESULT
Proof of Theorem 1.
From Lemma 7 we have that with probability at least (1 − δ ( t ) T − δ ( t ) β − δ ( t )HW − δ ( t ) g i ) , g ( t )
j is (Ω( k/m ) , Ω( m/k ) , 0)
-correlated with A ∗ j .
Further, Lemma 9 ensures that each iterate maintains the closeness property.
Now, applying Lemma 15 we have that, for η A ≤ Θ( m/k ) , with probability at least (1 − δ ( t ) alg )
any t ∈ [ T ] satisﬁes
[Equation]
where for 0 < ω < 1 / 2 with ω = Ω( k/m ) η A .
That is, the updates converge geometrically to
A ∗ .
Further, from Lemma 3, we have that the result on the error incurred by the coefﬁcients.
Here, δ ( t ) alg
=
δ ( t ) T + δ ( t ) β
+ δ ( t )HW + δ ( t ) g
i + δ ( t ) g ) .
That is, the updates converge geometrically to A ∗ .
Further, from Lemma 3, we have that the error in the coefﬁcients only depends on the error in the dictionary, which leads us to our result on the error incurred by the coefﬁcients.
This completes the proof of our main result.
C A PPENDIX : P ROOF OF L EMMAS
We present the proofs of the Lemmas used to establish our main result.
Also, see Table 3 for a map of dependence between the results, and Appendix D for proofs of intermediate results.
[Table]
calculations
Claim Claim
Bound on the noise term in the estimation of a coefﬁcient element in the support
Claim Claim
An intermediate result for ϑ ( R ) i 1 calculations
Claim Claim
Bound on the noise term in ex- pected gradient vector estimate
Claim 6 and Claim
Claim
An intermediate result for con- centration results
Lemma 2 ,Lemma 4 and
Claim 5 Claim
Bound on variance parameter for concentration of gradient vector
Claim
Claim 10 Bound on variance parameter for concentration of gradient matrix
Lemma 2 , Lemma 4 and
Claim
Lemma
Claim
Claim
Lemma
Claim
Claim
Lemma
Claim
Lemma
Lemma
Claim Claim
Lemma
Claim Claim
Lemma
Lemma
Claim
Lemma
Theorem
Proof of Lemma 1.
Let y ∈ R n be general sample generated as y =
A ∗ x ∗ , where x ∗ ∈
R m is a sparse random vector with support S = supp( x ∗ ) distributed according to D.4 .
The initial decoding step at the t -th iteration (shown in Algorithm 1) involves evaluating the innerproduct between the estimate of the dictionary A ( t ) , and y .
The i -th element of the resulting vector can be written as
[Equation]
where w i = (cid:104)
A ( t ) i , A ∗− i x ∗− i (cid:105) .
Now, since (cid:107) A ∗ i − A ( t ) i (cid:107) 2 ≤ (cid:15) t and
[Equation]
we have
[Equation]
Therefore, the term
[Equation]
Now, we focus on the w i and show that it is small.
By the deﬁnition of w i we have
[Equation]
Here, since var ( x ∗ (cid:96) )
= 1 , w i is a zero-mean random variable with variance var ( w i )
[Equation]
Now, each term in this sum can be bounded as,
[Equation]
Next, (cid:80)
[Equation]
i ,
[Equation]
2 can be upper-bounded as
[Equation]
Therefore
[Equation]
we have the following as per our assumptions on µ and k ,
[Equation]
using Gershgorin Circle Theorem (Gershgorin, 1931).
Therefore
[Equation]
we have
[Equation]
Finally, we have that
[Equation]
Now, we apply the Chernoff bound for sub-Gaussian random variables w i (shown in Lemma 12) to conclude that
[Equation]
Further, w i corresponding to each m should follow this bound, applying union bound we conclude that
[Equation]
Proof of Lemma 2.
Consider the ( r + 1) -th iterate x ( r +1) for the t -th dictionary iterate, where (cid:107)
A ( t ) i − A ∗
i (cid:107) ≤ (cid:15) t for all i ∈ [1 , m ] evaluated as the following by the update step described in
Algorithm 1,
[Equation]
where η (1) x < 1 is the learning rate or the step-size parameter.
Now
[Equation]
using Lemma 1 we know that x (0) (3) has the correct signed-support with probability at least (1 − δ ( t ) T ) .
Further, since A ( t )
[Equation]
A ∗ can be written as
[Equation]
we can write the ( r + 1) -th iterate of the coefﬁcient update step using (7) as
[Equation]
Further
[Equation]
the j -th entry of this vector is given by
[Equation]
We now develop an expression for the j -th element of each of the term in (8) as follows.
First, we can write the ﬁrst term as
[Equation]
Next, the second term in (8) can be expressed as η
[Equation]
Finally
[Equation]
we have the following expression for the third term,
[Equation]
Now using our deﬁnition of λ ( t ) j = |(cid:104) A ( t ) j − A ∗ j , A ∗ j (cid:105)| ≤ (cid:15) 2 t 2 , combining all the results for (8)
[Equation]
and using the fact that since A ( t ) is close to A ∗ , vectors A ( t ) j − A ∗ j and A ∗ j enclose an obtuse angle, we have the following for the j -th entry of the ( r + 1) -th iterate, x ( r +1) is given by
[Equation]
Here ξ ( r +1)
j is deﬁned as
[Equation]
Since, (cid:104)
A ∗ j , A ∗ i (cid:105) − (cid:104)
A ( t ) j , A ( t ) i (cid:105) =
(cid:104)
A ∗ j , A ∗ i − A
( t ) i (cid:105)
+ (cid:104)
A ∗ j − A
( t ) j , A ( t ) i (cid:105)
, we can write ξ ( r +1)
j as
[Equation]
where β ( t ) j is deﬁned as β
[Equation]
Note that β ( t ) j does not change for each iteration r of the coefﬁcient update step.
Further, by Claimwe show that | β ( t ) j | ≤ t β = O ( √ k(cid:15) t ) with probability at least (1 − δ ( t ) β ) .
Next, we deﬁne (cid:101) ξ ( r +1) j as
[Equation]
where ξ ( r +1)
j
[Equation]
Further, using Claim 1,
[Equation]
) since (cid:107) x ( r − 1) − x ∗ (cid:107)
=
O ( k ) .
Therefore, for the ( r + 1) -th iteration, we choose the threshold to be
[Equation]
(14) and the step-size by setting the “noise” component of (9) to be smaller than the “signal” part, specifically, half the signal component, i.e.,
[Equation]
Also, since we choose the threshold as τ
[Equation]
(cid:101) ξ ( r )max , where x (0)min =
C/ 2 , we have the following for the ( r + 1) -th iteration,
[Equation]
Therefore, for this step we choose η ( r +1) x as
[Equation]
Therefore, η ( r +1) x can be chosen as
[Equation]
, for a small constant c ( r +1) ( (cid:15) t , µ, k, n ) , η ( r +1) x .
In addition, if we set all η ( r ) x = η x , we have that η x = (cid:101) Ω( k √ n ) and therefore τ ( r ) =
τ = (cid:101) Ω( k 2 n ) .
Further, since we initialize with the hard-thresholding step, the entries in | x (0) | ≥ C/ 2 .
Here, we deﬁne (cid:101) ξ (0)max = C and η (0) x
= 1 / 2 , and set the threshold for initial step as η (0)
x (cid:101) ξ (0)max .
Proof of Lemma 3.
Using the deﬁnition of (cid:101) ξ ( (cid:96) )
i 1 as in (12), we have
[Equation]
| .
From Claim 2 we have that | β ( t ) i 1 | ≤ t β with probability at least (1 − δ ( t ) β ) .
Further, using Claim 1 , and letting C ( (cid:96) )
i :=
[Equation]
x
[Equation]
i − x
[Equation]
i |
[Equation]
| x
[Equation]
i − x ∗ i | ,
[Equation]
ξ (
[Equation]
) i 1 can be upper-bounded as
[Equation] [Equation]
Rearranging the expression for ( r + 1) -th update (9), and using (16) we have the following upperbound
[Equation]
Next, recursively substituting in for C ( r ) i 1 , where we deﬁne (cid:81) (cid:96)q = (cid:96) (1 − η ( q +1) x ) = 1 ,
C (
[Equation]
+1)
i≤
[Equation] [Equation]
(cid:81) q
[Equation] [Equation]
( q +
[Equation]
( t )
[Equation]
1 | r
+
(cid:80) (cid:96) =1 η ( (cid:96) )
x
r +1 (cid:81)
q = (cid:96) (1 −
[Equation]
( q +1) x ) + r +1 (cid:80) (cid:96) =1 η ( (cid:96) )
x (cid:101) ξ ( (cid:96) )
i 1 r +1 (cid:81) q
= (cid:96) (1 − η ( q +1) x ) .
Substituting for the upper-bound of (cid:101) ξ (
[Equation]
) i 1 from (16),
[Equation]
Here, α ( r +1) iis deﬁned as
[Equation]
Our aim now will be to express C ( (cid:96) )
i 1 for (cid:96) > 0 in terms of C (0)
i 2 .
Let each α ( (cid:96) )
j ≤ α ( (cid:96) ) i where j = i 1 , i 2 , . .
, i k .
Similarly, let C (0) j ≤ C (0)
i for j = i 1 , i 2 , . .
, i k , and all η ( (cid:96) )
x
= η x .
Then, using
Claim 3 we have the following expression for C ( R +1)
i,
[Equation]
Here, (1 − η x ) R ≤ (1 − η x + η x µ t √ n ) R ≤ δ R .
Next from Claim 4 we have that with probability at least (1 − δ ( t ) β ) ,
[Equation]
Therefore, for c x = µ t √ n / (1 −
[Equation]
n )
[Equation]
Now, using the deﬁnition of α ( R +1)
i
[Equation]
and using the result on sum of geometric series
[Equation]
we have
[Equation]
Therefore, C ( R ) i 1 is upper-bounded as
[Equation]
µ t √ n C (0)max δ R + C (0) i 1 δ R .
Further, since k = O ( √ n/µ log( n )) , kc x < 1 , therefore, we have
[Equation]
with probability at least (1 − δ ( t ) β ) .
Here, ( R + 1) kη x µ t √ n C (0)max δ R + C (0) i 1 δ R (cid:117) 0 for an appropriately large R .
Therefore, the error in each non-zero coefﬁcient is
[Equation]
with probability at least (1 − δ ( t ) β ) .
Proof of Lemma 4.
Using the expression for x ( R ) i 1 as deﬁned in (9), and recursively substituting for x ( r ) i 1 we have
[Equation]
where we set all η rx to be η x .
Further, on deﬁning
[Equation]
where γ ( R ) i 1 :
= (1 − η x ) R
( x (0) i 1 − x ∗ i 1 (1 − λ ( t ) i 1 )) , we have
[Equation]
Note that γ ( R ) i 1 can be made appropriately small by choice of R .
Further, by Claim 5 we have
[Equation]
with probability at least (1 − δ ( t ) β ) , where t β = O ( √ k(cid:15) t ) .
Proof of Lemma 5.
From Lemma 4 we have that for each j ∈ S ,
[Equation]
with probability at least (1 − δ ( t ) T − δ ( t ) β ) .
Further, let F x ∗ be the event that sign( x ∗ )
= sign( (cid:98) x ) , and let 1 F x ∗ denote the indicator function corresponding to this event.
As we show in Lemma 2, this event occurs with probability at least (1 − δ ( t ) β − δ ( t ) T ) .
Using this, we can write the expected gradient vector corresponding to the j -th sample as 1 F x ∗
[Equation]
Here, γ := E [( A ( t ) (cid:98) x − y )sign( x ∗ j ) 1 F x ∗ ] is small and depends on δ ( t ) T and δ ( t ) β , which in turn drops with (cid:15) t .
Therefore, γ diminishes with (cid:15) t .
Further, since 1 F x ∗ + 1 F x ∗ = 1 , and Pr [ F x ∗ ] = (1 − δ ( t ) β − δ ( t ) T ) , is very large,
[Equation]
Therefore, we can write g ( t ) j as
[Equation]
Since E [(1 − η x ) R A ( t ) S x (0) S ] can be made very small by choice of R , we absorb this term in γ .
Therefore,
[Equation]
Writing the expectation by sub-conditioning on the support,
[Equation]
S
[Equation] [Equation]
where we have used the fact that E x ∗ S [sign( x ∗ j )]
= 0 and introduced ∆ ( t )
j = E
[ A ( t ) S
ϑ ( R ) S sign( x ∗ j )] .
Next, since p j = E x ∗ S
[ x ∗ j sign( x ∗ j ) | j ∈ S ] , therefore,
[Equation]
Further, since q j = Pr [ j ∈ S ] =
O ( k/m ) ,
[Equation]
Further, by Claim 7 we have that
[Equation]
This completes the proof.
Proof of Lemma 6.
Let W =
{ j : i ∈ supp( x ∗ ( j ) ) } and then we have that
[Equation]
where (cid:98) x ( j ) ( i ) denotes the i -th element of the coefﬁcient estimate corresponding to the ( j ) -th sample.
Here, for (cid:96) =
| W | the summation
[Equation]
, has the same distribution as Σ (cid:96)j =1 z j , where each z j belongs to a distribution as
[Equation]
Also, E [( y − A ( t ) (cid:98) x )sign( (cid:98) x i )]
= q i E [ z ] ,
where q i =
Pr [ x ∗ i (cid:54) = 0]
=
Θ( km ) .
Therefore, since p = (cid:101) Ω( mk 2 ) , we have (cid:96) =
pq i
= (cid:101) Ω( k 3 ) non-zero vectors,
[Equation]
Let w j =
z
j − E
[ z ] , we will now apply the vector Bernstein result shown in Lemma 11.
For this, we require bounds on two parameters for these – L := (cid:107) w j (cid:107) and σ 2 := (cid:107) Σ
j E [ (cid:107) w j (cid:107) 2 ] (cid:107) .
Note that, since the quantity of interest is a function of x ∗ i , which are sub-Gaussian, they are only bounded almost surely .
To this end, we will employ Lemma 14 (Lemma 45 in (Arora et al., 2015)) to get a handle on the concentration.
Bound on the norm (cid:107) w (cid:107) : This bound is evaluated in Claim 8, which states that with probability at least (1 − δ ( t ) β − δ ( t ) T − δ ( t )HW ) ,
L := (cid:107) w (cid:107) =
(cid:107)
z − E [ z ] (cid:107) =
2 (cid:96) (cid:107) ( y − A
( t ) (cid:98) x )sign( (cid:98) x i ) | i ∈ S (cid:107) ≤ 2 (cid:96) (cid:107) (
y − A
( t ) (cid:98) x ) (cid:107) =
(cid:101)
O ( kt β (cid:96) ) .
Bound on variance parameter E [ (cid:107) w (cid:107) 2 ] : Using Claim 9, we have E [ (cid:107)
[Equation]
= O
[Equation]
k(cid:15)
[Equation]
t )
+ O ( kt 2 β ) .
Therefore, the bound on the variance parameter σ 2 is given by
[Equation]
From Claim 2 we have that with probability at least (1 − δ ( t ) β ) , t β =
O ( √ k(cid:15) t ) .
Applying vector
Bernstein inequality shown in Lemma 11 and using Lemma 14 (Lemma 45 in (Arora et al., 2015)), choosing (cid:96) = (cid:101) Ω( k 3 ) , we conclude
[Equation]
o
[Equation]
with probability at least (1 − δ ( t ) g i ) , where δ ( t ) g i = exp( − Ω( k )) .
Finally, substituting in (21) we have
[Equation]
with probability at least (1 − δ ( t ) g
i − δ ( t ) β − δ ( t ) T − δ ( t )HW ) .
Proof of Lemma 7.
Since we only have access to the empirical estimate of the gradient (cid:98) g ( t ) i , we will show that this estimate is correlated with ( A ( t ) j − A ∗ j ) .
To this end, ﬁrst from Lemma 6 we have that the empirical gradient vector concentrates around its mean, speciﬁcally,
[Equation]
with probability at least (1 − δ ( t ) g
i − δ ( t ) β − δ ( t ) T − δ ( t )HW ) .
From Lemma 5, we have the following expression for the expected gradient vector
[Equation]
Let g ( t ) j
= 4 ρ ( A ( t ) j − A ∗ j ) + v , where 4 ρ = p
j q j and v is deﬁned as
[Equation]
Then, (cid:98) g ( t ) i can be written as
[Equation]
23) where (cid:101) v = v + ( (cid:98) g
( t ) i − g ( t ) i ) .
Let (cid:107) (cid:101) v (cid:107) ≤ ρ (cid:107)
A
( t ) i − A ∗
i (cid:107) .
Using the deﬁnition of v as shown in (22) we have
[Equation]
Now for the ﬁrst term, since (cid:107) A ( t ) j (cid:107) = 1 , we have λ ( t )
j
[Equation]
therefore
[Equation]
Further, using Claim
[Equation]
Now, since (cid:107) A ( t ) − A ∗ (cid:107) ≤ 2 (cid:107) A ∗ (cid:107) , and further since (cid:107) A ∗ (cid:107) =
O ( (cid:112) m/n ) , we have that
[Equation]
+
[Equation]
A
[Equation]
= O ( (cid:113)
mn
[Equation]
Therefore, we need
[Equation]
Here, we use the fact that γ drops with decreasing (cid:15) t as argued in Lemma 5.
Therefore, if k =
O ( √ n ) we have
[Equation]
Now, using (23), we have
[Equation]
Substituting for (cid:107) (cid:101) v (cid:107) , this implies that (cid:107) (cid:100) g ( t ) j (cid:107) 2 ≤ 25 ρ 2 (cid:107) A ( t ) j − A ∗ j (cid:107) 2 .
Further, we also have the following lower-bound
[Equation]
Here, we use the fact that R.H.S. can be minimized only if (cid:101) v is directed opposite to the direction of
A ( t ) j − A ∗ j .
Now, we show that this gradient is ( ρ , 1 / 100 ρ , 0) correlated,
[Equation]
A
[Equation]
−
[Equation]
[Table]
Therefore, for this choice of k , there is no bias.
Further, since we choose 4 ρ = p
j q j , we have that ρ = Θ( k/m ) , therefore ρ + = 1 / 100 ρ = Ω
( m/k ) .
Applying Lemma 15 we have (cid:107) A ( t +1)
j
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
, for η A = O ( m/k ) with probability at least (1 − δ ( t ) T − δ ( t ) β − δ ( t ) g i ) .
Proof of Lemma 8.
Here, we will prove that (cid:98) g ( t ) deﬁned as
[Equation]
g
[Equation]
=
[Equation]
j
[Equation]
y
[Equation]
) − A
[Equation]
x
[Equation]
j )
[Equation]
sign( (cid:98) x
[Equation]
j ) ) (cid:62) , concentrates around its mean.
Notice that each summand ( y ( j ) − A ( t ) (cid:98) x ( j ) )sign( (cid:98) x ( j ) ) (cid:62) is a random matrix of the form ( y − A ( t ) (cid:98) x )sign( (cid:98) x ) (cid:62) .
Also, we have g ( t ) deﬁned as
[Equation]
To bound (cid:107) (cid:98) g ( t ) − g ( t ) (cid:107) , we are interested in (cid:107) (cid:80) pj =1 W j (cid:107) , where each matrix W j is given by
[Equation]
Noting that E [ W j ] = 0 , we will employ the matrix Bernstein result (Lemma 10) to bound (cid:107) (cid:98) g
( t ) − g ( t ) (cid:107) .
To this end, we will bound (cid:107) W j (cid:107) and the variance proxy
[Equation]
Bound on (cid:107) W j (cid:107) – First, we can bound both terms in the expression for W j by triangle inequality as
[Equation]
Here, we use Jensen’s inequality for the second term, followed by upper-bounding the expected value of the argument by (cid:107) ( y − A ( t ) (cid:98) x )sign( (cid:98) x ) (cid:62) (cid:107) .
Next, using Claim 8 we have that with probability at least (1 − δ ( t ) β − δ ( t ) T − δ ( t )HW ) , (cid:107) y − A ( t ) (cid:98) x
(cid:107) is (cid:101)
O ( kt β ) , and the fact that (cid:107) sign( x ) T (cid:107) =
[Equation]
[Equation]
Bound on the variance statistic v ( W j ) – For the variance statistic, we ﬁrst look at (cid:107) (cid:80) E [ W j W (cid:62) j ] (cid:107) ,
E [
[Equation]
(cid:62) − E [
[Equation]
y
[Equation]
t
[Equation]
sign
[Equation]
] × [sign( (cid:98) x
( j
[Equation]
)( y ( j
[Equation]
− A ( t
[Equation]
(cid:98) x ( j )
[Equation]
(cid:62) − ( E [( y
[Equation]
A ( t
[Equation]
(cid:98) x )sign( (cid:98) x ) (cid:62) )
(cid:62) ] .
Since E [( y − A ( t ) (cid:98) x )sign( (cid:98) x ) (cid:62) ]
E [( y − A ( t ) (cid:98) x )sign( (cid:98) x ) (cid:62) ] (cid:62) is positive semideﬁnite,
[Equation]
x ( j ) ) (cid:62) ] .
Now, since each (cid:98) x ( j ) has k non-zeros, sign( (cid:98) x ( j ) ) (cid:62) sign( (cid:98) x ( j ) )
=
k , and using Claim 10, with probability at least (1 − δ ( t ) T − δ ( t ) β )
[Equation] [Equation]
[Equation]
Similarly, expanding E [ W (cid:62) j W j ] , and using the fact that E [( y − A ( t ) (cid:98) x )sign( (cid:98) x ) (cid:62) ] (cid:62) E [( y −
A ( t ) (cid:98) x )sign( (cid:98) x ) (cid:62) ] is positive semi-deﬁnite.
Now, using Claim 8 and the fact that entries of
E [(sign( (cid:98) x ( j ) )sign( (cid:98) x ( j ) ) (cid:62) ] are q i on the diagonal and zero elsewhere, where q i
=
O
( k/m ) ,
[Equation]
Now, we are ready to apply the matrix Bernstein result.
Since, m = O ( n ) the variance statistic comes out to be O ( k 3 t 2 β pm ) (cid:107) A ∗ (cid:107) 2 , then as long as we choose p = (cid:101) Ω( mk 2 ) , with probability at least
[Equation]
where δ ( t ) g
= ( n + m ) exp( − Ω( m (cid:112) log( n )) .
Proof of Lemma 9.
The update step for the i -th dictionary element at the s + 1 iteration can be written as
[Equation]
Here, g ( t ) i is given by the following as per Lemma 5 with probability at least (1 − δ ( t ) T − δ (
[Equation]
) β )
[Equation]
Substituting the expression for g ( t ) i in the dictionary update step,
[Equation]
where ∆ ( t ) j = E [ A ( t ) ϑ
( R ) sign( x ∗ j )] j .
Therefore, the update step for the dictionary (matrix) can be written as
[Equation] [Equation]
25) where, U =
A ( t ) diag( p i q i λ ( t ) i ) and V =
A ( t ) Q , with the matrix Q given by,
[Equation]
and using the following intermediate result shown in Claim 7,
[Equation] [Equation]
, for i = j, = O ( p
[Equation]
(cid:15) t ) , for i (cid:54) =
j, we have (cid:107) Q i (cid:107) =
O ( √ mq
i,j p i (cid:15) t ) .
Hence, we have
[Equation]
≤ O ( mq i,j p i (cid:15) t ) .
Therefore,
[Equation]
We will now proceed to bound each term in (25).
Starting with ( A ( t ) − A ∗ )
diag(1 −
η A p i q i ) , and using the fact that p i
=
O (1) , q i
= O
( k/m ) , and (cid:107) A ( t ) − A ∗ (cid:107) ≤ 2 (cid:107)
A ∗ (cid:107) , we have (cid:107) ( A ( t ) − A ∗ )
diag(1 −
η A p i q i ) (cid:107) ≤ (1 − min i η A p
i
[Table]
Next, since (cid:107) A ( t ) j (cid:107) = 1 , we have λ
[Equation]
and λ
[Equation]
therefore
[Equation]
Using the results derived above, and the the result derived in Lemma 8 which states that with probability at least (1 − δ ( t ) β − δ ( t ) T − δ ( t
[Equation]
− δ ( t
[Equation]
g ) , (cid:107) (cid:98) g
( t ) − g ( t ) (cid:107) =
O ∗ ( km (cid:107) A ∗ (cid:107) ))
we have
[Equation]
A ∗ (cid:107) .
D A PPENDIX : P ROOFS OF INTERMEDIATE RESULTS
Claim 1 ( Incoherence of A ( t ) ) .
If A ∗ ∈ R n × m is µ -incoherent and (cid:107) A ∗
i − A ( t ) i (cid:107) ≤ (cid:15) t holds for each i ∈ [1 . . .
m ] , then A ( t ) ∈ R n × m is µ t -incoherent, where µ t = µ + 2 √ n(cid:15) t .
Proof of Claim 1.
We start by looking at the incoherence between the columns of A ∗ , for j (cid:54)
=
i , (cid:104)
A ∗ i , A ∗ j (cid:105) =
(cid:104)
A
[Equation] [Equation]
Since (cid:104)
A ∗ i , A ∗ j (cid:105) ≤ µ √ n , |(cid:104)
A ( t ) i , A ( t ) j (cid:105)| ≤ (cid:104)
A ∗ i , A ∗ j (cid:105) − (cid:104)
A ∗ i − A ( t
[Equation]
i , A
[Equation]
j (cid:105) − (cid:104)
A ( t ) i , A ∗ j − A ( t ) j (cid:105) , ≤ µ √ n + 2 (cid:15) t .
Claim 2 ( Bound on β ( t ) j : the noise component in coefﬁcient estimate that depends on (cid:15) t ) .
With probability (1 − δ ( t ) β ) , | β ( t ) j | is upper-bounded by t β
= O ( √ k(cid:15) t ) ,
where δ ( t ) β
= 2 k exp
[Equation]
Proof of Claim 2.
We have the following deﬁnition for β ( t ) j from (11),
[Equation]
Here, since x ∗ i are independent sub-Gaussian random variables, β ( t )
j is a sub-Gaussian random variable with the variance parameter evaluated as shown below
[Equation]
Therefore, by Lemma
[Equation]
Now, we need this for each β ( t ) j for j ∈ supp( x ∗ ) , union bounding over k coefﬁcients
[Equation]
where δ ( t ) β
= 2 k exp( − t 2 β 18 k(cid:15) 2 t ) .
Choosing t β
= O ( √ k(cid:15) t )
, we have that δ ( t ) β
= 2 k exp( − 1 O ( (cid:15) t ) ) .
Claim 3 ( Error in coefﬁcient estimation for a general iterate ( r + 1) ) .
The error in a general iterate r of the coefﬁcient estimation is upper-bounded as
[Equation]
i
[Equation]
Proof of Claim 3 .
From (17) we have the following expression for C ( r +1)
i
[Equation]
Our aim will be to recursively substitute for C ( (cid:96) − 1) ito develop an expression for C ( r +1) ias a function of C 0max .
To this end, we start by analyzing the iterates C (1) i 1 , C (2) i 1 , and so on to develop an expression for C ( r +1) ias follows.
Expression for C (1) i 1 – Consider C (1) i
[Equation]
Expression for C (2) i 1 – Next, C (2) i 1 is given by
[Equation]
Further, we know from (26) we have
[Equation]
Therefore, since (cid:80)
[Equation]
Expression for C (3) i 1 – Next, we writing C (3) i 1 ,
[Equation]
Here, using (27) we have the following expression for
[Equation]
Substituting for C (2) i 2 in the expression for C (3)
i
[Equation]
and rearranging the terms in the expression for C (3) i 1 , we have
[Equation]
Expression for C (4) i 1 –
Now
[Equation]
consider C (4)
i
[Equation]
Substituting for C (3) i 2 from (28), C (2)
i 2 from (27), C (1)
i 2 using (26), and rearranging,
[Equation] [Equation]
Notice that the terms have a binomial series like form.
To reveal this structure, let each α ( (cid:96) )
j ≤ α (
[Equation]
)max where j = i 1 , i 2 , . .
, i k .
Similarly, let C (0) j ≤ C (0)max for j = i 1 , i 2 , . .
, i k .
Therefore, we have
[Equation]
Further upper-bounding the expression
[Equation]
we have
[Equation]
Therefore,
[Equation]
Expression for C ( r +1)
i– With this, we are ready to write the general term,
[Equation]
Claim 4 ( An intermediate result for bounding the error in coefﬁcient calculations ) .
With probability (1 − δ ( t ) T − δ ( t ) β ) ,
[Equation]
Proof of Claim 4 .
Using (18), the quantity α ( (cid:96) )
i is deﬁned as
[Equation]
Therefore, we are interested in
[Equation]
Consider the ﬁrst term which depends on C (0) i .
Since (1 − η x ) ≤ (1 − η x + η x µ t √ n (cid:1)
[Equation]
we have
[Equation]
where δ R is a small constant
[Equation]
and a parameter which determines the number of iterations R required for the coefﬁcient update step.
Now
[Equation]
coming back to the quantity of interest
[Equation]
Now, using sum of geometric series result, we have that (cid:96) (cid:80)
[Equation]
and
[Equation]
Therefore, with probability at least (1 − δ ( t ) β ) ,
[Equation]
where λ ( t ) i ≤ (cid:15) 2 t 2 and | β ( t ) i | = t β with probability at least (1 − δ ( t ) β ) using Claim 2.
Claim 5 ( Bound on the noise term in the estimation of a coefﬁcient element in the support ) .
With probability (1 − δ ( t ) β ) , each entry ϑ ( R ) i 1 of ϑ ( R ) is upper-bounded as | ϑ ( R )
i 1 | ≤ O ( t β ) .
Proof of Claim 5.
From (19) ϑ ( R ) i 1 is deﬁned as
[Equation]
where γ ( R ) i 1 :
= (1 − η x ) R
( x (0) i 1 − x ∗ i 1 (1 − λ ( t ) i 1 )) .
Further, ξ ( r ) i 1 is as deﬁned in (10),
[Equation]
Therefore
[Equation]
we have the following expression for ϑ ( R ) i
[Equation]
Now ϑ ( R )
i 1 can be upper-bounded as
[Equation]
Since from Claim 6 we have
[Equation]
Further, since 1 − (1 − η x ) r − 1 ≤
[Equation]
we have that
R
[Equation]
Therefore, |
[Equation]
Now, since each | β ( t ) i | =
t β with probability at least (1 − δ ( t ) β ) for the t -th iterate, and k =
O ∗ ( √ n µ log( n ) ) , therefore kc x < 1 , we have that
[Equation]
with probability at least (1 − δ ( t ) β ) .
Claim 6 ( An intermediate result for ϑ ( R ) i 1 calculations ) .
For c x = µ t √ n / (1 − µ t √ n ) , we have
[Equation]
r
[Equation]
i
[Equation]
Proof of Claim 6.
Here, from Claim 3 we have that for any i 1 ,
C
[Equation]
therefore C ( r − 1) iis given by
C
[Equation]
i
[Equation]
Further, the term of interest C ( r − 1)
i
[Equation]
) R − r can be upper-bounded by
C ( r − 1)
i
[Equation]
1 −
[Equation]
From the deﬁnition of α ( (cid:96) )
i from (18), α ( r − 1) ican be written as
[Equation]
Therefore, we have
[Equation]
Next, to get a handle on α ( (cid:96) )max (cid:0) 1 − η x + η x µ t √ n (cid:1) r − (cid:96) − 2 , consider the following using the deﬁnition of α ( (cid:96) )
i from (18), where η ( i ) x
= η x for all i ,
[Equation]
Therefore,
[Equation]
Therefore,
[Equation]
Further
[Equation]
since (1 − η x ) ≤ (1 − η x + η x µ t √ n ) ,
[Equation]
Therefore
[Equation]
combining all the results we have that
[Equation]
for a constant c x = µ t √ n / (1 − µ t √ n ) ,
C
[Equation]
Claim 7 ( Bound on the noise term in expected gradient vector estimate ) .
(cid:107) ∆ ( t ) j (cid:107) where ∆ ( t ) j :=
E [ A ( t ) ϑ
( R ) sign( x ∗ j )] is upper-bounded as,
[Equation]
Proof of Claim
[Equation]
From (30) we have the following deﬁnition for ϑ ( R ) j
[Equation]
where β ( t ) j is deﬁned as the following (11)
[Equation]
Consider E x ∗ S [ ϑ ( R ) S sign( x ∗ j ) | S ] , where ϑ ( R ) S is a vector with each element as deﬁned in (30).
Therefore, the elements of the vector E x ∗ S [ ϑ ( R ) S sign( x ∗ j ) | S ]] are given by
[Equation]
Consider the general term of interest
[Equation]
Further, since
[Equation]
we have that
[Equation] [Equation]
Further, for ♠ s :=
E x ∗ S [ C ( r − 1) s sign( x ∗ s − x ( r ) s )sign( x ∗ j ) | S ]
we have that
[Equation]
s
[Equation]
In addition, for (cid:80) s (cid:54) =
i ♠
s we have that
[Equation] [Equation]
Therefore, using the results for ♣ and (cid:80) s (cid:54) =
i ♠ s , we have that E x ∗ S
[ ϑ ( R ) j sign( x ∗ j ) | S ] = γ
( R ) i for i
=
j , and for i (cid:54) =
j we have
[Equation]
Here, from Claim 6, for c x = µ t √ n / (1 − µ t √ n ) we have
[Equation]
Further, due to our assumptions on sparsity, kc x ≤ 1 ; in addition by Claim 2, and with probability at least (1 − δ ( t ) β ) we have | β ( t )max | ≤ t β , substituting,
[Equation] [Equation]
= O
[Equation]
) , with probability at least (1 − δ ( t ) β ) .
Combining results from (31), (32) and substituting for the terms in (33) using the analysis above,
[Equation]
Note that since γ ( R )
i
[Equation]
− x ∗ i
[Equation]
1 − λ
[Equation]
) i )) can be made small by choice of R .
Also, since Pr [ i, j ∈ S ] = q i,j , we have
[Equation]
Claim 8 ( An intermediate result for concentration results ) .
With probability (1 − δ ( t ) β − δ ( t ) T − δ ( t )HW ) (cid:107) y − A ( t ) (cid:98) x
(cid:107) is upper-bounded by (cid:101)
O ( kt β ) .
Proof of Claim 8.
First, using Lemma 4 we have
[Equation]
Therefore, the vector (cid:98) x S , for S ∈ supp( x ∗ ) can be written as
[Equation] [Equation]
where (cid:98) x has the correct signed-support with probability at least (1 − δ T ) using Lemma 2.
Using this result, we can write (cid:107) y − A ( t ) (cid:98) x (cid:107) as
[Equation]
Now, since Λ ( t ) ii ≤ (cid:15) 2 t 2 we have
[Equation]
With x ∗ S being independent and sub-Gaussian, using Lemma 13, which is a result based on the
Hanson-Wright result (Hanson and Wright, 1971) for sub-Gaussian random variables, and since (cid:107) A ( t ) S − A ∗ S (cid:107) ≤
[Equation]
A
[Equation]
t ) S
[Equation]
A ∗ S (cid:107)
[Equation]
≤ √ k(cid:15) t , we have that with probability at least (1 − δ ( t )HW ) (cid:107)♣
[Equation]
, where δ ( t )HW = exp( − 1 O ( (cid:15) t ) ) .
Now, consider the (cid:107)♣(cid:107) F , since (cid:107) A ( t ) S − A ∗ S (cid:107) F ≤ √ k(cid:15) t (
cid:107)♣(cid:107) F := (cid:107)
[Equation]
1 −
[Equation]
t
[Equation]
A
[Equation]
A
[Equation]
t
[Equation]
+
[Equation]
F
[Equation]
t
[Equation]
(cid:107) A ∗ S (cid:107) F , ≤ √
[Equation]
1 −
[Equation]
Consider the (cid:107)♠(cid:107) term.
Using Claim 5, each ϑ ( R ) j is bounded by O ( t β ) .
with probability at least (1 − δ ( t ) β )
Therefore, (cid:107)♠(cid:107)
= (cid:107) A ( t ) S
ϑ ( R ) S (cid:107) ≤ (cid:107)
A ( t ) S (cid:107)(cid:107)
[Equation]
Again, since (cid:107)
[Equation]
Finally, combining all the results and using the fact that (cid:107) A ∗ S (cid:107) F ≤ √ k
[Equation]
Claim 9 ( Bound on variance parameter for concentration of gradient vector ) .
For z := ( y −
A ( t ) (cid:98) x )sign( (cid:98) x i ) | i ∈ S the variance parameter E
[ (cid:107) z (cid:107) 2 ] is bounded as E [ (cid:107) z (cid:107) 2 ]
=
O ( k(cid:15) 2 t )
+ O ( kt 2 β ) with probability at least (1 − δ ( t ) β − δ ( t ) T ) .
Proof of Claim 9.
For the variance E
[ (cid:107) z (cid:107) 2 ] , we focus on the following,
[Equation]
S ] .
Here, (cid:98) x S is given by
[Equation]
Therefore, E [ (cid:107) z (cid:107) 2 ] can we written as
[Equation]
We will now consider each term in (35) separately.
We start with ♥ .
Since x ∗ S s are conditionally independent of S , E
[ x ∗ S x ∗ (cid:62) S ]
= I .
Therefore
[Equation]
we can simplify this expression as
[Equation]
Rearranging the terms we have the following for ♥ ,
[Equation]
A
[Equation]
Λ
[Equation] [Equation]
+
[Equation]
A
[Equation]
−
[Equation]
( I − Λ ( t ) S ) (cid:107)
[Equation]
F | i ∈ S ] .
Therefore, ♥ can be upper-bounded as
[Equation]
For ♥ 1 , since (cid:107) A ( t ) S (cid:107) ≤ √ k(cid:15) t + 2 , we have ♥ 1 :=
[Equation] [Equation]
A
[Equation] [Equation]
| i
[Equation] [Equation]
A
[Equation]
(cid:107)
[Equation]
Next, since (1 − λ ( t ) j ) ≤ 1 , we have the following bound for ♥♥ 2 :=
E [ (cid:107) ( A ∗ S − A ( t ) S )( I − Λ ( t ) S ) (cid:107) 2 F | i ∈ S ] ≤ E [ (cid:107) A ∗ S − A ( t ) S (cid:107)
[Equation]
Further, ♥ 3 can be upper-bounded by using bounds for ♥ 1 and ♥ 2 .
Combining the results of upperbounding ♥ 1 , ♥ 2 , and ♥ 3 we have the following for (36)
[Equation]
Next, by Claim 5, ϑ ( R ) j is upper-bounded as | ϑ ( R )
j | ≤ O ( t β ) .
with probability (1 − δ ( t ) β ) .
Therefore, the term ♦ , the second term of (35), can be bounded as
[Equation]
Finally, combining all the results, the term of interest in (35) has the following form
[Equation]
Claim 10 ( Bound on variance parameter for concentration of gradient matrix ) .
With probability (1 − δ ( t ) T − δ ( t ) β ) , the variance parameter (cid:107) E
[( y − A ( t ) (cid:98) x )( y − A (cid:98) x ) (cid:62) ] (cid:107) is upper-bounded by
O ( k 2 t 2 β m ) (cid:107) A ∗ (cid:107) 2 .
Proof of Claim 10.
Let F x ∗ be the event that sign( x ∗ )
= sign( (cid:98) x ) , and let 1 F x ∗ denote the indicator function corresponding to this event.
As we show in Lemma 2, this event occurs with probability at least (1 − δ ( t ) β − δ ( t ) T ) , therefore,
[Equation]
Here, γ is small.
Under the event F x ∗ , (cid:98) x has the correct signed-support.
Again, since 1 F x ∗ = 1 − 1 ¯ F x ∗ ,
[Equation]
Now, using Lemma 4 with probability at least (1 − δ ( t ) T − δ ( t ) β ) , (cid:98) x
S admits the following expression
[Equation]
Therefore we have
[Equation]
Using the expression above E [( y − A ( t ) (cid:98) x )( y − A ( t ) (cid:98) x ) (cid:62) ] can be written as
[Equation]
x ∗ S − A ( t ) S
ϑ ( R ) S ) (cid:62) ] .
Sub-conditioning, we have
E [( y − A ( t ) (cid:98) x )( y − A ( t ) (cid:98) x ) (cid:62) ] =
E S
[( A ∗ S − A ( t ) S ( I − Λ ( t ) S ))
E x ∗ S [ x ∗ S x ∗ (cid:62) S | S ]
( A ∗ (cid:62) S − ( I − Λ ( t ) S )
A ( t ) (cid:62) S )]
− E S
[ A ( t ) S E x ∗ S [ ϑ ( R ) S x ∗ (cid:62) S | S ]
( A ∗ (cid:62) S − ( I
[Equation]
Λ
[Equation]
S
[Equation]
A
[Equation] [Equation]
S )]
− E S
[( A
[Equation]
A
[Equation]
S ( I
[Equation]
S ))
E
[Equation]
[Equation] [Equation]
| S ]
A
[Equation]
) (cid:62) S ] + E S
[ A ( t ) S E x
[Equation]
S [ ϑ ( R ) S ( ϑ ( R ) S ) (cid:62) | S ] A ( t ) (cid:62) S ] .
Now, since E x ∗ S [ x ∗ S x ∗ (cid:62) S | S ] = I , (cid:107) E
[( y − A ( t ) (cid:98) x )( y − A ( t ) (cid:98) x ) (cid:62) ] (cid:107) ≤ (cid:107) E S
[( A ∗ S − A ( t ) S ( I − Λ ( t ) S ))
( A ∗ (cid:62) S − ( I − Λ ( t ) S )
A ( t ) (cid:62) S )] (cid:107) (cid:124) (cid:123)(cid:122) (cid:125) ♣
[Equation]
Let’s start with the ﬁrst term ( ♣ ) of (37)
[Equation]
which can be written as
[Equation]
Now consider each term of equation (38).
First, since
[Equation]
and E S [ 1 i,j ∈ S ]
= O ( k 2 m 2 ) , we can upper-bound ♣ 1 :
= (cid:107) E S
[ A ∗ S
A ∗
[Equation]
S ] (cid:107) as
[Equation]
where 1 m × m denotes an m × m matrix of ones.
Now, we turn to ♣ 2 :
= (cid:107) E S
[ A ∗ S ( I − Λ ( t ) S )
A ( t ) (cid:62) S ] (cid:107) in (38), which can be simpliﬁed as
[Equation]
Further, since A ( t ) is ( (cid:15) t , 2) -close to A ∗ , we have that (cid:107) A ( t ) (cid:107) ≤ (cid:107)
A ( t ) − A
[Equation]
+ (cid:107) A ∗ (cid:107) ≤ 3 (cid:107)
A ∗ (cid:107) , therefore
[Equation]
Similarly, ♣ 3 (38) is also O ( k 2 m ) (cid:107)
A ∗ (cid:107) 2 .
Next, we consider ♣ 4 :
= (cid:107) E S [ A ( t ) S
( I − Λ ( t ) S ) 2 A ( t ) (cid:62) S ] (cid:107) in (38) which can also be bounded similarly as
[Equation]
Therefore, we have the following for ♣ in (37)
[Equation]
Consider ♠ in (37).
Letting M =
E x ∗ S [ ϑ ( R ) S x ∗ (cid:62) S | S ] , and using the analysis similar to that shown in 7, we have that elements of M ∈ R k × k are given by
[Equation]
We have the following,
[Equation]
( t ) S M ( A ∗ (cid:62) S − ( I − Λ ( t ) S )
A ( t ) (cid:62) S )] .
Therefore, since E S [ 1 i,j ∈ S | S ]
=
O ( k 2 m 2 ) , and (cid:107) 1 m × m (cid:107) = m , ♠ := (cid:107)
E S
[ A ( t ) S M ( A ∗ (cid:62) S − ( I − Λ ( t ) S )
A ( t ) (cid:62) S )] (cid:107)
[Equation]
Therefore,
[Equation]
Similarly, ♥ in (37) is also bounded as ♠ .
Next, we consider ♦ in (37).
In this case, letting
E x ∗ S [ ϑ ( R ) S ( ϑ ( R ) S ) (cid:62) | S ] = N , where N ∈ R k × k is a matrix whose each entry N i,j ≤ | ϑ ( R )
i || ϑ
( R ) j | .
Further, by Claim 5, each element ϑ ( R ) j is upper-bounded as
[Equation]
with probability at least (1 − δ ( t ) β ) .
Therefore, ♦ = (cid:107)
[Equation]
Again, using the result on | ϑ ( R ) i 1 | , we have
[Equation]
Combining all the results for ♣
[Equation]
♠
[Equation]
♥ and ♦
[Equation]
we have,
[Equation]
E A DDITIONAL E XPERIMENTAL R ESULTS
We now present some additional results to highlight the features of NOODL.
Speciﬁcally, we compare the performance of NOODL (for both dictionary and coefﬁcient recovery) with the state-of-theart provable techniques for DL presented in Arora et al.
(2015) (when the coefﬁcients are recovered via a sparse approximation step after DL) 1 .
We also compare the performance of NOODL with the popular online DL algorithm in Mairal et al.
(2009), denoted by Mairal ‘ 09 .
Here, the authors show that alternating between a (cid:96) 1 -based sparse approximation and dictionary update based on block co-ordinate descent converges to a stationary point, as compared to the true factors in case of NOODL.
Data Generation: We generate a ( n = 1000) × ( m = 1500) matrix, with entries drawn from
N (0 , 1) , and normalize its columns to form the ground-truth dictionary A ∗ .
Next, we perturb A ∗ with random Gaussian noise, such that the unit-norm columns of the resulting matrix, A (0) are 2 / log( n ) away from A ∗ , in (cid:96) 2 -norm sense, i.e., (cid:15)
= 2 / log( n ) ; this satisﬁes the initialization assumptions in A.4 .
At each iteration, we generate p = 5000 samples Y ∈ R 1000 × 5000 as Y =
A ∗ X ∗ , where X ∗ ∈ R m × p has at most k = 10 , 20 , 50 , and 100 , entries per column, drawn from the Radamacher distribution.
We report the results in terms of relative Frobenius error for all the experiments, i.e., for a recovered matrix (cid:99) M , we report (cid:107) (cid:99) M − M ∗ (cid:107) F / (cid:107) M ∗ (cid:107) F .
To form the coefﬁcient estimate for Mairal ‘ 09 via Lasso (Tibshirani, 1996) we use the FISTA (Beck and Teboulle, 2009) algorithm by searching across 10 values of the regularization parameter at each iteration.
Note that, although our phase transition analysis for NOODL shows that p = m sufﬁces, we use p = 5000 in our convergence analysis for a fair comparison with related techniques.
E. C OEFFICIENT R ECOVERY
Table 4 summarizes the results of the convergence analysis shown in Fig.
Here, we compare the dictionary and coefﬁcient recovery performance of NOODL with other techniques.
For
Arora15(‘‘biased’’) and Arora15(‘‘unbiased’’) , we report the error in recovered coefﬁcients after the HT step ( X HT ) and the best error via sparse approximation using Lasso 2 Tibshirani (1996), denoted as X Lasso , by scanning over 50 values of regularization parameter.
For
Mairal ‘ 09 at each iteration of the algorithm we scan across 10 values 3 of the regularization parameter, to recover the best coefﬁcient estimate using Lasso ( via FISTA), denoted as X Lasso .
We observe that NOODL exhibits signiﬁcantly superior performance across the board.
Also, we observe that using sparse approximation after dictionary recovery, when the dictionary suffers from a bias, leads to poor coefﬁcient recovery 4 , as is the case with Arora15(‘‘biased’’) ,
Arora15(‘‘unbiased’’) , and Mairal ‘ 09 .
This highlights the applicability of our approach in real-world machine learning tasks where coefﬁcient recovery is of interest.
In fact, it is a testament to the fact that, even in cases where dictionary recovery is the primary goal, making progress on the coefﬁcients is also important for dictionary recovery.
In addition, the coefﬁcient estimation step is also online in case of NOODL, while for the stateof-the-art provable techniques (which only recover the dictionary and incur bias in estimation) need additional sparse approximation step for coefﬁcient recovery.
Moreover, these sparse approximation techniques (such as Lasso) are expensive to use in practice, and need signiﬁcant tuning.
E. C OMPUTATIONAL T IME
In addition to these convergence results, we also report the computational time taken by each of these algorithms in Table 4.
The results shown here were compiled using 5 cores and 200 GB RAM of Intel Xeon E 5 − 2670 Sandy Bridge and Haswell E5-2680v3 processors.
The primary takeaway is that although NOODL takes marginally more time per iteration as compared to other methods when accounting for just one Lasso update step for the coefﬁcients, it (a) is in fact faster per iteration since it does not involve any computationally expensive tuning procedure to scan across regularization parameters; owing to its geometric convergence property (b) achieves orders of magnitude superior error at convergence, and as a result, (c) overall takes signiﬁcantly less time to reach such a solution.
Further, NOODL’s computation time can be further reduced via implementations using the neural architecture illustrated in Section 4.
Note that since the coefﬁcient estimates using just the HT step at every step may not yield a usable result for Arora15(‘‘unbiased’’) and Arora15(‘‘biased’’) as shown in Table 4, in practice, one has to employ an additional (cid:96) 1 -based sparse recovery step.
Therefore, for a fair comparison, we account for running sparse recovery step(s) using Lasso (via the Fast Iterative ShrinkageThresholding Algorithm (FISTA) (Beck and Teboulle, 2009) ) at every iteration of the algorithms
Arora15(‘‘biased’’) and Arora15(‘‘unbiased’’) .
For our technique, we report the average computation time taken per iteration.
However, for the rest of the techniques, the coefﬁcient recovery using Lasso (via FISTA) involves a search over various values of the regularization parameters ( 10 values for this current exposition).
As a result, we analyze the computation time per iteration via two metrics.
First of these is the average computation time taken per iteration by accounting for the average time take per Lasso update (denoted as “ Accounting for one Lasso update ”), and the second is the average time taken per iteration to scan over all ( 10 ) values of the regularization parameter (denoted as “ Overall Lasso search ”) .
[Table]
As shown in Table
[Table]
in comparison to NOODL the techniques described in Arora et al.
(2015) still incur a large error at convergence
[Table]
while the popular online DL algorithm of Mairal et al.
(2009) exhibits very slow convergence rate.
Combined with the convergence results shown in Fig.
[Table]
we observe that due to NOODL’s superior convergence properties
[Table]
it is overall faster and also geometrically converges to the true factors.
This again highlights the applicability of NOODL in practical applications, while guaranteeing convergence to the true factors
[Table]
F A PPENDIX : S TANDARD R ESULTS
Deﬁnition 6 (sub-Gaussian Random variable) .
Let x ∼ subGaussian( σ 2 ) .
Then, for any t > 0 , it holds that
[Equation]
F.
C ONCENTRATION RESULTS
Lemma 10 (Matrix Bernstein (Tropp, 2015)) .
Consider a ﬁnite sequence W k ∈ R n × m of independent, random, centered matrices with dimension n .
Assume that each random matrix satisﬁes
E [ W k ] = 0 and (cid:107) W k (cid:107) ≤
R
almost surely.
Then, for all t ≥ 0 ,
[Equation]
where σ 2 := max {(cid:107) (cid:80)
[Equation]
E
[Equation]
] (cid:107) , (cid:107) (cid:80) k
E
[ W
[Equation]
] (cid:107)} .
Furthermore,
E [ (cid:107) (cid:80)
[Equation]
Lemma 11 (Vector Bernstein (Tropp, 2015)) .
Consider a ﬁnite sequence w k ∈ R n of independent, random, zero mean vectors with dimension n .
Assume that each random vector satisﬁes E [ w k ]
=and (cid:107) w k (cid:107) ≤ R almost surely.
Then, for all t ≥ 0 ,
[Equation]
where σ 2 := (cid:107) (cid:80) k
E [ (cid:107) w
[Equation]
2 ] (cid:107) .
Furthermore,
[Equation]
Lemma 12.
Chernoff Bound for sub-Gaussian Random Variables Let w be an independent subGaussian random variables with variance parameter σ 2 , then for any t > 0 it holds that
[Equation]
Lemma 13 ( Sub-Gaussian concentration (Rudelson and Vershynin, 2013) ) .
Let M ∈ R n × m be a ﬁxed matrix.
Let w be a vector of independent, sub-Gaussian random variables with mean zero and variance one.
Then, for an absolute constant c ,
[Equation]
F.
Lemma 14 ((Arora et al., 2015) Lemma 45) .
Suppose that the distribution of Z satisﬁes Pr [ (cid:107) Z (cid:107) ≥
L (log(1 /ρ ))
C ] ≤ ρ ] for some constant C > 0 , then
If p =
n O then
(cid:107) Z ( j ) (cid:107) ≤ (cid:101) O
( L ) holds for each j with probability at least (1 − ρ ) and, 2.
[Equation]
E
[Equation]
Z
[Equation]
n −
[Equation] [Equation]
In particular, if 1 p (cid:80) pj =1 Z ( j ) (1 − 1 (cid:107) Z (cid:107)≥ (cid:101) Ω( L ) ) is concentrated with probability (1 − ρ ) , then so is 1 p (cid:80) pj
=1 Z ( j ) .
Lemma 15 (Theorem 40 (Arora et al., 2015)) .
Suppose random vector g ( t ) is a ( ρ , ρ + , ζ t ) - correlated with high probability with z ∗ for t ∈
[ T ] where T ≤ poly ( n ) , and η A satisﬁes 0 < η A ≤ 2 ρ + , then for any t ∈ [ T ] ,
[Equation]
In particular, if (cid:107) z (0) − z ∗ (cid:107) ≤ (cid:15) 0 and ζ t ≤ ( ρ ) o
((1 − 2 ρ η ) t ) (cid:15) 20 + ζ , where ζ = max t ∈ [ T ] ζ t , then the updates converge to z ∗ geometrically with systematic error ζ/ρ in the sense that
[Equation]