U NDERSTANDING APPROXIMATE AND UNROLLED DICTIONARY LEARNING FOR PATTERN RECOVERY
Anonymous authors
Paper under double-blind review
A BSTRACT
Dictionary learning consists of ﬁnding a sparse representation from noisy data and is a common way to encode data-driven prior knowledge on signals.
Alternating minimization (AM) is standard for the underlying optimization, where gradient descent steps alternate with sparse coding procedures.
The major drawback of this method is its prohibitive computational cost, making it unpractical on large real-world data sets.
This work introduces an approximate formulation of dictionary learning based on unrolling and compares it to alternating minimization to ﬁnd the best trade-off between speed and precision.
We study the asymptotic behavior and convergence rate of gradients estimates in both methods.
We show that unrolling performs better on the support of the inner problem solution and during the ﬁrst iterations.
Finally, we apply unrolling on pattern learning in magnetoencephalography (MEG) with the help of a stochastic algorithm and compare the performance to a state-of-the-art method
I NTRODUCTION
Pattern learning provides insightful information on the data in various biomedical applications.
Typical examples include the study of magnetoencephalography (MEG) recordings, where one aims to analyze the electrical activity in the brain from measurements of the magnetic ﬁeld around the scalp of the patient (Dupr ´ e la Tour et al., 2018).
One may also mention neural oscillations study in the local ﬁeld potential (Cole & Voytek, 2017) or QRS complex detection in electrocardiograms (Xiang et al., 2018) among others.
Dictionary learning (Olshausen & Field, 1997; Aharon et al., 2006; Mairal et al., 2009) is particularly efﬁcient on pattern learning tasks, such as blood cells detection
(Yellin et al., 2017) and MEG signals analysis (Dupr ´ e la Tour
et
al., 2018).
This framework assumes that the signal can be decomposed into a sparse representation in a redundant basis of patterns – also called atoms.
In other words, the goal is to recover a sparse code Z ∈ R n × T and a dictionary D ∈ R m × n from noisy measurements Y ∈ R m × T which are obtained as the linear transformation DZ , corrupted with noise B ∈ R m × T : Y =
DZ + B .
Theoretical elements on identiﬁability and local convergence have been proven in several studies (Gribonval et al., 2015; Haeffele & Vidal, 2015; Agarwal et al., 2016; Sun et al., 2016).
Sparsity-based optimization problems related to dictionary learning generally rely on the usage of the (cid:96) 0 or (cid:96) 1 regularizations.
In this paper, we study Lasso-based (Tibshirani, 1996) dictionary learning where the dictionary D is learned in a set of constraints C by solving
[Equation]
Dictionary learning can be written as a bi-level optimization problem to minimize the cost function with respect to the dictionary only, as mentioned in Mairal et al.
[Equation]
with
[Equation]
Computing the data representation Z ∗ ( D ) is often referred to as the inner problem, while the global minimization is the outer problem.
The most classical constraint set C is the unit norm, where each atom is normalized to avoid scale-invariant issues.
C can also correspond to normalized convolutional kernels to perform Convolutional Dictionary Learning (Grosse et al., 2007).
Classical dictionary learning methods solve this bi-convex optimization problem through Alternating Minimization (AM) (Mairal et al., 2009).
It consists in minimizing the cost function F over Z with a ﬁxed dictionary D and then performing projected gradient descent to optimize the dictionary with a ﬁxed Z .
While AM provides a simple strategy to perform dictionary learning, it can be inefﬁcient on large-scale data sets due to the need to resolve the inner problems precisely for all samples.
In recent years, many studies have focused on algorithm unrolling (Scetbon et al., 2021;
Tolooshams et al., 2020) to overcome this issue.
The core idea consists of unrolling the algorithm, which solves the inner problem, and then computing the gradient with respect to the dictionary with the help of back-propagation through the iterates of this algorithm.
Gregor & LeCun (2010) popularized this method and ﬁrst proposed to unroll ISTA (Daubechies et al., 2004)
– a proximal gradient descent algorithm designed for the Lasso – to speed up the computation of Z ∗ ( D ) .
The N + 1 -th layer of this network – called LISTA – is obtained as Z N +1 =
ST λL ( W 1 Y + W 2 Z N ) , with ST being the soft-thresholding operator.
This work has led to many contributions aiming at improving this method and providing theoretical justiﬁcations in a supervised (Chen et al., 2018; Liu & Chen, 2019) or unsupervised (Moreau & Bruna, 2017; Ablin et al., 2019) setting.
For such unrolled algorithms, the weights W 1 and W 2 can be re-parameterized as functions of D – as illustrated in Figure A in appendix – such that the output Z N ( D ) matches the result of N iterations of ISTA, i.e.
[Equation]
and
[Equation]
, where L = (cid:107) D (cid:107)
[Equation]
Then, the dictionary can be learned by minimizing the loss F ( Z N ( D ) , D ) over D with backpropagation.
We will refer to this approach as Deep Dictionary Learning (DDL).
Variants of DDL with different kinds of regularization (Scetbon et al., 2021; Tolooshams et al., 2020; Lecouat et al., 2020), image processing based on metric learning (Tang et al., 2020), and classiﬁcation tasks with scattering (Zarka et al., 2019) have been proposed in the literature, among others.
While these techniques have achieved good performance levels on several signal processing tasks, the reasons they speed up the learning process are still unclear.
In this work, we study unrolling in Lasso-based dictionary learning as an approximate bi-level optimization problem.
What makes this work different from Bertrand et al. (2020), Ablin et al.
and Tolooshams & Ba (2021) is that we study the instability of non-smooth bi-level optimization and unrolled sparse coding out of the support, which is of major interest in practice with a small number of layers.
In Section 2, we analyze the convergence of the Jacobian computed with automatic differentiation and ﬁnd out that its stability is guaranteed on the support of the sparse codes only. De facto, numerical instabilities in its estimation make unrolling inefﬁcient after a few dozen iterations.
In Section 3, we empirically show that unrolling leads to better results than AM only with a small number of iterations of sparse coding, making it possible to learn a good dictionary in this setting.
Then we propose a stochastic approach to make this method usable on large data sets, and we apply it to pattern learning in magnetoencephalography (MEG) in Section 4.
We do so by adapting unrolling to rank one convolutional dictionary learning on multivariate time series (Dupr ´ e la Tour et al., 2018).
We show that there is no need to unroll more than a few dozen iterations to obtain satisfying results, leading to a signiﬁcant gain of time compared to a state-of-the-art algorithm.
B I - LEVEL OPTIMIZATION FOR APPROXIMATE DICTIONARY LEARNING
As Z ∗ ( D ) does not have a closed-form expression, G cannot be computed directly.
A solution is to replace the inner problem Z ∗ ( D ) by an approximation Z N ( D ) obtained through N iterations of a numerical optimization algorithm or its unrolled version.
This reduces the problem to minimizing
G N ( D ) (cid:44) F ( Z N ( D ) , D ) .
The ﬁrst question is how sub-optimal global solutions of G N are compared to the ones of G .
Proposition 2.1 shows that the global minima of G N converge as fast as the numerical approximation Z N in function value.
Proposition 2.1 Let D ∗ =
arg min D ∈C G ( D ) and D ∗ N = arg min D ∈C G N ( D ) , where N is the number of unrolled iterations.
We denote by K ( D ∗ ) a constant depending on D ∗ , and by C ( N ) the convergence speed of the algorithm, which approximates the inner problem solution.
We have
G N ( D ∗ N ) − G ( D ∗ ) ≤ K ( D ∗ ) C ( N ) .
The proofs of all theoretical results are deferred to Appendix C. Proposition 2.1 implies that when Z N is computed with FISTA (Beck & Teboulle, 2009), the function value for global minima of
G N converges with speed C ( N ) =
1 N 2 towards the value of the global minima of F .
Therefore, solving the inner problem approximately leads to suitable solutions for equation 2, given that the optimization procedure is efﬁcient enough to ﬁnd a proper minimum of G N .
As the computational cost of z N increases with N , the choice of N results in a trade-off between the precision of the solution and the computational efﬁciency, which is critical for processing large data sets.
Moreover, learning the dictionary and computing the sparse codes are two different tasks.
The loss
G N takes into account the dictionary and the corresponding approximation Z N ( D ) to evaluate the quality of the solution.
However, the dictionary evaluation should reﬂect its ability to generate the same signals as the ground truth data and not consider an approximate sparse code that can be recomputed afterward.
Therefore, we should distinguish the ability of the algorithm to recover a good dictionary from its ability to learn the dictionary and the sparse codes at the same time.
In this work, we use the metric proposed in Moreau & Gramfort (2020) for convolutions to evaluate the quality of the dictionary.
We compare the atoms using their correlation and denote as C the cost matrix whose entry
i, j compare the atom i of the ﬁrst dictionary and j of the second.
We deﬁne a sign and permutation invariant metric S ( C ) = max σ ∈ S n 1 n (cid:80) ni
=1 | C σ ( i ) ,i | , where S n is the group of permutations of [1 , n ] .
This metric corresponds to the best linear sum assignment on the cost matrix C , and it can be computed with the Hungarian algorithm.
Note that doing so has several limitations and that evaluating the dictionary is still an open problem.
Without loss of generality, let
T = 1 and thus z ∈ R n in the rest of this section.
Gradient estimation in dictionary learning.
Approximate dictionary learning is a non-convex problem, meaning that good or poor local minima of G N may be reached depending on the initialization, the optimization path, and the structure of the problem.
Therefore, a gradient descent on G N has no guarantee to ﬁnd an adequate minimizer of G .
While complete theoretical analysis of these problems is arduous, we propose to study the correlation between the gradient obtained with G N and the actual gradient of G , as a way to ensure that the optimization dynamics are similar.
Once z ∗ ( D ) is known, Danskin (1967, Thm 1) states that g ∗ ( D ) =
∇ G ( D ) is equal to ∇ 2 F ( z ∗ ( D ) , D ) , where ∇ 2 indicates that the gradient is computed relatively to the second variable in F .
Even though the inner problem is non-smooth, this result holds as long as the solution z ∗ ( D ) is unique.
In the following, we will assume that D (cid:62) D is invertible on the support of z ∗ ( D ) , which implies the uniqueness of z ∗ ( D ) .
This occurs with probability one if D is sampled from a continuous distribution (Tibshirani, 2013).
AM and DDL differ in how they estimate the gradient of G .
AM relies on the analytical formula of g ∗ and uses an approximation z N of z ∗ , leading to the approximate gradient g 1 N ( D ) = ∇ 2 F ( z N ( D ) , D ) .
We evaluate how well g 1 N approximates g ∗ in Proposition 2.2.
Proposition 2.2 Let D ∈ R m × n .
Then, there exists a constant L 1 > 0 such that for every number of iterations N
[Equation]
Proposition 2.2 shows that g 1 N converges as fast as the iterates of ISTA converge.
DDL computes the gradient automatically through z N ( D ) .
As opposed to AM, this directly minimizes the loss
G N ( D ) .
Automatic differentiation yields a sub-gradient g 2 N ( D ) such that
[Equation]
(4) where J N : R m × n → R n is the weak Jacobian of z N ( D ) with respect to D and J + N denotes its adjoint.
The product between J + N and ∂ 1 F ( z N ( D ) , D ) is computed via automatic differentiation.
Proposition 2.3 Let D ∈ R m × n .
Let S ∗ be the support of z ∗ ( D ) , S N be the support of z N and (cid:101) S N = S N ∪ S ∗ .
Let f ( z , D ) =
12 (cid:107)
Dz − y (cid:107) 22 be the data-ﬁtting term in F .
Let R ( J , (cid:101) S ) =
J + (cid:0) ∇ 21 , 1 f ( z ∗ , D ) (cid:12) 1 (cid:101) S (cid:1) + ∇ 22 , 1 f ( z ∗ , D ) (cid:12) 1 (cid:101) S .
Then there exists a constant L 2 > 0 and a subsequence of (F)ISTA iterates z φ ( N ) such that for all N ∈ N :
[Equation]
This sub-sequence z φ ( N ) corresponds to iterates on the support of z ∗ .
Proposition 2.3 shows that g 2 N may converge faster than g 1 N once the support is reached.
Ablin et al.
(2020) and Tolooshams & Ba (2021) have studied the behavior of strongly convex functions, as it is the case on the support, and found similar results, allowing Tolooshams & Ba (2021) to show that automatic differentiation leads to a better gradient estimation once on the support under minor assumptions.
However, we are also interested in characterizing the behavior outside of the support, where the gradient estimation is difﬁcult because of the sub-differential.
In practice, automatic differentiation uses the sign operator as a sub-gradient of (cid:107)·(cid:107) 1 .
The convergence behavior of g 2 N is also driven by R ( J N , (cid:102) S N ) and thus by the weak Jacobian computed via back-propagation.
We ﬁrst compute a closed-form expression of the weak Jacobian of z ∗ ( D ) and z N ( D ) .
We then show that R ( J N , (cid:102) S N ) ≤ L (cid:107) J N − J ∗ (cid:107) and we analyze the convergence of J N towards J ∗ .
Study of the Jacobian.
The computation of the Jacobian can be done by differentiating through
ISTA.
In Theorem 2.4, we show that J N +1 depends on J N and the past iterate z N , and converges towards a ﬁxed point.
This formula can be used to compute the Jacobian during the forward pass, avoiding the computational cost of back-propagation and saving memory.
Theorem 2.4 At iteration N + 1 of ISTA, the weak Jacobian of z N +1 relatively to D l , where D l is the l -th row of D , is given by induction:
[Equation]
∂ ( z N ) ∂D l will be denoted by J Nl .
It converges towards the weak Jacobian J ∗ l of z ∗ ( D ) , whose values are
[Equation]
on the support S ∗ of z ∗ , and 0 elsewhere.
Moreover, R ( J ∗ , S ∗ ) = 0 .
This result is similar to Bertrand et al.
(2020) where the Jacobian of z is computed over λ to perform hyper-parameter optimization in Lasso-type models.
Using R ( J ∗ , S ∗ )
= 0 , we can write
[Equation]
as (cid:13)(cid:13) ∇ 21 , 1 f ( z ∗ , D ) (cid:13)(cid:13) 2 = L .
If the back-propagation were to output an accurate estimate J N of the weak Jacobian J ∗ , (cid:13)(cid:13)(cid:13) R ( J N , (cid:102) S N ) (cid:13)(cid:13)(cid:13) would be 0, and the convergence rate of g 2 N could be twice as fast as the one of g 1 N .
To quantify this, we now analyze the convergence of J N towards J ∗ .
In
Proposition 2.5, we compute an upper bound of (cid:13)(cid:13)
J Nl − J ∗ l (cid:13)(cid:13) with possible usage of truncated backpropagation (Shaban et al., 2019).
Truncated back-propagation of depth K corresponds to an initial estimate of the Jacobian J N − K = 0 and iterating the induction in Theorem 2.4.
Proposition 2.5 Let N be the number of iterations and K be the back-propagation depth.
We assume that ∀ n ≥ N − K, S ∗ ⊂ S n .
Let ¯ E N = S n \ S ∗ , let L be the largest eigenvalue of D (cid:62) : ,S ∗ D : ,S ∗ , and let µ n be the smallest eigenvalue of D (cid:62) : ,S n D : ,S n − 1 .
Let B n = (cid:13)(cid:13)(cid:13) P E n − D (cid:62) : , ¯ E n D †(cid:62) : ,S ∗ P S ∗ (cid:13)(cid:13)(cid:13) , where
P S is the projection on R S and D † is the pseudo-inverse of D .
We have
[Equation]
Proposition 2.5 reveals multiple stages in the Jacobian estimation.
First, one can see that if all iterates used for the back-propagation lie on the support S ∗ , the Jacobian estimate has a quasi-linear convergence, as shown in the following corollary.
Corollary 2.6 Let µ > 0 be the smallest eigenvalue of D (cid:62) : ,S ∗ D : ,S ∗ .
Let K ≤ N be the backpropagation depth and let ∆ N = F ( z N , D ) − F ( z ∗ , D )
+ L 2 (cid:107)
z
N − z ∗ (cid:107) .
Suppose that ∀ n ∈ [ N − K, N ]; S n ⊂ S ∗ .
Then, we have
[Equation]
[Figure]
However, for sample 2, full back-propagation makes the convergence unstable, and truncated backpropagation improves its behavior, as described in Proposition 2.5.
The proportion of stable and unstable samples in this particular example is displayed in Figure 2.
Once the support is reached, ISTA also converges with the same linear rate (1 − µL ) .
Thus the gradient estimate g 2 N converges almost twice as fast as g 1 N in the best case – with optimal sub-gradient – as
O ( K (1 − µL ) 2 K ) .
This is similar to Ablin et al.
(2020, Proposition.5) for strongly convex functions.
Second, Proposition 2.5 shows that (cid:13)(cid:13)
J ∗ l − J Nl (cid:13)(cid:13) may increase when the support is not well-estimated, leading to a deterioration of the gradient estimate.
This is due to an accumulation of errors materialized by the sum in the right-hand side of the inequality, as the term B N (cid:107) z ∗ (cid:107) may not vanish to 0 as long as S N (cid:54)⊂ S ∗ .
Interestingly, once the support is reached at iteration S < N , the errors converge linearly towards 0, and we recover the fast estimation of g ∗ with g 2 .
Therefore, Lasso-based DDL should either be used with a low number of steps or truncated back-propagation to ensure stability. These results apply for all linear dictionaries, including convolutions.
[Figure]
Numerical illustrations.
We will now illustrate these theoretical results on simple data depending on the number N of unrolled iterations.
The data are generated from a random Gaussian dictionary D of size 30 × 50 , with Bernoulli-Gaussian sparse codes z with sparsity 0.3 and σ 2 z = 1 , and Gaussian noise of variance σ 2 noise = 0 .
More details are available in Appendix A. Figureconﬁrms the linear convergence of J Nl once the support is reached.
However, the convergence might be unstable when the number of iteration grows, leading to exploding gradient, as illustrated in the second case.
When this happens, using a small number of iterations or truncated back-propagation becomes necessary to prevent accumulating errors.
It is also of interest to look at the proportion of unstable Jacobians, as done in Figure 2.
We recover behaviors observed in the ﬁrst and second case in Figure 1.
40% samples suffer from numerical instabilities in this example.
This has a negative impact on the gradient estimation outside of the support.
We display the convergence behavior of the gradients estimated by AM and by DDL with different back-propagation depths (20, 50, full) for simulated data and images in Figure 3.
We unroll FISTA instead of ISTA to make the convergence faster.
We observed similar behaviors for both algorithms in early iterations but using ISTA required too much memory to reach full convergence.
As we optimize using a line search algorithm, we are mainly interested in the ability of the estimate to provide an adequate descent direction.
Therefore, we display the convergence in angle deﬁned as the cosine similarity (cid:104) g , g ∗ (cid:105)
[Equation]
The angle provides a good metric to assert that the two gradients are correlated and thus will lead to similar optimization paths.
We also provide the convergence in norm in appendix.
We compare g 1 N and g 2 N with the relative difference of their angles with g ∗ , deﬁned as
[Equation]
When its value is positive, DDL provides the best descent direction.
Generally, when the back-propagation goes too deep, the performance of g 2 N decreases
[Figure]
compared to g 1 N , and we observe large numerical instabilities.
This behavior is coherent with the
Jacobian convergence patterns studied in Proposition 2.5.
Once on the support, g 2 N reaches back the performance of g 1 N as anticipated.
In the case of a real image, unrolling beats AM by up to 20% in terms of gradient direction estimation when the number of iterations does not exceed 50, especially with small back-propagation depth.
This highlights that the principal interest of unrolled algorithms is to use them with a small number of layers – or, equivalently, a small number of iterations.
A PPROXIMATE DICTIONARY LEARNING IN PRACTICE
This section introduces practical guidelines on Lasso-based approximate dictionary learning with unit norm constraint, and we provide empirical justiﬁcations for its ability to recover the dictionary.
We also propose a strategy to scale DDL with a stochastic optimization method.
We provide a full description of all our experiments in Appendix A.
We optimize with projected gradient descent combined to a line search to compute high-quality steps sizes.
The computations have been performed on a GPU NVIDIA Tesla V100-DGXS 32GB using PyTorch (Paszke et al., 2019).
Improvement of precision.
As stated before, a low number of iterations allows for efﬁcient and stable computations, but this makes the sparse code less precise.
One can learn the steps sizes of (F)ISTA to speed up convergence and compensate for imprecise representations, as proposed by
Ablin et al.
(2019) for LISTA.
To avoid poor results due to large degrees of freedom in unsupervised learning, we propose a method in two steps to reﬁne the initialization of the dictionary before relaxing the constraints on the steps sizes:
We learn the dictionary with ﬁxed step sizes equal to 1 L where L = (cid:107) D (cid:107) 2 , given by convergence conditions.
Lipschitz constants or upper bounds are computed at each gradient step with norms, or the FFT for convolutions, outside the network graph’s scope.
Then, once convergence is reached, we jointly learn the step sizes and the dictionary.
Both are still updated using gradient descent with line search to ensure stable optimization.
The use of LISTA-like algorithms with no ground truth generally aims at improving the speed of sparse coding when high precision is not required.
When it is the case, the ﬁnal sparse codes can be computed separately with FISTA (Beck & Teboulle, 2009) or coordinate descent (Wu et al., 2008) to improve the quality of the representation.
3. O PTIMIZATION DYNAMICS IN APPROXIMATE DICTIONARY LEARNING
In this part, we study empirical properties of approximate dictionary learning related to global optimization dynamics to put our results on gradient estimation in a broader context.
[Figure] [Figure]
Unrolling v. AM.
In Figure 4, we show the number of gradient steps before reaching convergence, the behavior of the loss F N , and the recovery score deﬁned at the beginning of the section for synthetic data generated by a Gaussian dictionary.
As a reminder, S ( C ) = max σ ∈ S n 1 n (cid:80) ni =1 | C σ ( i ) ,i | where C is the correlation matrix between the columns of the true dictionary and the estimate.
The number of iterations corresponds to N in the estimate z N ( D ) .
First, DDL leads to fewer gradient steps than AM in the ﬁrst iterations.
This suggests that automatic differentiation better estimates the directions of the gradients for small depths.
However, computing the gradient requires backpropagating through the algorithm, and DDL takes 1.5 times longer to perform one gradient step than AM on average for the same number of iterations N .
When looking at the loss and the recovery score, we notice that the advantage of DDL for the minimization of F N is minor without learning the steps sizes, but there is an increase of performance concerning the recovery score.
DDL better estimates the dictionary for small depths, inferior to 50.
When unrolling more iterations, AM performs as well as DDL on the approximate problem and is faster.
Approximate DL.
Figure 4 also shows that high-quality dictionaries are obtained before the convergence of F N , either with AM or DDL.
40 iterations are sufﬁcient to reach a reasonable solution concerning the recovery score, even though the loss is still very far from the optimum at this stage.
This suggests that computing optimal sparse codes at each gradient step is unnecessary to recover the dictionary.
To illustrate that, we display in Figure 5 the PSNR of a noisy image reconstruction depending on the number of iterations, compared to full AM dictionary learning with 1000 iterations.
As for synthetic data, optimal performance is reached very fast.
In this particular case, the model converges after 80 seconds with approximate DL unrolled for 20 iterations of FISTA compared to 600 seconds in the case of standard DL.
Note that the speed rate highly depends on the value of λ .
Higher values of λ tend to make FISTA converge faster, and unrolling becomes unnecessary in this case.
On the contrary, unrolling is more efﬁcient than AM for lower values of λ .
Loss landscape.
The ability of gradient descent to ﬁnd adequate local minima strongly depends on the structure of the problem.
To quantify this, we evaluate the variation of PSNR depending on the Signal to Noise Ratio (SNR) – deﬁned as 10 log 10 (cid:16) σ 2 σ 2 b (cid:17) where σ 2 b is the variance of the noise – for 50 random initializations in the context of convolutional dictionary learning on a task of image denoising, with 20 unrolled iterations.
Figure 5 shows that approximate CDL is robust to random initialization when the level of noise is not too high.
In this case, all local minima are similar in terms of reconstruction quality.
We provide a visualization of the loss landscape with the help of ideas presented in Li et al.
The algorithm computes a minimum, and we chose two properly rescaled vectors to create a plan from this minimum.
The 3D landscape is displayed on this plan in Figure B using the Python library
K3D-Jupyter 2 .
We also compare in Figure 5 ( right ) the shapes of local minima in 1D by computing the values of the loss along a line between two local minima.
These visualizations conﬁrm that (approximate) dictionary learning locally behaves like a convex function with similar local minima.
3. S
[Figure]
For the moment, we only presented full-batch gradient descent methods, which do not scale due to memory issues.
In order to apply DDL in realistic settings, we propose to adapt Stochastic Gradient Descent (SGD), commonly used for neural networks.
The major advantage is that the sparse coding is not performed on all data at each forward pass, leading to signiﬁcant time and memory savings.
This is critical to learn dictionaries on large data sets.
The major issue is that the choice of gradient steps is critical to the optimization process in dictionary learning, and SGD methods based on simple heuristics like rate decay are difﬁcult to tune in this context.
We propose to leverage a new method introduced in Vaswani et al.
(2019), which consists of performing a stochastic line search.
The algorithm computes a good step size at each epoch, after which a heuristic decreases the maximal step.
Figure 6 displays the recovery score function of the time for various mini-batch sizes on a problem with 10 5 samples.
The data were generated as in Figure 1 but with a larger dictionary ( 50 × 100 ).
The algorithm achieves good performances with small mini-batches and thus limited memory usage.
We also compare this method with Online dictionary learning (Mairal et al., 2009) in
Figure E. It shows that our method speeds up the dictionary recovery, especially for lower values of λ .
Note that this strategy can also be adapted very easily for convolutional models by taking sub-windows of the full signal and performing SGD, as demonstrated in Section 4.
A PPLICATION TO PATTERN LEARNING IN MEG SIGNALS
In magnetoencephalography (MEG), the measurements of the magnetic ﬁelds over the scalp consist of hundreds of simultaneous recordings, which provide information on the neural activity during a large period, from minutes to hours.
Convolutional dictionary learning makes it possible to learn cognitive patterns corresponding to physiological activities (Dupr´e la Tour et al., 2018).
As the electromagnetic waves propagate through the brain at the speed of light, every sensor measures the same waveform simultaneously but not at the same intensity.
The authors propose to rely on multivariate convolutional sparse coding (CSC) with rank-1 constraint to leverage this physical property and learn prototypical patterns.
In this case, space and time patterns are disjoint in each atom: D k = u k v Tk where u gathers the spatial activations on each channel and v corresponds to the temporal pattern.
[Figure]
The algorithm uses 40 atoms, 30 unrolled iterations and 100 iterations with batch size 20.
We recover heartbeat (0) and blinking (1) artifacts, and an auditory evoked response (2) among others.
[Figure]
where λ max =
(cid:13)(cid:13)
D T y (cid:13)(cid:13) ∞ is the maximal value of λ for which the sparse codes are not 0.
The best setups achieve between 80% and 90% average correlation with alphacsc results in aboutseconds, while the original computations took around 1400 seconds to run.
This leads to the model
[Equation]
) where n is the number of atoms, T is the total recording time, t is the kernel size, and S is the number of sensors.
We propose to learn u and v with Stochastic Deep CDL unrolled for a few iterations to speed up the computations of the atoms.
Figure 7 reproduces the multivariate CSC experiments of alphacsc 3 (Dupr ´ e la Tour et al., 2018) on the dataset sample of MNE (Gramfort et al., 2013) –minutes of recordings with 204 channels sampled at 150Hz with visual and audio stimuli.
The algorithm recovers the main waveforms and spatial patterns with approximate sparse codes and without performing the sparse coding on the whole data set at each gradient iteration, which leads to a signiﬁcant gain of time.
We are able to distinguish several meaningful patterns as heartbeat and blinking artifacts or auditive evoked response.
As this problem is unsupervised, it is difﬁcult to provide robust quantitative quality measurements.
Therefore, we compare our patterns toimportant patterns recovered by alpahcsc in terms of correlation in Table 1.
Good setups achieve between 80% and 90% average correlation ten times faster.
C ONCLUSION
Dictionary learning is an efﬁcient technique to learn patterns in a signal but is challenging to apply to large real-world problems.
This work showed that approximate dictionary learning, which consists in replacing the optimal solution of the Lasso with a time-efﬁcient approximation, offers a valuable trade-off between computational cost and quality of the solution compared to complete Alternating
Minimization.
This method, combined with a well-suited stochastic gradient descent algorithm, scales up to large data sets, as demonstrated on a MEG pattern learning problem.
Future work.
This work provided a theoretical study of the asymptotic behavior of unrolling in approximate dictionary learning.
In particular, we showed that numerical instabilities make DDL usage inefﬁcient when too many iterations are unrolled.
However, the super-efﬁciency of DDL in the ﬁrst iterations remains unexplained, and our ﬁrst ﬁndings would beneﬁt from theoretical justiﬁcations.
Ethics Statement.
The MEG data conform to ethic guidelines (no individual names, collected under individual’s consent, ...).
Reproducibility Statement.
We provide the code in the supplementary materials, a full description of all our experiments in Appendix A, and the proofs of our theoretical results in Appendix C
A F ULL DESCRIPTION OF THE EXPERIMENTS
This section provides complementary information on the experiments presented in the paper.
A. C ONVERGENCE OF THE J ACOBIANS - F IGURE 1 AND F IGURE
We generate a normalized random Gaussian dictionary D of dimension 30 × 50 , and sparse codes z from a Bernoulli Gaussian distribution of sparsity 0.3 and σ 2 = 1 .
The signal to process is y =
Dz + b where b is an additive Gaussian noise with σ 2 noise = 0 .
The Jacobians are computed for a random perturbation D + b D of D where b D is a Gaussian noise of scale 0 .
5 σ 2 D .
J Nl corresponds to the approximate Jacobian with N iterations of ISTA with λ = 0 .
J ∗ l corresponds the true Jacobian computed with sparse codes obtained after 10 4 iterations of ISTA with λ = 0 .
In Figure 2, the norm (cid:13)(cid:13)
J Nl − J ∗ l (cid:13)(cid:13) is computed for 50 samples.
A. C ONVERGENCE OF THE GRADIENT ESTIMATES - F IGURE
Synthetic data.
We generate a normalized random Gaussian dictionary D of dimension 30 × 50 , and 1000 sparse codes z from a Bernoulli Gaussian distribution of sparsity 0.3 and σ 2 = 1 .
The signal to process is y
=
Dz + b where b is an additive Gaussian noise with σ 2 noise = 0 .
The gradients are computed for a random perturbation D + b D of D where b D is a Gaussian noise of scale 0 .
5 σ 2 D .
Noisy image.
A 128 × 128 black-and-white image is degraded by a Gaussian noise with σ 2 noise = 0 . 1 and normalized.
We processed 1000 patches of dimension 10 × 10 from the image, and we computed the gradients for a dictionary composed of 128 random patches.
g N corresponds to the gradient for N iterations of FISTA with λ = 0 .
g ∗ corresponds to the true gradient computed with a sparse code obtained after 10 4 iterations of FISTA.
A. O PTIMIZATION DYNAMICS ON SYNTHETIC DATA - F IGURE
We generate a normalized random Gaussian dictionary D of dimension 30 × 50 , and sparse codes z from a Bernoulli Gaussian distribution of sparsity 0.3 and σ 2 = 1 .
The signal to process is y
=
Dz + b where b is an additive Gaussian noise with σ 2 noise = 0 .
The initial dictionary is taken as a random perturbation D + b D of D where b D is a Gaussian noise of scale 0 .
5 σ 2 D .
N corresponds to the number of unrolled iterations of FISTA.
F ∗ is the value of the loss for 10iterations minus 10 − 3 .
S ∗ is the score obtained after 10 3 iterations plus 10 − 3 .
The optimization is done with λ = 0 .
We compare the number of gradient steps (left), the loss values (center), and the recovery scores (right) for 50 different dictionaries.
DDL with steps sizes learning is evaluated on 100 iterations only due to memory and optimization time issues.
A. O PTIMIZATION DYNAMICS AND LOSS LANDSCAPES ON IMAGES - F IGURE
A 128 × 128 black-and-white image is degraded by a Gaussian noise and normalized.
A. S TOCHASTIC DDL ON SYNTHETIC DATA - F IGURE
We generate a normalized random Gaussian dictionary D of dimension 50 × 100 , and 10 5 sparse codes z from a Bernoulli Gaussian distribution of sparsity 0.3 and σ 2 = 1 .
The signal to process is y
=
Dz + b where b is an additive Gaussian noise with σ 2 noise = 0 .
The initial dictionary is taken as a random gaussian dictionary.
We compare stochastic and full-batch line search projected gradient descent with 30 unrolled iterations of FISTA and λ = 0 .
1 , without steps sizes learning.
Stochastic DDL is run for 10 epochs with a maximum of 100 iterations for each epoch.
A. P ATTERN LEARNING IN MEG - F IGURE
Stochastic Deep CDL on 6 minutes of recordings of MEG data with 204 channels and a sampling rate of 150Hz.
We remove the powerline artifacts and high-pass ﬁlter the signal to remove the drift which can impact the CSC technique.
The signal is also resampled to 150 Hz to reduce the computational burden.
This preprocessing procedure is presented in alphacsc , and available in the code in the supplementary materials.
The algorithm learns 40 atoms of 1 second on mini batches of 10 seconds, with 30 unrolled iterations of FISTA, λ scaled = 0 . 3 , and 10 epochs with 10 iterations per epoch.
The number of mini-batches per iteration is 20, with possible overlap.
B E XTRA FIGURES AND EXPERIMENTAL RESULTS
LISTA - Figure A. Illustration of LISTA for Dictionary Learning with initialization Z 0 = 0 for
N = 3 .
W 1 D =
L ( D ) (cid:62) , W 2 D
=
( I − 1 L ( D ) (cid:62) D ) , where L = (cid:107) D (cid:107) 2 .
The result Z N ( D ) output by the network is an approximation of the solution of the LASSO.
[Equation]
Loss landscape in 2D - Figure
B.
We provide a visualization of the loss landscape with the help of ideas presented in Li et al.
The algorithm computes a minimum, and we chose two properly rescaled vectors to create a plan from this minimum.
The 3D landscape is displayed on this plan in the appendix using the Python library K3D-Jupyter.
This visualization and the visualization in 1D conﬁrm that (approximate) dictionary learning locally behaves like a convex function with smooth local minima.
Gradient convergence in norm - Figure C. Gradient estimates convergence in norm for synthetic data ( left ) and patches from a noisy image right ).
The setup is similar to Figure 3.
Both gradient estimates converge smoothly in early iterations.
When the back-propagation goes too deep, the performance of g 2 N decreases compared to g 1 N , and we observe large numerical instabilities.
This behavior is coherent with the Jacobian convergence patterns studied in Proposition 2.5.
Once on the support, g 2 N reaches back the performance of g 1 N .
[Figure]
Computation time to reach 0.95 recovery score - Figure D.
The setup is similar to Figure 6.
A random Gaussian dictionary of size 50 × 100 generates the data from 10 5 sparse codes with sparsity 0.3.
The approximate sparse coding is solved with λ = 0 .
1 and 30 unrolled iterations of FISTA.
The algorithm achieves good performances with small mini-batches and thus limited memory usage.
Stochastic DDL can process large amounts of data and recovers good quality dictionaries faster than full batch DDL.
Sto DDL vs.
Online DL - Figure E. We compare the time Online DL from spams 4 (Mairal et al., 2009) and Stochastic DDL need to reach a recovery score of 0.95 with a batch size of 2000.
Online DL is run with 10 threads.
We repeat the experiment 10 times for different values of λ from 0.1 to 1.0.
The setup is similar to Figure D, and we initialize both methods randomly.
Stochastic
DDL is more efﬁcient for smaller values of λ , due to the fact that sparse coding is slower in this case.
For higher values of λ , both methods are equivalent.
Another advantage of Stochastic DDL is its modularity.
It works on various kinds of dictionary parameterization thanks to automatic differentiation, as illustrated on 1-rank multivariate convolutional dictionary learning in Figure 7.
C P ROOFS OF THEORETICAL RESULTS
This section gives the proofs for the various theoretical results in the paper.
[Figure]
[Figure]
i m
[Figure]
u t
Time to reach a score of
[Figure]
[Figure]
m e ( s )
[Figure]
C. P ROOF OF P ROPOSITION 2.1.
Proposition 2.1 Let D ∗ =
arg min D ∈C G ( D ) and D ∗ N = arg min D ∈C G N ( D ) , where N is the number of unrolled iterations.
We denote by K ( D ∗ ) a constant depending on D ∗ , and by C ( N ) the convergence speed of the algorithm, which approximates the inner problem solution.
We have
[Equation]
Let G ( D ) (cid:44) F ( Z ∗ ( D ) , D ) and G N ( D ) (cid:44) F ( Z N ( D ) , D ) where Z ∗ ( D ) =
arg min Z ∈ R n × T F ( Z , D ) and Z N ( D ) =
FISTA ( D , N ) .
Let D ∗ =
arg min D ∈C G ( D ) and D ∗ N = arg min D ∈C G N ( D ) .
We have
[Equation]
By deﬁnition of D ∗ N
[Equation]
[Equation]
The convergence rate of FISTA in function value for a ﬁxed dictionary D is
[Equation]
[Equation]
Therefore
[Equation] [Equation]
Hence
[Equation]
C. P ROOF OF P ROPOSITION 2.
Proposition 2.2 Let D ∈ R m × n .
Then, there exists a constant L 1 > 0 such that for every number of iterations N
[Equation]
We have
[Equation]
z 0 ( D ) =
and the iterates ( z N ( D ))
N ∈ N converge towards z ∗ ( D ) .
Hence, they are contained in a closed ball around z ∗ ( D ) .
As ∇ 2 F ( · , D ) is continuously differentiable, it is locally Lipschitz on this closed ball, and there exists a constant L 1 ( D ) depending on D such that
[Equation]
C. P ROOF OF P ROPOSITION 2.3.
Proposition 2.3 Let D ∈ R m × n .
Let S ∗ be the support of z ∗ ( D ) , S N be the support of z N and (cid:101) S N = S N ∪ S ∗ .
Let f ( z , D ) =
12 (cid:107)
Dz − y (cid:107) 22 be the data-ﬁtting term in F .
Let R ( J , (cid:101) S ) =
J + (cid:0) ∇ 21 , 1 f ( z ∗ , D ) (cid:12) 1 (cid:101) [Equation] + ∇ [Equation]
1 f
[Equation]
z [Equation] , D ) [Equation]
1 (cid:101) S .
Then there exists a constant L 2 > 0 and a subsequence of (F)ISTA iterates z φ ( N ) such that for all N ∈ N :
[Equation]
This sub-sequence z φ ( N ) corresponds to iterates on the support of z ∗ .
We have
[Equation]
We adapt equation (6) in Ablin et al.
[Equation]
(19) where
[Equation]
As z N and z ∗ are on (cid:102) S N
[Equation]
As stated in Proposition 2.2, ∇ 2 f ( · , D ) is locally Lipschitz, and R D ,
z N is the Taylor rest of ∇ 2 f ( · , D ) .
Therefore, there exists a constant L D , z such that
[Equation]
[Equation]
We know that 0 ∈ ∇ 1 f ( z ∗ , D )+ λ∂ (cid:107)·(cid:107) 1 ( z ∗ ) .
In other words, ∃ u ∗ ∈ λ∂ (cid:107)·(cid:107) 1 ( z ∗ ) s.t. ∇ 1 f ( z ∗ , D )+ u ∗ = 0 .
Therefore we have:
[Equation] [Equation]
Let L z , z be the Lipschitz constant of ∇ 1 f ( · , D ) .
(F)ISTA outputs a sequence such that there exists a sub-sequence ( z φ ( N ) ) N ∈ N which has the same support as z ∗ .
For this sub-sequence, u ∗ ∈ λ∂ (cid:107)·(cid:107) 1 ( z φ ( N ) ) .
Therefore, there exists R z , z φ ( N ) such that
[Equation]
For this sub-sequence
[Equation]
we can adapt Proposition 2 from Ablin et al.
Let L 2 =
L D , z + L z , z , we have
[Equation]
Theorem 2.4 At iteration N + 1 of ISTA, the weak Jacobian of z N +1 relatively to D l , where D l is the l -th row of D , is given by induction:
[Equation]
∂ ( z N ) ∂D l will be denoted by J Nl .
It converges towards the weak Jacobian J ∗ l of z ∗ ( D ) , whose values are
[Equation]
on the support S ∗ of z ∗ , and 0 elsewhere.
Moreover, R ( J ∗ , S ∗ ) = 0 .
We start by recalling a Lemma from Deledalle et al.
[Equation]
Lemma C.
The soft-thresholding ST µ deﬁned by ST µ ( z ) = sgn
( z ) (cid:12) ( | z | − µ )
+ is weakly differentiable with weak derivative dST µ ( z ) dz
[Equation]
Coordinate-wise, ISTA corresponds to the following equality:
[Equation]
The Jacobian is computed coordinate wise with the chain rule:
[Equation]
p = ( n (cid:88) [Equation] =
[Equation]
∂ ∂D
[Equation]
n (cid:88) j =
D ji y j )) (31)
Last term: ∂ ∂D
[Equation]
n (cid:88) j = D ji y [Equation] = δ
[Equation]
y
[Equation]
Second term: ∂ ∂D
[Equation]
(cid:88) p
[Equation]
( z [Equation]
[Equation]
(33) ∂D
ji D jp ∂D lk = 
 2 D lk if j = l and i = p = k
D lp if j = l and i = k and p (cid:54) =
k
D li if j = l and i (cid:54) =
k and p =
k else (34)
Therefore:
[Equation]
Hence:
[Equation]
This leads to the following vector formulation:
[Equation]
On the support of z ∗ , denoted by S ∗
[Equation]
this quantity converges towards the ﬁxed point:
[Equation] [Equation]
Elsewhere, J ∗ l is equal to 0.
To prove that R ( J ∗ , S ∗ ) = 0 , we use the expression given by equation
[Equation]
C. P ROOF OF P ROPOSITION 2.5 AND C OROLLARY 2.
Proposition 2.5 Let N be the number of iterations and K be the back-propagation depth.
We assume that ∀ n ≥ N − K, S ∗ ⊂ S n .
Let ¯ E N = S n \ S ∗ , let L be the largest eigenvalue of D (cid:62) : ,S ∗ D : ,S ∗ , and let µ n be the smallest eigenvalue of D (cid:62) : ,S n D : ,S n − 1 .
Let B n = (cid:13)(cid:13)(cid:13) P E n − D (cid:62) : , ¯ E n D †(cid:62) : ,S ∗ P S ∗ (cid:13) (cid:13)(cid:13) , where
P S is the projection on R S and D † is the pseudo-inverse of D .
We have
[Equation]
We denote by G the matrix ( I − 1 L D (cid:62) D ) .
For z N with support S N and z ∗ with support S ∗ , we have with the induction in Theorem 2.
[Equation]
[Equation] [Equation]
where u Nl = − 1 L (cid:0) D l z (cid:62) N + ( D (cid:62) l z N − y l )
I (cid:1) and the other terms on ¯ S N and ¯ S ∗ are 0.
We can thus decompose their difference as the sum of two terms, one on the support S ∗ and one on this complement E N = S N \ S ∗
[Equation]
Recall that we assume S ∗ ⊂ S N .
Let’s study the terms separately on S ∗ and E N = S N \ S ∗ .
These two terms can be decompose again to constitute a double recursion system,
[Equation]
We deﬁne as P S N ,E N the operator which projects a vector from E N on ( S N , E N ) with zeros on
S N .
As S ∗ ∪ E N = S N , we get by combining these two expressions,
[Equation]
Taking the norm yields to the following inequality,
[Equation]
Denoting by µ N the smallest eigenvalue of D (cid:62) : ,S N D : ,S N − 1 , then (cid:13)(cid:13) G S N ,S N − 1 (cid:13)(cid:13) =
(1 − µ N L ) and we get that
[Equation]
S ∗ ( u
[Equation]
l
[Equation]
S
[Equation]
(cid:13)(cid:13)(cid:13) (cid:17) .
The back-propagation is initialized as J N − K l = 0 .
Therefore (cid:13)(cid:13)
J N − K l − J
[Equation]
l
[Equation]
= (cid:107) J ∗ l (cid:107) .
Moreover (cid:13)(cid:13)
u N − k l − u
[Equation]
≤
[Equation]
(cid:107)
[Equation]
− z
[Equation]
l (cid:13)(cid:13) .
Finally, (cid:13)(cid:13)(cid:13) ( u ∗ l ) E N − k − D (cid:62) : ,E N
[Equation]
k D †(cid:62) : ,S ∗ ( u ∗ l ) S ∗ (cid:13)(cid:13)(cid:13) can be rewritten with projection matrices P E N − k and P ¯ S ∗ to obtain
[Equation] [Equation]
[Equation] [Equation]
E N
[Equation]
D
[Equation]
S
[Equation]
L (cid:107) D
[Equation]
(cid:107) (cid:107) z
[Equation]
l (cid:107) .
Let B N − k
= (cid:13)(cid:13)(cid:13) P E N − k − D (cid:62) : ,
E N − k D †(cid:62) : ,S ∗ P S ∗ (cid:13)(cid:13)(cid:13) .
We have (cid:13)(cid:13)
J Nl − J ∗ l
[Equation]
≤
[Equation]
We now suppose that the support is reached at iteration N − s , with s ≥ K .
Therefore, ∀ n ∈ [ N − s, N ] S n = S ∗ .
Let ∆ n = F ( z n , D ) − F ( z ∗ , D ) + L 2 (cid:107) z n − z ∗ (cid:107) .
On the support, F is a µ -strongly convex function and the convergence rate of ( z N ) is
[Equation]
Thus
[Equation]
we obtain
[Equation]
Corollary
[Equation]
6 Let µ > 0 be the smallest eigenvalue of D (cid:62) : ,S ∗ D : ,S ∗ .
Let K ≤ N be the backpropagation depth and let ∆ N = F ( z N , D ) − F ( z ∗ , D )
+ L 2 (cid:107)
z
N − z ∗ (cid:107) .
Suppose that ∀ n ∈ [ N − K, N ]; S n ⊂ S ∗ .
Then, we have
[Equation]
(cid:16) 1 − µ L (cid:17) K − 1 (cid:107) D l (cid:107) 4∆ N − K L.
The term 2 L (cid:107) D l (cid:107) (1 − µL ) s (cid:80) K − 1 k = s − 1 (cid:81)
ki = s − 1 (1 − µ N − i L ) (cid:16) (cid:13)(cid:13)
z N − k l − z ∗ l (cid:13)(cid:13) + B N − k (cid:107) ( u ∗ l ) (cid:107) (cid:17) vanishes when the algorithm is initialized on the support.
Otherwise, it goes to 0 as s, K → N and N → ∞ because ∀ n > N − s, µ n = µ < 1 .
D I TERATIVE ALGORITHMS FOR SPARSE CODING RESOLUTION .
ISTA.
Algorithm to solve min z 1 2 (cid:107) y
[Equation]
(cid:107) 22 +
[Equation]
Algorithm y , D , λ , N z 0 = 0 , n = Compute the Lipschitz constant L of D (cid:62) D while n < N do u
[Equation]
1 ← [Equation] D
[Equation]
y ) z [Equation] 1 ← ST
[Equation]
u n +1 ) n ← n +end while
FISTA.
Algorithm to solve min z 12 (cid:107) y − Dz (cid:107)
+ λ (cid:107) z [Equation] Algorithm y , D ,[-λ , N z 
-]
[Equation] , N z [Equation] n
[Equation] [Equation] = Compute the Lipschitz constant L of D (cid:62) D while n < N do u
[Equation] n ← n +end while