U NDERSTANDING (cid:96) 4 - BASED D ICTIONARY L EARNING : I NTERPRETATION , S TABILITY , AND R OBUSTNESS
A BSTRACT
Recently, the (cid:96) 4 -norm maximization has been proposed to solve the sparse dictionary learning (SDL) problem.
The simple MSP (matching, stretching and projection) algorithm proposed by Zhai et al. (2019a) has shown to be surprisingly efﬁcient and effective.
This paper aims to better understand this algorithm from its strong geometric and statistical connections with the classic PCA and ICA, as well as their associated ﬁxed-point style algorithms.
Such connections provide a uniﬁed way of viewing problems that pursue principal , independent , or sparse components of high-dimensional data.
Our studies reveal additional good properties of (cid:96) 4 -maximization: not only is the MSP algorithm for sparse coding insensitive to small noise, but it is also robust to outliers and resilient to sparse corruptions.
We provide statistical justiﬁcation for such inherently nice properties.
To corroborate the theoretical analysis, we also provide extensive and compelling experimental evidence with both synthetic data and real images
I NTRODUCTION
The explosion of massive amounts of high-dimensional data has become the modern-day norm for a large number of scientiﬁc and engineering disciplines and hence presents a daunting challenge for both computation and learning.
Rising to this challenge, sparse dictionary learning (SDL) provides a potent framework in representation learning that exploits the blessing of dimensionality: real data tends to lie in or near some low-dimensional subspaces or manifolds, even though the ambient dimension is often extremely large (e.g. the number of raw pixels in an image).
More speciﬁcally,
SDL (Olshausen & Field (1997); Mairal et al.
; Spielman et al.
; Sun et al.
; Bai et al.
; Qu et al.
concerns the problem of learning a compact, sparse representation from raw, unlabelled data: given a data matrix Y =
[ y 1 , y 2 , . . .
, y p ] ∈ R n × p that contains p n -dimensional samples, one aims to ﬁnd a linear transformation (i.e. a dictionary ) D ∈ R n × m and an associated maximally sparse representation X =
[ x 1 , x 2 , . . .
, x p ] ∈ R m × p that satisﬁes
[Equation]
As the data matrix Y can represent a variety of signals (e.g. images, audios, languages, and genetics etc) in practical applications, SDL provides a versatile structure-seeking formulation that has found widespread applications in computational neuroscience, image processing, computer vision, and machine learning at large (Olshausen & Field, 1996; 1997; Argyriou et al., 2008; Ranzato et al., 2007; Elad & Aharon, 2006;
Wright et al., 2008; Yang et al., 2010; Zhang et al., 2013; Mairal et al., 2014; Zhang et al., 2014; 2019).
Related Works.
Motivated by this practical signiﬁcance, there has been a growing surge of interest recently (e.g. Rambhatla et al.
; Bai et al.
Gilboa et al.
; Nguyen et al.
Chatterji & Bartlett (2017); Mensch et al.
(2016)) that aims to tackle SDL.
In attempts to recover the sparse signals X , these existing work adopt an (cid:96) 0 - or (cid:96) 1 -penalty function to promote the underlying sparsity and give various optimization algorithms for the resulting objectives (some of those are heuristics while a few others have theoretical convergence guarantees).
Although these penalty functions are indeed sparsity-promoting, the resulting optimization problems must be solved one row at a time, hence resulting as many optimization problems as the ambient dimension n .
Consequently, (cid:96) 0 - or (cid:96) 1 -based objectives result only in local methods (i.e. cannot yield the entire solution at once) and hence entail prohibitive computational burden.
Another prominent approach in SDL is Sumof-Squares (SOS), proposed by and articulated in a series of recent work (Barak et al.
; Ma et al.
; Schramm & Steurer (2017)).
The key idea there is to utilize the properties of higher order SOS polynomials to correctly recover one column of the dictionary at a time, for which there are m columns in total.
However, the computational complexity of these recovery methods are quasi-polynomial, thus again resulting in large computational expense.
Very recently, in the complete dictionary learning 1 setting, a novel global approach has been suggested in Zhai et al.
(2019a;b) that presents a formulation which can efﬁciently recover the sparse signal matrix X once and for all.
In particular, Zhai et al.
(2019b) has shown that if the generative model for Y = D
o X o ∈ R n × p satisﬁes that D o ∈ O ( n ; R ) is orthonormal and X o ∈ R n × p is
Bernoulli-Gaussian sparse, 2 then maximizing the (cid:96) 4 -norm 3 of AY over O ( n ; R ) :
[Equation]
subject to A ∈ O ( n ; R ) ( or AA ∗ =
I ) , (2) is able to ﬁnd the ground truth dictionary D o up to an arbitrary signed permutation.
Moreover, Zhai et al.
(2019b) has proposed the simple “ Matching, Stretching, and Projection ” (MSP) algorithm, which is shown to be experimentally efﬁcient and effective, for solving the program in equation 2:
[Equation]
where U t V ∗ t are from the singular value decomposition: U t Σ t V ∗ t =
SVD [( A t Y ) ◦
3 Y ∗ ] .
In this paper, we here give an alternative (arguably simpler and more revealing) derivation of the
MSP algorithm (3).
Consider the Lagrangian formulation of the constrained optimization problem given in equation 2.
The necessary condition of critical points, ∇ A 14 (cid:107) AY (cid:107)
= ∇ A (cid:104) Λ , AA ∗ − I (cid:105) for some Lagrangian multipliers Λ , implies:
[Equation]
As the optimization is over the orthogonal group O ( n ; R ) , restricting the condition in equationonto the orthogonal group yields a necessary condition for any critical point A :
[Equation]
Hence the critical point A can be viewed as a “ ﬁxed point ” of the map: P O ( n ; R ) (cid:2) (( · ) Y ) ◦ 3 Y ∗ (cid:3) from
O ( n ; R ) to itself.
The MSP algorithm in equation 3 is to ﬁnd the ﬁxed point(s) of this map.
Notice that the orthonormal constraint A ∈ O ( n ; R ) in equation 2 can be viewed as enforcing the orthogonality of n unit vectors simultaneously.
So, more ﬂexibly and generally, one may choose to compute any k , for 1 ≤ k ≤ n , leading orthonormal bases of D o by solving the program:
[Equation]
subject
[Equation]
St
[Equation]
(6) where St ( n, k ; R ) is the Stiefel manifold .
The orthogonal group O ( n ; R ) and the unit sphere S n −can be viewed as two special cases of the Stiefel manifold St ( n, k ; R ) , with k = n and k = 1 , respectively.
In some speciﬁc tasks such as dictionary learning and blind deconvolution, optimization over the unit sphere has been widely practiced, such as in Sun et al. (2015); Bai et al.
; Zhang et al.
; Kuo et al.
The more general setting of maximizing a convex function over any compact set also has been studied by Journ´ee et al.
(2010) in the context of sparse PCA, which has provided convergence guarantees for this class of programs.
Our Contributions.
Our contributions are twofold.
First, by taking a suitable analytical angle, we reveal novel geometric and statistical connections between PCA, ICA and the (cid:96) 4 -norm maximization based SDL.
We then show that algorithm-wise, the ﬁxed-point type MSP algorithm for (cid:96) 4 -norm maximization has the same nature as the classic power-iteration method for PCA (Jolliffe, 2011) and the FastICA algorithm for ICA (Hyv¨arinen & Oja, 1997).
This interpretation gives a uniﬁed view for problems that pursue principal , independent , or sparse components from high-dimensional data and enriches our understanding of low-dimensional structure recovery frameworks, classical and new, at both formulation and algorithmic fronts.
Second, and more importantly from a practical perspective, we examine how MSP performs under a variety of more realistic conditions, when the measurements Y could be contaminated with noise, outliers, or sparse corruptions.
We show that, similar to PCA, (cid:96) 4 -norm maximization and the MSP algorithm are inherently stable to small noise.
Somewhat surprisingly though, unlike PCA, the MSP algorithm is further robust to outliers and resilient to sparse gross errors!
We provide characterizations of these desirable properties of MSP.
The claims are further corroborated with extensive experiments on both synthetic data and real images.
Taken as a whole, our results contribute to the broad landscape of dictionary learning by afﬁrming that (cid:96) 4 -maximization based SDL and the corresponding global algorithm MSP provide a valuable toolkit to the existing literature.
SDL VERSUS PCA AND ICA 2. P URSUIT OF P RINCIPAL , I NDEPENDENT , OR S PARSE C OMPONENTS
Relation with the Geometric Interpretation of PCA.
For a data matrix Y ∈ R n × p , Principal
Component Analysis (PCA), aiming to ﬁnd the top (top k ) left singular vector (vectors) of Y ,
[Equation]
subject to W ∈ St ( n, k ; R ) ,
[Equation]
) can be understood as ﬁnding a direction (a k -dimensional subspace) in row ( Y ) in which Y has the largest (cid:96) 2 -norm (Frobenius norm).
For instance, ﬁnding the direction with the largest (cid:96) 2 -norm over the unit sphere can be viewed as calculating the spectral norm (or the largest singular value), of matrix Y .
In comparison, we may view equation 6:
[Equation]
subject to W ∈ St ( n, k ; R ) as ﬁnding a direction, or a k -dimensional subspace, in row ( Y ) where the projection of Y has the largest (cid:96) 4 -norm.
For instance, ﬁnding the direction with the largest (cid:96) 4 -norm over the unit sphere (equation 6) can be viewed as calculating the induced (cid:107)·(cid:107) 2 , 4 norm of matrix Y : (cid:107) Y (cid:107) 2 , 4 .
= max a ∈ S n − 1 (cid:107) a ∗ Y (cid:107) 4 .
Relation with the Statistical Interpretation of PCA.
View each column y j , j ∈ [ p ] of data matrix Y ∈ R n × p as an n dimensional random vector that is i.i.d.
drawn from a distribution of random variable y and let Y c denote the centered Y : Y c .
=
Y [ I − 1 p 11 ∗ ] , where 1 ∈ R p is a vector of all 1 ’s. Then, ﬁnding the top k principal components of Y c : max W ∈ St ( n,k ; R ) 12 (cid:107) W ∗ Y c (cid:107)is to ﬁnd k uncorrelated projections of y ∈ R n that has the top k sample variance (i.e. 2 nd order moment)
(Jolliffe, 2011; Helwig, 2017).
Similar to PCA, the (cid:96) 4 -norm maximization of centered data matrix Y c : max W ∈ St ( n,k ; R ) 14 (cid:107) W ∗ Y c (cid:107) 44 can be viewed as ﬁnding k uncorrelated projections of y that have the top k sample 4 th order moment, whose statistical meaning is better revealed below.
Relation with ICA and Nonnormality.
The (cid:96) 4 -norm maximization over the Stiefel manifold is strongly related to ﬁnding the maximal or minimal kurtosis in Independent Component Analysis (ICA)
(Hyv¨arinen & Oja, 1997; 2000):
In order to identify one component of a given random vector y ∈ R n , ICA aims to ﬁnd a unit vector (a direction) w ∈ S n − 1 that maximizes or minimizes the kurtosis of w ∗ y , deﬁned as:
[Equation]
Kurtosis is widely used for evaluating the nonnormality of a random variable; see DeCarlo (1997);
Hyv¨arinen & Oja (1997; 2000).
According to Huber (1985), the nonnormality of data carries “abnormal” hence interesting information in real data for many applications (e.g. Lee et al.
; Cain et al.
Thus, extracting the 4 th order moment helps understand such statistics of real datasets (Hyv¨arinen et al., 2009) and even their topology (Carlsson, 2009).
One may also ﬁnd that the (cid:96) 4maximization based dictionary learning formulation is similar to maximizing kurtosis (equation 8) with spherical constraint w ∈ S n − 1 , in fact, these two formulations are exactly the same if one only wants to ﬁnd one column of the dictionary.
Intuitively, such coincidence occurs due to the fact that maximizing (cid:96) 4 -norm or kurtosis have the same effect— they both promote the “spikiness” (Zhang et al.
; Li & Bresler (2018)) (or “peak” (DeCarlo, 1997)) of a distribution.
More rigorous analysis regarding the similarity between (cid:96) 4 -based dictionary learning and ICA is beyond the scope of this paper and we leave them for future research.
2. F IXED -P OINT S TYLE A LGORITHMS
In optimization, the (cid:96) 4 -norm maximization in equation 6 over the Stiefel manifold St ( k, n ; R ) is a special type of nonconvex optimization problem – convex maximization over a compact set.
Although Journ´ee et al.
; Zhai et al.
(2019b) have shown that the MSP algorithm is guaranteed to ﬁnd critical points, the experiments in Zhai et al.
(2019b) suggest that the MSP algorithm ﬁnds global maxima of the (cid:96) 4 -norm efﬁciently and effectively.
For better understanding, in this section we illustrate some striking similarities between the MSP algorithm and the “power-iteration” type algorithms for solving PCA as well as ICA.
Fixed-point Perspective of Power Iteration.
For a general data matrix Y ∈ R n × p , ﬁnding the top singular value of Y is equivalent to solving the following optimization problem:
[Equation]
For this constrained optimization, the Lagrangian multiplier method gives the necessary condition: ∇ w ϕ ( w )
=
Y Y ∗ w = λ w , similar to equation 4.
If we restrict this condition onto the sphere, we obtain the ﬁxed point condition w = P S n − 1 [ ∇ w ϕ ( w )] .
The classic power-iteration method
[Equation]
is precisely computing this ﬁxed point, which is arguably the most efﬁcient and widely used algorithm to solve equation 9, for PCA (or computing SVD of Y ).
Fixed-point Perspective of FastICA.
In order to maximize (or minimize) the kurtosis over S n − 1 ,
[Equation]
subject to
[Equation]
Hyv ¨ arinen & Oja (1997) has proposed the following ﬁxed-point type iteration:
[Equation]
which enjoys cubic (at least quadratic) rate of convergence, under the ICA model assumption.
Fixed-point Perspective of MSP.
For the (cid:96) 4 -norm maximization program:
[Equation]
subject to W
[Equation]
St ( n, k
[Equation]
R )
[Equation]
through a similar derivation to that in Section 1, one can show that the MSP iteration in equationfor the orthogonal group generalizes to the Stiefel manifold case as:
[Equation]
where U t Σ t V ∗ t =
SVD [ Y ( Y ∗ W t ) ◦ 3 ] .
The above iteration has the same nature as the power iteration in equation 10 and equation 12, since they all solve a ﬁxed-point type problem, by projecting gradient of the objective function ∇ ϕ ( · ) , ∇ φ ( · ) , ∇ ψ ( · ) onto the constraint manifold S n − 1 and
St ( n, k ; R ) , respectively.
Table 1 summarizes these striking similarities.
[Table]
W
[Table]
W
[Table]
S TABILITY AND R OBUSTNESS OF (cid:96) 4 -N ORM M AXIMIZATION
Even though the MSP algorithm for (cid:96) 4 -norm maximization is similar to power-iteration for PCA, in real applications, PCA often requires modiﬁcations to improve its robustness (Cand`es et al., 2011;
Xu et al., 2010; 2012).
In this section, we want to examine the stability and robustness of the (cid:96) 4maximization for different types of imperfect measurement models: small noise, outliers, and sparse corruptions of large magnitude.
3. D IFFERENT M ODELS FOR I MPERFECT M EASUREMENTS
We adopt the same Bernoulli-Gaussian model as in prior works (Spielman et al., 2012; Sun et al., 2015; Bai et al., 2018; Zhai et al., 2019b) to test the stability and robustness of the (cid:96) 4 -maximization framework.
Assume our clean observation matrix Y ∈ R n × p is produced by the product of a ground truth orthogonal dictionary D o and a Bernoulli-Gaussian matrix X o ∈ R n × p :
Y = D
[Equation]
X o ,
[Equation]
Now let us assume we only observe different types of imperfect measurements of Y :
Noisy Measurements: Y N :=
Y + G , where G ∈ R n × p is matrix that satisﬁes g i,j ∼ iid N (0 , η 2 ) and η > 0 controls the variance of the noise.
Measurements with Outliers: Y O :=
[ Y , G (cid:48) ] , where Y O contains extra columns ( G (cid:
) ∈
R n × τp )generated from an independent Gaussian process g (cid:48)
i,j ∼ iid N (0 , 1) .
Here, τ controls the portion of the outliers w.r.t.
the clean data size p .
Measurements with Sparse Corruptions: Y C :
= Y + σ B ◦ S , where σ > 0 controls the scale of corrupting entries, 7 B ∈ R n × p is a Bernoulli matrix so b i,j ∼ iid Ber ( β ) with β ∈ (0 , 1) controlling the ratio of the sparse corruptions, and S ∈ R n × p has entries s i,j drawn i.i.d.
from a Rademacher distribution:
[Equation]
with probability
[Equation]
with
[Equation]
3. S TATISTICAL A NALYSIS AND J USTIFICATION
The analysis for the stability and robustness of the (cid:96) 4 -norm maximization follows similar statistical analysis techniques in Zhai et al.
(2019b) to show that the global maximum of
[Equation]
subject to W ∈ O ( n
[Equation]
R ) (16) satisﬁes W ∗ (cid:63) D o ∈ SP ( n ) .
8 Note here Y (cid:5) denotes different imperfect measurements:
noisy ( Y N ), with outliers ( Y O ), and with sparse corruptions ( Y C ).
Below we provide the expectation and concentration results for (cid:107) W ∗ Y (cid:5) (cid:107) 44 over the data distribution.
We show that E (cid:107) W ∗ Y (cid:5) (cid:107) 44 is largely determined by (cid:107) W D o (cid:107) 44 , a quantity that indicates a “distance” from W ∗ D o to SP ( n ) .
As shown in Lemma 2.3 and Lemma 2.4 in Zhai et al. (2019b), the only global maximizers of (cid:107) W ∗ D o (cid:107) 4 4 are signed permutation matrices, and W ∗ D
o converges to a signed permutation matrix as (cid:107) W ∗ D
o (cid:107)reaches its global maximum.
Proposition 3.1 (Expectation of Objective with Small Noise) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Gaussian matrix G ∈ R n × p , g i,j ∼ iid N (0 , η 2 ) independent of X o , let Y N =
Y + G denote the data with noise.
Then the expectation ofnp (cid:107) W
[Equation]
Y N (cid:107) 44 satisﬁes:
[Equation]
where C θ,η is a constant which depends on θ and η .
Proof See Appendix A.1.
Theorem 3.2 (Concentration of Objective with Small Noise) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Gaussian matrix G ∈ R n × p , g i,j ∼ iid N (0 , η 2 ) independent of X o , let Y N =
Y + G denote the input with noise, then:
[Equation]
when p = Ω (cid:0) (1 + η 2 ) 4 n 2 ln
[Equation]
/δ 2 (cid:1) .
Proof See Appendix A.2.
Proposition 3.3 (Expectation of Objective with Outliers) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid
BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Gaussian matrix G (cid:48) ∈ R n × τp , g (cid:48)
i,j ∼ iid N (0 , 1) independent of X o , let Y O =
[ Y , G (cid:48) ] denote the data with outliers G (cid:48) .
Then the expectation of 1 np
[Equation]
[Equation]
where C θ,τ is a constant depends on θ, τ .
Proof See Appendix A.
Theorem 3.4 (Concentration of Objective with Outliers) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid
BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Gaussian matrix G (cid:48) ∈ R n × τp , g (cid:48)
i,j ∼ iid N (0 , 1) independent of X o , let Y O =
[ Y , G (cid:48) ] denote the input with outlier G (cid:48) , then:
[Equation]
when p =
Ω( τ 2 n 2 ln n/δ 2 ) .
Proof See Appendix A.4.
In the above results, Proposition 3.1 and 3.3 reveal that both normalized 1 np E (cid:107) W ∗ Y N (cid:107) 44 , 1 np E (cid:107) W ∗ Y O (cid:107) 44 are only determined by (cid:107) W ∗ D o (cid:107) 44 .
Moreover, as shown in Theorem 3.2 and 3.4, when p is large enough ( p =
Ω((1 + η 2 ) 4 n 2 ln n ) , Ω( τ 2 n 2 ln n ) respectively), both 1 np (cid:107) W ∗ Y N (cid:107) 44 , 1 np (cid:107) W ∗ Y O (cid:107) 4 4 concentrate around their expectation with high probability.
Therefore, the (cid:96) 4 -norm maximization formulation (cid:107) W ∗ Y (cid:5) (cid:107) 44 for dictionary learning is insensitive to dense Gaussian noise and robust to Gaussian outliers.
Proposition 3.5 (Expectation of Objective with Sparse Corruptions) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Bernoulli matrix B ∈ R n × p , b i,j ∼ iid
Ber ( β ) independent of X o , let Y C
= Y + σ B ◦ S denote the data with sparse corruptions, and
S ∈ R n × p is deﬁned in equation 15.
Then the expectation of 1 np (cid:107) W ∗ Y C (cid:107) 44 satisﬁes:
[Equation]
where C θ,σ,β is a constant depending on θ, σ and β .
Proof See Appendix A.5.
Theorem 3.6 (Concentration of Objective with Sparse Corruptions) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Bernoulli matrix B ∈ R n × p , b i,j ∼ iid
Ber ( β ) independent of X o , let Y C
= Y + σ B ◦ S denote the input with sparse corruptions, and
S ∈ R n × p is deﬁned in equation 15, then:
[Equation]
when p = Ω (cid:0) σ 8 βn 2 ln n/δ 2 (cid:1) .
Proof See Appendix A.6.
Unlike the cases with noise and outliers, Proposition 3.5 and
Theorem 3.6 indicate that 1 np E (cid:107) W ∗ Y C (cid:107) 44 depends on both (cid:107) W ∗ D o (cid:107) 44 and (cid:107) W (cid:107) 44 ;
when p = Ω( σ 8 βn 2 ln n ) , the objective 1 np (cid:107) W ∗ Y C (cid:107) 44 concentrates around this expectation with high probability.
Nevertheless, when the magnitude of σ 4 β (1 − 3 β ) is signiﬁcantly smaller than 3 θ (1 − θ ) , the landscape of the objective (cid:107) W ∗ Y C (cid:107) 44 would largely be determined by (cid:107) W ∗ D o (cid:107) 44 only.
As shown in Figure 1, this is indeed the case whenever: (a) the sparsity level θ of ground truth signal X o , is “reasonably” small (neither diminishing to 0 nor larger than 0.5);(b) β , the sparsity level of the corruption, is small (smaller than 0.5); (c) σ , the magnitude of the sparse errors, is not signiﬁcantly larger than the intrinsic variance of the sparse signal (the intrinsic variance of the sparse signal Bernoulli-Gaussian model is 1).
Therefore, besides the insensitivity to small noise and robustness to outliers, the (cid:96) 4 -maximization based dictionary learning also shows resilience to sparse corruptions under reasonable conditions.
S IMULATIONS AND E XPERIMENTS 4. Q UANTITATIVE E VALUATION : S IMULATIONS ON S YNTHETIC D ATA
Single Trial of MSP.
In this simulation, we run the MSP algorithm from equation 3, using the imperfect measurements Y (cid:5) of different models ( Y N , Y O , Y C ).
As shown in Figure 2, the normalized value of (cid:107) W ∗ D o (cid:107) 44 /n reaches global maximum with all types of inputs when varying the level of noise, outliers, and sparse corruptions.
Moreover, as the scale of imperfect measurements increase, Figure 2 shows that (a) the iterations for convergence increases and (b) the ﬁnal objective value (cid:107) W ∗ D o (cid:107) 4 4 decreases almost negligibly.
This numerical experiment suggests that the MSP algorithm is able to identify the ground truth orthogonal transformation D o despite different types of imperfect measurements.
Phase Transition.
Next, we conduct extensive simulations to study the relation between recovery accuracy and sample size p .
We run the experiments by increasing the sample size p w.r.t.
the scale of imperfect measurements η, τ, β , respectively.
As shown in Figure 3, the MSP algorithm (3)
[Figure]
demonstrates a clear phase transition behavior w.r.t.
noise, outliers, and sparse corruptions.
Such phenomena suggest that the algorithm is inherently stable and robust to certain amounts of noise, outliers, and sparse corruptions.
The results corroborate our concentration results in Theorem 3.2, 3.4 and 3.6—a larger sample size p increases the accuracy and robustness of the MSP algorithm (3) for all types of nuisances.
[Figure]
Comparison with Prior Arts.
We also compare the MSP algorithm (Zhai et al., 2019b) with previous complete dictionary learning algorithms: Subgradient (SG) (Bai et al., 2018) and Riemannian
Trust Region (RTR) methods (Sun et al., 2015).
While SG and RTR demonstrate slightly better accuracy in some cases (e.g. small Gaussian noise and Gaussian outliers), both algorithms appear unstable to sparse corruptions.
Meanwhile, the MSP algorithm is stable to all imperfect measurement models mentioned in this paper and runs signiﬁcantly faster than the others
[Table]
4. Q UALITATIVE E VALUATION : E XPERIMENTS ON R EAL I MAGES AND P ATCHES
Besides simulations, we also conduct extensive experiments to verify the stability and robustness of the MSP algorithm with real imagery data, at both image level and patch level.
Throughout these experiments, rather than visualize all bases, we routinely show the top bases learned—heuristically, the top bases are those with the largest coefﬁcients (here, in terms of (cid:96) 1 -norm).
Image Level.
At the image level, we ﬁrst vectorize all 60,000 images in the MNIST dataset (LeCun et al., 1998) into a single matrix Y ∈ R 784 × 60 , 000 , then create imperfect measurements based on models speciﬁed in Section 3: Y N (MNIST with noise), Y O (MNIST with outliers), Y C
(MNIST with sparse corruptions).
We run the MSP algorithm 3 with Y , Y N , Y C , Y O and compare the learned bases.
Figure 4(a), (c), (e), and (g) show examples of Y , Y N , Y O , and Y C , and Figure 4(b), (d), (f), and (h) show top 10 bases learned from Y , Y N , Y C , Y O , respectively.
Despite that we use different types of imperfect measurements of MNIST, the top bases learned from MSP algorithm 3 are very much the same.
This result corroborates our analysis: the (cid:96) 4 -maximization and the MSP algorithm is inherently insensitive to noise, robust to outliers, and resilient to sparse corruptions.
[Figure]
Patch Level.
A classic application of dictionary learning involves learning sparse representations of image patches (Elad & Aharon, 2006; Mairal et al., 2007).
In this section, we extend the experiments of Zhai et al.
(2019b) to learn patches from gray scale and color images.
First, we construct a data matrix Y by vectorizing each 8 × 8 patch from the 512 × 512 gray scale image, “Barbara” (see
Figure 5).
We then run the MSP algorithm with 100 iterations on both Y and a noisy version Y N , and the learned top bases are visualized in Figure 5.
[Figure]
Analogously, we apply the same scheme to a 256 × 256 color image, “Duck” (see Figure 6), converting each 8 × 8 × 3 patch into a column vector (in R 192 ) of Y .
Notice this forces the algorithm to learn bases involving all three channels simultaneously, rather than one at a time.
After running the MSP algorithm for 100 iterations, we visualize the top bases learned from both Y and corresponding Y N in Figure 6.
We next consider the problem of learning a “global dictionary” (Mairal et al., 2007) for patches from many different images.
To construct our data matrix Y , we randomly sample 100,000 8 × 8 ×
[Figure] [Figure]
patches from the CIFAR-10 data-set (Krizhevsky et al., 2009).
A noisy data matrix Y N , is then generated by adding Gaussian noise.
Again, we apply the MSP algorithm with 200 iterations to learn 192 bases and visualize the results in Figure 7.
We leave the experiments of CIFAR-10 with outliers and sparse corruptions in the Appendix due to limited space.
In each of these experiments, the top bases in the learned dictionary remain relatively unchanged with the addition of noise.
To quantify this similarity, we take the top bases from the noisy dictionary and ﬁnd the closest top clean base for each.
If the bases are nearly identical, then the inner product of each of these pairs should be close to 1.
Table 3 reports the statistics.
[Table]
C ONCLUSION AND D ISCUSSIONS
In this paper, we ﬁnd the (cid:96) 4 -norm maximization based dictionary learning and corresponding MSP algorithm introduced by Zhai et al.
(2019b) have strong geometric and statistical connections to classic data analysis methods PCA and ICA.
These connections seem to be the reason why they all admit similar simple and efﬁcient algorithms.
Empirically, we have observed that (cid:96) 4 -norm maximization is surprisingly insensitive to noise, robust to outliers, and resilient to sparse corruptions.
Moreover, such empirical observations corroborate our concentration analysis—larger data samples ( p ) improve the stability and robustness of the (cid:96)
4 -maximization based dictionary learning.
From experiments on real images, we observed that top bases learned are rather stable but tail bases can be less stable (see Figure 10 in Appendix C).
We conjecture this phenomenon happens due to the fact that real images generally do not follow the uniformly sparse Bernoulli Gaussian model (of equation 14).
Generalizing dictionary learning to non-uniformly sparsely generated data would be a good topic for future study.
Finally, note that Bai et al.
(2018) has established subgradient descent can ﬁnd global optimal solutions for dictionary learning, an interesting theoretical guarantee since dictionary learning problems do not satisfy the standard structural (i.e. generalized convexity) assumptions that guarantee global convergence (Ben-Tal & Nemirovski, 2001;
Zhou et al., 2017a;
Nesterov, 2013; Zhou et al., 2017b; Bubeck et al., 2015; Zhou et al., 2017c; Bertsekas & Scientiﬁc, 2015; Zhou et al., 2018).
It would be desirable, although highly challenging, to establish similar global convergence guarantees for our MSP algorithm.
We leave that for future work.
A P ROOF OF S ECTIONA. P ROOF OF P ROPOSITION 3.
Claim A.1 (Expectation of Objective with Small Noise) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid
BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Gaussian matrix G ∈ R n × p , g i,j ∼ iid N (0 , η 2 ) independent of X o , let Y N =
Y + G denote the data with noise.
Then the expectation of 1 np (cid:107) W ∗ Y N (cid:107) 44 satisﬁes
[Equation]
Proof Let W ∗ D o = M ∈ O ( n ; R ) , notice that the orthogonal transformation ( W ∗ G ) of a Gaussian matrix ( G ) is still a Gaussian matrix and satisﬁes { W ∗ G } i,j ∼ N (0 , 1) , and it is independent of Y ( X o ) .
We abuse the notation a bit let G
=
W ∗ G in the following calculation, since W ∗ G is also a Gaussian matrix independent of X o
and it will not affect the ﬁnal result.
[Equation]
 
E
[Equation]
o
[Equation]
G (cid:32) n (cid:88) k
= m i,k x k,j (cid:33)+ 6 E X o , G   n
i,j (cid:32) n (cid:88)
k
= m i,k x k,j (cid:33) 2   + E G g 4 i,j   = p (cid:88)
j = n (cid:88) i =
 
E X o , G (cid:32) n (cid:88) k
= m i,k x k,j (cid:33)+ 6 η 2 E X o (cid:32) n (cid:88) k
= m i,k x k,j (cid:33) 2   + 3 npη= p (cid:88)
j
[Equation]
i
[Equation]

[Equation]
o
[Equation]
i,
[Equation] [Equation]
(cid:33)
[Equation]
  +
[Equation]
2 +
[Equation]
npη
[Equation]
npη
[Equation] [Equation]
24) therefore,
[Equation]
which completes the proof.
A. P ROOF OF T HEOREM 3.
Claim A.2 (Concentration of Objective with Small Noise) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Gaussian matrix G ∈ R n × p , g i,j ∼ iid N (0 , η 2 ) independent of X o , let Y N =
Y + G denote the input with noise, then:
[Equation]
26) when p =
Ω (cid:0) (1 + η 2 ) 4 n 2 ln n/δ 2 (cid:1) .
Proof According to Proposition 3.1, we know that
[Equation]
Moreover, since Y N =
[ y 1 + g 1 , y 2 + g 2 , . . .
, y p + g p ] , whose columns are independent.
Let
[Equation]
note that we abuse the notation a bit by assuming D ∗ o G = G , since D ∗ o G is also a Gaussian matrix independent of X
o
and it will not affect the ﬁnal result.
Deﬁne function
[Equation]
Next, we will check assumption 1, 2, and 3 then apply Lemma B.4.
Assumption 1: µ ( n, p ) .
Since z i,j = x i,j + g i,j , we know that with probability θ , z ∼ N (0 , 1+ η 2 ) and with probability 1 − θ , z ∼ N (0 , 1) .
Hence,
[Equation]
Speciﬁcally, we set B = ln p , which yields
[Equation]
Assumption 2: Lipschitz Constant L f .
By Proposition 3.1, ∀ W ∈ O ( n ; R ) , we know that
[Equation]
Hence, ∀ W 1 , W 2 ∈ O ( n ; R ) , we have
[Equation]
moreover,
[Equation] [Equation]
Hence, we know that
[Equation]
35) which yields
[Equation]
Assumption 2: Lipschitz Constant ¯ L f .
Since f ¯ z ( W ) = (cid:107) W ∗ ¯ z (cid:107)
[Equation]
we have
[Equation]
(cid:12) (cid:88) i (cid:8)(cid:2)
( W ∗ 1 ¯ z )
i − ( W ∗ 2 ¯ z ) 2 i (cid:3) (cid:2) ( W ∗ 1 ¯ z ) 2 i + ( W ∗ 2 ¯ z ) 2 i (cid:3)(cid:9)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12) ≤ (cid:40)(cid:88) i (cid:2) ( W ∗ 1 ¯ z )
i − ( W ∗ 2 ¯ z ) 2 i (cid:3)(cid:41) 1 / 2 (cid:40)(cid:88) i (cid:2) ( W ∗ 1 ¯ z ) 2 i
+ ( W ∗ 2 ¯ z ) 2 i (cid:3)(cid:41) 1 /= (cid:13)(cid:13) ( W ∗ 1 ¯ z ) ◦ 2 − ( W ∗ 2 ¯ z ) ◦ 2 (cid:13)(cid:13)(cid:124) (cid:123)(cid:122) (cid:125)
Γ(cid:13)(cid:13) ( W ∗ 1 ¯ z ) ◦
+ ( W ∗ 2 ¯ z ) ◦ 2 (cid:13)(cid:13)(cid:124) (cid:123)(cid:122)
(cid:125) Γ
[Equation]
For Γ 1 , we have
[Equation]
For Γ 2 , we have
[Equation]
Hence, we know that
[Equation]
which yields
[Equation]
Assumption 3: Upper Bound R 1 .
Notice that ∀ W ∈ O ( n ; R ) ,
[Equation]
Assumption 3: Upper Bound R 2 .
Assume the support for x is S , so we know that know that ∀ i ∈ [ n ] ,
[Equation]
i
[Equation] [Equation]
where v ∼ N ( 0 , I ) is a Gaussian vector.
Let P S : R n (cid:55)→ R n be a projection that satisﬁes ∀ q ∈ R n (cid:0) P S q
[Equation]
i = (cid:40)(cid:112)
[Equation]
if i ∈
[Equation]
q
[Equation]
otherwise .
Moreover, ∀ W ∈ O ( n ; R ) , let w i denote the i th column vector of W , hence
E [
[Equation]
=
E  
[Equation]
i
[Equation]
n (cid:88)
j = (cid:104)
[Equation]

 ≤ E S n (cid:88) i = n (cid:88)
j =
E (cid:104) (cid:104) P S w i , v (cid:105)
[Equation]
S n
[Equation]
Notice that v ∼ N ( 0 , I ) , therefore ∀ i ∈ [ n ] , we have
E (cid:104) P S w i , v (cid:105) 8 = 105 (cid:107) P S w i (cid:107) 82 , (46) hence
E S n (cid:88) i = n (cid:88)
j =
[Equation]
(cid:105)
[Equation]
E S n (cid:88) i = n (cid:88)
j = (cid:107)
[Equation]
(cid:107) 42 (cid:107)
[Equation]
(cid:107)= n (cid:88) i = n (cid:88)
j = (cid:88) k
[Equation]
E
[Equation]
(cid:34)
[Equation]
(cid:35) ≤ 105(1 + η
[Equation]
)n
[Equation]
i =
[Equation]
n
[Equation]
j =
[Equation]
k
[Equation] [Equation]
[Equation]
Therefore, we can conclude that
[Equation]
] ≤ 105(1 + η 2 ) 4 = R 2 .
Applying Lemma B.4 for Concentration.
Now we apply Lemma B.4 with
[Equation] [Equation]
we have
[Equation]
<p
[Equation]
52) for a constant C > 10 4 ,
when p =
Ω (cid:0) (1 + η 2 ) 4 n 2 ln n/δ 2 (cid:1) .
A. P ROOF OF P ROPOSITION 3.
Claim A.3 (Expectation of Objective with Outliers) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid
BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Gaussian matrix G (cid:48) ∈ R n × τp , g (cid:48)
i,j ∼ iid N (0 , 1) independent of X o , let Y O =
[ Y , G (cid:48) ] denote the data with outliers G (cid:48) .
Then the expectation of 1 np (cid:107) W ∗ Y O (cid:107) 44 satisﬁes:
[Equation]
Proof Notice that
[Equation]
(54) and
E
[Equation] [Equation]
Moreover, the orthogonal rotation ( W ∗ G (cid:48) ) of a standard Gaussian matrix G (cid:48) is also a standard
Gaussian matrix, therefore,
[Equation]
Hence,
[Equation]
which completes the proof.
A. P ROOF OF T HEOREM 3.
Claim A.4 (Concentration of Objective with Outliers) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid
BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Gaussian matrix G (cid:48) ∈ R n × τp , g (cid:48)
i,j ∼ iid N (0 , 1) independent of X o , let Y O =
[ Y , G (cid:48) ] denote the input with outlier G (cid:48) , then:
[Equation]
when p =
Ω( τ 2 n 2 ln n/δ 2 ) .
Proof Notice that
[Equation]
Apply Lemma B.5 (substitute δ with δ 2 ) to Γ 1 , we know that
[Equation]
for two constants c 1 ≥ 4 × 10 4 , c 2 > 13 ,
[Equation]
and the last inequality holds when p =
Ω
( θn 2 ln n/δ 2 ) .
Moreover, apply Lemma B.5 again (let θ = 1 , Y = G (cid:
) ∈ R n × τp and substitute δ with δ 2 τ ), we have Γ
[Equation]
<
exp
[Equation]
for two constants c 1 ≥ 4 × 10 4 , c 2 > 13 ,
[Equation]
and the last inequality holds when p =
Ω( τ 2 n 2 ln n/δ 2 ) .
Combine θ ≤ 1 and τ can be larger than
[Equation]
we conclude that
[Equation]
(cid:1)
A. P ROOF OF P ROPOSITION 3.
Claim A.5 (Expectation of Objective with Sparse Corruptions) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Bernoulli matrix B ∈ R n × p , b i,j ∼ iid Ber ( β ) independent of X o , let Y C
= Y + σ B ◦ S denote the data with sparse corruptions, and S ∈ R n × p is deﬁned in equation 15.
Then the expectation of 1 np (cid:107) W
[Equation]
(cid:107) 44 satisﬁes:
[Equation]
Proof Let W ∗ D o = M ∈ O ( n ; R ) , notice that
[Equation]
hence
[Equation]
Moreover,
• Γ
[Equation]
E X
[Equation]
B
[Equation]
Γ
[Equation]
E X
o
[Equation]
• Γ 2 :
[Equation]
• Γ 3 :
[Equation] [Equation]
Substitute E Γ 1 , E Γ 2 , E Γ 3 , E Γ 4 , E Γ 5 back to equation 65, yields
[Equation]
A. P ROOF OF T HEOREM 3.
Claim A.6 (Concentration of Objective with Sparse Corruptions) ∀ θ ∈ (0 , 1) , let X o ∈ R n × p , x i,j ∼ iid BG ( θ ) .
Let D o ∈ O ( n ; R ) be an orthogonal matrix and assume Y
= D o
X o .
For any orthogonal matrix W ∈ O ( n ; R ) and any random Bernoulli matrix B ∈ R n × p , b i,j ∼ iid Ber ( β ) independent of X o , let Y C
= Y + σ B ◦ S denote the input with sparse corruptions, and S ∈ R n × p is deﬁned in equation 15, then:
[Equation]
when p = Ω (cid:0) σ 8 βn 2 ln n/δ 2 (cid:1) .
Proof According to Proposition 3.5, we know that
[Equation] [Equation]
Moreover, since Y C =
[ y 1 + σ b 1 ◦ s 1 , y 2 + σ b 2 ◦ s 2 , . . .
, y p + σ b p ◦ s p ] , whose columns are independent.
Let
Z = [
[Equation] [Equation]
[Equation] [Equation]
where
C =
D ∗ o ( B ◦ S ) .
Note that the columns of Z are independent, we can deﬁne function f
z ( W ) :
O ( n ; R ) (cid:55)→ R as
[Equation]
Next, we will check assumption 1,
[Equation]
and 3 then apply Lemma B.4.
Assumption 1: µ ( n, p ) .
Since Z = X o + σ C , we have {
[Equation]
Moreover, since
[Equation] [Equation]
we have {
[Equation]
Thus we know that
[Equation]
Speciﬁcally, we set B = p 14 , which yields
[Equation]
Assumption 2: Lipschitz Constant L f .
By Proposition 3.5, ∀ W ∈ O ( n ; R ) , we know that
[Equation]
Hence ∀ W 1 , W 2 ∈ O ( n ; R )
[Equation]
we have
[Equation]
from the derivation of equation 34, we have
[Equation]
and therefore, we have
[Equation]
which yields
[Equation]
Assumption 2: Lipschitz Constant ¯ L f .
Since f ¯ z ( W ) = (cid:107) W ∗ ¯ z (cid:107) 44 , apply the same derivation as equation 37, we have
[Equation]
which yields
[Equation] [Equation]
Assumption 3: Upper Bound R 1 .
Notice that ∀ W ∈ O ( n ; R ) ,
[Equation]
Assumption 3: Upper Bound R 2 .
Assume the support for x is S , so we know that ∀ i ∈ [ n ] ,
[Equation]
i
[Equation]
otherwise
[Equation]
(88) where v ∼ N ( 0 , I ) is a Gaussian vector and c = D ∗ o ( b ◦ s ) is the sparse corruption vector after orthogonal rotation D ∗ o .
Let P S : R n (cid:55)→ R n be a projection onto the support S , that is, ∀ q ∈ R n :
[Equation]
Let M =
D
o W
[Equation]
and h = b ◦ s
[Equation]
we have
[Equation]
Moreover, we have:
[Equation]
We now provide upper bound for E ( (cid:104) P S w i , v (cid:105) r ) and E ( (cid:104) m i , h (cid:105) r ) for r = 2 , 4 , 6 , 8 respectively.
Since E h =
and h =
b ◦ s .
Let S (cid:48) denote the support of h , we have
[Equation]
Now we discuss these cases separately:
• With probability c 1 β 4 ( c 1 ≤ 1) , all k 1 , k 2 , k 3 , k 4 are in S , in this case, we have
[Equation]
• With probability c 2 β 3 ( c 2 ≤ 1) , only three among k 1 , k 2 , k 3 , k 4 are in S (cid:48) , in this case, we have
[Equation]
• With probability c 3 β 2 ( c 3 ≤ 1) , only two among k 1 , k 2 , k 3 , k 4 are in S (cid:48) , in this case, we have
[Equation]
• With probability c 4 β ( c 4 ≤ 1) , only one among k 1 , k 2 , k 3 , k 4 is in S (cid:48) , in this case, we have
[Equation]
Hence
[Equation]
combine equation
[Equation]
equation
[Equation]
equation
[Equation]
and equation
[Equation]
we know that
[Equation]
for a constant c > 1 .
Similarly
[Equation]
one can conclude that
[Equation]
Moreover, since v ∼ N ( 0 , I ) , we know that
[Equation]
[Equation] [Equation]
Hence, with the similar technique applied above
[Equation]
we have
[Equation]
for a constant c > 1 .
Similarly, we have
[Equation]
for a constant c > 1 .
Therefore, combine equation 91, we have
[Equation]
≤ cθ +
[Equation]
+
[Equation]
for a constant c > 1 .
Hence, combine equation 90, we have
[Equation]
for a constant c > 1 .
Therefore, we can conclude that
[Equation]
for a constant c > 1 .
Applying Lemma B.4 for Concentration.
Now we apply Lemma B.4 with 1.
[Equation]
for a constant c > 1 , we have
[Equation]
for a constant C > 96 , when p =
Ω (cid:0) σ 8 βn 2 ln n/δ 2 (cid:1) .
B R ELATED L EMMAS AND I NEQUALITIES
Lemma B.1 (Two-sided Bernstein’s Inequality)
Given p random variables x 1 , x 2 , . . .
x p , if ∀ i ∈
[ p ] , | x i | ≤ b almost surely, then
[Equation]
Proof See Proposition 2.14 in Wainwright (2019), one can easily generalize it to two-sided case.
Lemma B.2 (Entry-wise Truncation of a Bernoulli Gaussian Matrix)
Let X ∈ R n × p , where x i,j ∼ iid BG ( θ ) and let (cid:107)·(cid:107) ∞ denote the maximum element (in absolute value) of a matrix, then
[Equation]
Proof A Bernoulli Gaussian variable x i,j , ∀ i ∈ [ n ] , j ∈ [ p ] satisﬁes x i,j = b i,j · g i,j , where b i,j ∼ iid Ber ( θ ) , g i,j ∼ iid N (0 , 1) and therefore
[Equation]
[Equation]
By union bound, we have:
[Equation]
Lemma B.3 ( (cid:15) − Net Covering of Stiefel Manifolds)
There is a covering (cid:15) − net S (cid:15) for Stiefel manifold M = { W ∈ R n × r | W ∗ W = I } , ( n ≥ r ) in operator norm
[Equation]
subject to
[Equation]
113) of size |S (cid:15) | ≤ (cid:0) 6 (cid:15) (cid:1) nr .
Proof See Lemma D.4 in Zhai et al.
(2019b).
Lemma B.4 (Uniform Concentration bound over O ( n ; R ) )
Let Z ∈ R n × p be a random matrix whose columns z 1 , z 2 , . .
, z p are i.i.d.
drawn from a distribution P .
∀
z ∼ P , let f z ( · ) :
O ( n ; R ) (cid:55)→ R denote a function that maps O ( n ; R ) to R .
∀ B > 0 , let ¯ Z denote the truncation of Z : ¯ z i,j = (cid:
) z i,j if | z
i,j | ≤ B, otherwise.
Assume that:
P (max i,j | z i,j | > B ) < µ ( n, p ) , where µ ( n, p ) → 0 as p increase, B depends on p .
E f z ( · ) is L f -Lipschitz and f ¯ z ( · ) is ¯ L
f -Lipschitz, that is, ∀ W 1 , W 2 ∈ O ( n ; R ) :
[Equation]
∀ W ∈ O ( n ; R ) , f ¯ z ( W ) ≤ R 1 , E [ f 2 z ( W )]
≤ R
[Equation]
Then:
[Equation]
when p > ρ , where ρ depends on n
[Equation]
Proof By assumption
[Equation]
we have
[Equation]
Uniform bound over O ( n ; R ) .
∀ (cid:15) > 0 , lemma B.3 shows there exists an (cid:15) -nets S (cid:15) that covers
O ( n ; R ) :
[Equation]
(118) and |S (cid:15) | satisﬁes |S (cid:15) | ≤ (6 /(cid:15) ) n 2 .
Together with Lipschitz assumption 2, we know that sup
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12)(cid:12) (cid:12) p
[Equation]
≤
[Equation]
≤
[Equation]
Hence, let (cid:15) =
nδ 2( L f
[Equation]
¯ L f
[Equation]
we have
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Tail bound within each B ( W l , (cid:15) ) .
Then, ∀ l ∈ [ S (cid:15) ] , we apply point-wise control to a given point
W l ∈ S .
Later we will provide a uniform concentration bound over O ( n ; R ) .
By triangle inequality, we have
[Equation]
∀ W ∈ B ( W l , (cid:15) ) , by assumption 1 and 3, we have
[Equation]
Let ρ be the lower bound of p , such that ∀ p > ρ , we have (cid:112) R 2 µ ( n, p ) <
nδ 4 .
Hence, when p > ρ , we have
[Equation] [Equation]
which implies
P
[Equation]
p
[Equation]
p
[Equation]
j
[Equation]
where the last inequality is achieved by Bernstein’s inequality (Lemma B.1), along with assumption 3 and
E f 2¯ z ( W ) ≤
E f 2 z ( W ) ≤ R 2 .
Combine equation 125 and equation 121, we have
[Equation]
<
exp
[Equation]
Summary.
Therefore, we conclude that
[Equation]
when p > ρ
[Equation]
Lemma B.5 (Concentration Bound of the Clean Objective over O ( n ; R ) ) ∀ θ ∈ (0 , 1] , if X ∈ R n × p , x i,j ∼ iid BG ( θ ) , for any δ > 0 , the following inequality holds
[Equation]
for some constants c 1 > 10 4 , c 2 > 3360 .
Moreover
[Equation]
when p = Ω( θn 2 ln n/δ 2 ) .
Proof See Lemma 2.2 in Zhai et al.
(2019b), note that the sparsity condition θ ∈ (0 , 1) of the original Lemma in Zhai et al.
(2019b) can be easily generalized to θ = 1 .
C A DDITIONAL E XPERIMENTAL R ESULTS
[Figure]
[Figure] [Figure]