I MPLICIT B IAS OF P ROJECTED S UBGRADIENT M ETHOD G IVES P ROVABLE R OBUST R ECOVERY OF S UBSPACES OF U NKNOWN C ODIMENSION
Anonymous authors
Paper under double-blind review
A BSTRACT
Robust subspace recovery (RSR) is a fundamental problem in robust representation learning.
Here we focus on a recently proposed RSR method termed Dual Principal Component Pursuit (DPCP) approach, which aims to recover a basis of the orthogonal complement of the subspace and is amenable to handling subspaces of high relative dimension.
Prior work has shown that DPCP can provably recover the correct subspace in the presence of outliers, as long as the true dimension of the subspace is known. We show that DPCP can provably solve RSR problems in the unknown subspace dimension regime, as long as orthogonality constraints -adopted in previous DPCP formulations- are relaxed and random initialization is used instead of spectral one.
Namely, we propose a very simple algorithm based on running multiple instances of a projected sub-gradient descent method (PSGM), with each problem instance seeking to ﬁnd one vector in the null space of the subspace.
We theoretically prove that under mild conditions this approach will succeed with high probability.
In particular, we show that 1) all of the problem instances will converge to a vector in the nullspace of the subspace and 2) the ensemble of problem instance solutions will be sufﬁciently diverse to fully span the nullspace of the subspace thus also revealing its true unknown codimension.
We provide empirical results that corroborate our theoretical results and showcase the remarkable implicit rank regularization behavior of PSGM algorithm that allows us to perform RSR without being aware of the subspace dimension.
I NTRODUCTION
Robust subspace recovery (RSR) refers to methods designed to identify an underlying linear subspace (with dimension less than the ambient data dimension) in a dataset which is potentially corrupted with outliers (i.e., points that do not lie in the linear subspace).
Many methods for RSR have been proposed in the literature over the past several years Xu et al.
; You et al.
(2017a); Lerman & Maunu (2018).
Formulations based on convex relaxations and decompositions of the data matrices into a low-rank matrices plus sparse corruptions – either uniformly at random as in Robust PCA (RPCA)
Candès et al.
(2011) or column-sparse corruptions as in Xu et al.
; McCoy & Tropp (2011)– can, in certain situations, be shown to provably recover the true subspace when the dimension is unknown.
However, these theoretical guarantees often require the dimension of the subspace, d , to be signiﬁcantly less than the ambient dimension of the data, D , and these methods are not suitable to the more challenging high relative subspace dimension regime (i.e., when dD ≈ 1 ).
The Dual Principal Component Pursuit approach for RSR.
Recently, progress has been made towards solving the RSR problem in the high relative dimension regime by a formulation termed
Dual Principal Component Pursuit (DPCP), which is provably robust in recovering subspaces of high relative dimension, Tsakiris & Vidal (2018).
As implied by its name, DPCP follows a dual perspective of RSR aiming to recover a basis for the orthogonal complement of the inliers’ subspace.
However, a key limitations of DPCP is that it requires a priori knowledge of the true subspace dimension. DPCP for c = 1 .
Let ˜ X ∈ R D × ( N + M ) denote the data matrix deﬁned as ˜ X
=
[ X O ] Γ where X ∈ R D
× N is a matrix containing N inliers as its columns, O ∈ R D
× M is a matrix containing M outliers, and Γ is an unknown permutation matrix.
DPCP was ﬁrst formulated in Tsakiris & Vidal (2018) for handling subspaces of codimension c
= D − d equal to 1 (i.e., the subspace is a hyperplane with dimension d = D − 1 ) formulated as the following optimization problem:
[Equation]
Note that problem (1) is nonconvex due to the spherical constraint imposed on vector b ∈ S D −which is a normal vector of the D − 1 dimensional hyperplane.
In Tsakiris & Vidal (2018) the authors showed that the global minimizer of (1) will give a normal vector of the underlying true hyperplane as long as both inliers and outliers are well-distributed or the ratio between the number of inliers and number of outliers is sufﬁciently small.
Following a probabilistic point of view, the authors in Zhu et al.
(2018) presented an improved theoretical analysis of DPCP giving further insights on the remarkable robustness of DPCP in recovering the true underlying subspaces even in datasets heavily corrupted by outliers.
Moreover, the authors introduced a projected subgradient method which converges to a normal vector of the true subspace at a linear rate.
Recursive DPCP for known c > 1 .
The authors of Zhu et al.
(2018) also proposed an extension to DPCP which allows for subspaces with codimension c > 1 via a projected subgradient algorithm which attempted to learn c normal vectors to the subspace in a recursive manner.
Speciﬁcally, after convergence to a normal vector, the projected subgradient algorithm is initialized with a vector orthogonal to the previously estimated normal vector.
However, for that approach to be successful the knowledge of the true subspace codimension c becomes critical.
Speciﬁcally, if an underestimate of the true codimension c is assumed the recovered basis for the null space, ˆ B , will fail to span the whole null space, S ⊥ .
On the other hand, an overestimate of c will lead to columns of ˆ B corresponding to vectors that lie in S .
Orthogonal DPCP for known c .
In Zhu et al.
(2019), an alternative to (1) was proposed which attempts to solve for c normal vectors to the subspace at once via the formulation
[Equation]
The authors also propose an optimization algorithm based on the projected Riemannian subgradient method (RSGM), which builds on similar ideas as the projected subgradient method of Zhu et al.
(2018) and enjoys a linear converge rate when the step size is selected based on a geometrically diminishing rule.
In Ding et al.
(2021) a theoretical analysis is provided on the geometric properties of 2 showing the merits of DPCP in handling datasets a) highly contaminated by outliers (in the order of M =
O ( N 2 ) ) and b) subspaces of high relative dimension.
However again, it is critical to note that a key shortcoming of this approach is the fact that because all minimizers of problem (2) will be orthogonal matrices, a prerequisite for recovering the correct orthogonal complement of the inliers subspace is the a priori knowledge of the true codimension c (see Fig. 1).
Contributions.
In this work, we address this key limitation by proposing a framework that allows us to perform robust subspace recovery in high relative subspace dimension regime without requiring a priori knowledge of the true subspace dimension.
In particular, our proposed approach will be based on the simple approach of solving multiple, parallel instances of the DPCP formulation for solving for a single normal vector to the subspace,
[Equation]
(3) where c (cid:48) is assumed to be a safe overestimate of c i.e., c (cid:48) ≥ c .
Contrary to (2), the objective function in (3) decouples over the columns
b i of matrix B =
[ b 1 b 2 · · · b c (cid:48) ] and thus can be can be solved in a parallel manner by independently applying a projected subgradient algorithm (referred to as PSGM) from c (cid:48) different random initialization .
Moreover, we observe that with random initialization we can get vectors sufﬁciently spread on the sphere that lead PSGM (initialized with those vectors) to return normal vectors of S .
These are all linearly independent when c (cid:48) ≤ c and thus can span S ⊥
when c (cid:48)
=
c . If c (cid:48) > c then PSGM will return c (cid:48) − c redundant vectors that will still lie in S ⊥ yet they will be linearly dependent (see Figure 1). That being said, we show that this simple strategy permits us to robustly recover the true subspace even without knowledge of the true codimension c .
As is detailed in Sections 3 and 4, this remarkable behavior of PSGM originates from the implicit bias that is induced in the optimization problem due to a) the relaxation of orthogonality constraints in (3) and b) the random initialization scheme that is adopted.
Our speciﬁc contributions are as follows:
First, we focus on a continuous version of (3) i e., in the case that inliers and outliers are distributed under continuous measures which induces a benign landscape in DPCP and hence is easier to be analyzed.
We prove that DPCP problem formulated as in (3) can be solved via a projected subgradient algorithm that implicitly biases solutions towards low-rank matrices ˆ B ∈ R D × c (cid:48) whose columns are the projections of the randomly initialized columns of B 0 onto S ⊥ .
As a result, ˆ B almost surely spans S ⊥ as long as it is randomly initialized with c (cid:48) ≥ c . 2.
Second, we analyze the discrete version which is more challenging, yet of more practical interest, showing that iterates of DPCP-PSGM converge to a scaled and perturbed version of the initial matrix B 0 .
This compelling feature of DPCP-PSGM allows to derive a sufﬁcient condition and a probabilistic bound guaranteeing when the matrix ˆ B ∈ R D × c (cid:48) spans S ⊥ .
We provide empirical results both on simulated and a real datasets , corroborating our theory and showing the robustness of our approach even without knowledge of the true subspace codimension.
R ELATED WORK
Subspace Recovery.
Learning underlying low-dimensional subspace representations of data has been a central topic of interest in machine learning research.
Principal Component Analysis (PCA) has been the most celebrated method of this kind and is based on the minimization of the perpendicular distances of the data points from the estimates linear subspace, Jolliffe & Cadima (2016).
Albeit, it is originally formulated as nonconvex optimization problem, PCA can be easily solved in closed form using a singular value decomposition (SVD) operation (see e.g. Vidal et al.
Despite its great success, PCA is prone to failure when handling datasets that contain outliers i.e., data points whose deviation from the inliers’ subspace is “large” in the (cid:96) 2 norm sense.
Robust Subspace Recovery (RSR).
To remedy this weakness of PCA robust subspace recovery (RSR) methods attempt to identify the outliers in the dataset an recover the true underlying lowdimensional subspace of the inliers Lerman & Maunu (2018); Maunu et al.
A classical approach to this problem is RANSAC Fischler & Bolles (1981), which is given a time budget and randomly chooses per iteration d points and then ﬁts a d -dimensional subspace to those points and checks how the proposed subspace ﬁts the remaining data points.
RANSAC then outputs the subspace that agrees with the largest number of points.
However, RANSAC’s reliance on randomly sampling points to propose subspaces can be highly inefﬁcient when the number of outliers is high (as well as the fact that RANSAC also needs knowledge of the true subspace dimension, d ).
The need to tackle inherent shortcomings of RANSAC pertaining to computational complexity issues inspired alternative convex formulations of RSR, Xu et al.
; You et al.
(2017b)
; Rahmani & Atia (2017); Zhang & Lerman (2014).
In Xu et al.
(2012) the authors decompose the data matrix as a sum of a low-rank and a column-sparse component.
However, theoretical guarantees obtained for convex formulations only hold for subspaces of relatively low-dimensional subspaces i.e., for d (cid:28) D where d and D denote the subspace and the ambient dimension, respectively.
To the best of our knowledge, existing RSR algorithms rely heavily on one of two key assumptions. The subspace is very low-dimensional relative to the ambient dimension ( d (cid:28) D ) or 2) The subspace dimension is a priori known .
Undoubtedly, the second hypothesis is rather strong in real-world applications, and many applications also do not satisfy the ﬁrst assumption.
Moreover, heuristic strategies for selecting the dimension of the subspace are hard to be applied in the RSR setting since they incur computationally prohibitive procedures, Lerman & Maunu (2018).
Relation to Orthogonal Dictionary Learning (ODL).
Note that objective functions in the form of (3) show up beyond RSR problems i.e., in orthogonal dictionary learning (ODL), sparse blind deconvolution, etc., Qu et al.
Speciﬁcally, based on a similar formulation the authors in Bai et al.
(2019) proved that c (cid:48)
=
O ( c log c ) independent random initial vectors sufﬁce in order to recover with high probability a dictionary of size D × c with high accuracy.
In this paper we aim to recover a basis of the orthogonal complement of a subspace of unknown dimension instead of accurately estimating a dictionary hence our goal differs from that in Bai et al.
Implicit bias in Robust Recovery Problems.
The notions of implicit bias and implicit regularization have been used interchangeably in the nonconvex optimization literature for describing the tendency of optimization algorithms to converge to global minima of minimal complexity with favorable generalization properties in overparameterized models, Gunasekar et al.
In the context of robust recovery, the authors in You et al.
(2020) showed that Robust PCA can be suitably reparametrized in such a way to favor low-rank and sparse solutions without using any explicit regularization.
In this work, we use the term implicit bias for describing the convergence of DPCPPSGM to low-rank solutions, which are not necessarily global minimizers, that span the orthogonal complement of the subspace when a) orthogonality constraints in DPCP formulation are relaxed b) DPCP is overparameterized i.e., c (cid:48) ≥ c and c) PSGM randomly initialized. To the best of our knowledge our work is the ﬁrst that explores implicit bias in projected subgradient algorithms that minimize objective functions with sphere/orthogonality constraints.  D UAL P RINCIPAL C OMPONENT P URSUIT AND THE P ROJECTED S UBGRADIENT M ETHOD We re-write the DPCP formulation given in (1) as
[Equation]
In Zhu et al.
(2018), the authors proposed a projected subgradient descent algorithm for addressing (4) that consists of a subgradient step followed by a projection onto the {+sphere i.e., b k +
 = ˆ b k+}
[Equation]
µ k (cid:0) X Sgn( X (cid:62) b k ) + O Sgn( O (cid:62) b k ) (cid:1) and ˆ b k +
=
P S D − 1 ( b k +1 ) , (5) where µ k is the -adaptively updated per iteration- step size and ˆ b k is the unit (cid:96) 2 norm vector corresponding to the k th iteration.
The convergence properties of the projected subgradient algorithm described above depend on speciﬁc quantities denoted as c X , min and c X , max that reﬂect the geometry of the problem and are deﬁned as c X , min = 1 N min b ∈ S
d − 1 ∩S (cid:107) X (cid:62) b (cid:107) 1 and c X , max = 1 N max b ∈ S D − 1 ∩S (cid:107) X (cid:62) b (cid:107) 1 .
Note that the more well distributed the inliers are in the subspace S the higher the value of the quantity c X , min (called as permeance statistic which ﬁrst appeared in Lerman et al.
as it becomes harder to ﬁnd a vector b in the subspace S that is orthogonal to the inliers.
Moreover, c X , min and c X , max converge to the same value as N → ∞ provided the inliers are uniformly distributed in the subspace i.e., c X , min → c d , c X , max → c d , where c d is given as the average height of the unit hemisphere on R d , c d
[Equation]
= [Equation] [Equation] (cid:40) 2 π , if d is even , 1 , if d is odd where k !!
= (cid:40) k
( k − 2)( k − 4) · · · 4 · 2 , k is even , k
( k − 2)( k − 4) · · · 3 · 1 , k is odd (6)
Similarly to c X , min , c X , max , we will also be interested in quantities c O , min , c O , max which indicate how well-distributed the outliers are in the ambient space.
These quantities are deﬁned as c O , min = min b ∈ S D − 1 1 M (cid:107) O (cid:62) b (cid:107) 1 and c O , max = max b ∈ S D − 1 1 M (cid:107) O (cid:62) b (cid:107) 1 .
c O , max can be viewed as the dual permeance statistic and is bounded away from small values while its difference from c O , min tends to zero as M → ∞ .
Further, if the outliers are uniformly distributed on the sphere, then c O , max → c D and c O , min → c D where c D is deﬁned as in (6), (Zhu et al., 2018).
Finally, we also deﬁne the quantities η O = 1 M max b ∈ S D − 1 (cid:107) ( I − bb (cid:62) )
O Sgn( O (cid:62) b ) (cid:107) 2 and η X = M max b ∈ S D − 1 (cid:107) ( P S − bb (cid:62) ) X Sgn( X (cid:62) b ) (cid:107) 2 .
As M → ∞ and assuming outliers in O are welldistributed we get O Sgn( O (cid:62) b ) → c D b thus η O → 0 (Tsakiris & Vidal, 2018).
Likewise, η X →as N → ∞ provided that inliers are uniformly distributed in the d -dimensional subspace.
The following theorem (see full version in Appendix) provides convergence guarantees of the projected subgradient method that was proposed in Zhu et al.
(2018) for addressing problem (1).
Theorem 1 (Informal Theorem 3 of Zhu et al.
Let { ˆ b k } the sequence generated by the projected subgradient algorithm in Zhu et al.
(2018), with initialization ˆ b 0 such that [Equation] and
[Equation]
where θ 0 denotes the principal angle of b 0 from S ⊥ .
If the step size µ k is updated according to a piecewise geometrically diminishing rule given as
[Equation]
where β < 1 , (cid:98)·(cid:99) is the ﬂoor function, then the iterates b k converge to a normal vector of S .
D UAL P RINCIPAL C OMPONENT P URSUIT IN SUBSPACES OF UNKNOWN CODIMENSION
Current theoretical results provide guarantees for recovering the true inlier subspace, when the proposed algorithms know a priori of the subspace codimension c , which is a rather strong requirement and is far from being true in real word applications.
Here we describe our proposed approach, which consists of removing the orthogonality constraint on B , along with a theoretical analysis that gives guarantees of recovering the true underlying subspace even when the true codimension c is unknown.
First we analyze a continuous version of DPCP, which arises when the number of inliers and outliers are distributed according to continuous measures and their number tends to ∞ .
The continuous
DPCP incurs an optimization problem with a benign landscape that allows us to better illustrate the favorable properties of DPCP-PSGM when it comes to the convergence of its iterates.
Then we extend the results to the discrete case that deals with a ﬁnite number of inliers and outliers yielding a more challenging optimization landscape.
4. PSGM’ S ITERATES CONVERGENCE IN THE CONTINUOUS VERSION OF DPCP
The following lemma provides the continuous version of the discrete objective function given in (3).
Lemma
In the continuous case, the discrete DPCP problem given in (3) is reformulated as,
[Equation]
(9) where f b : S D − 1 → R , f b ( z ) = | z (cid:62) b | , φ i is the principal angle of b i from the inliers subspace S and p is the probability of occurrence of an outlier.
Note that µ S D − 1 , µ S D − 1 ∩ S are the continuous measures associated with the outliers and inliers, respectively.
Evidently, (9) attains its global minimum for vectors
b i s that are orthogonal to the inliers’ subspace.
Based on (3) and due to Lemma 2, we can now minimize the objective function of the “continuous version” of DPCP by employing a projected subgradient methods (PSGM) that performs the following steps per iteration
[Equation]
{+Lemma 
 A+} projected subgradient algorithm consisting of the steps described in (10) using a piecewise geometrically diminishing step size rule (see (8) in Theorem 1) will almost surely asymptotically converge to a matrix ˆ B ∗ ∈ R D
× c (cid:48) whose columns ˆ b ∗ i , i = 1 , 2 , . . .
, c (cid:48) will be normal vectors of the inliers’ subspace when randomly initialized with vectors b 0 i ∈ S D − 1 , i = 1 , 2 , . .
c (cid:48) uniformly distributed over the sphere S D − 1 .
Lemma 3 allows us to claim that we can always recover c (cid:48) ≥ c normal vectors to the inliers’ subspace using a PSGM algorithm consisting of steps given in (10).
However, this does not tell the whole story yet, since our ultimate objective is to recover a matrix ˆ B that spans S ⊥ .
Thus, it remains to show that the rank of ˆ B is equal to the true and unknown codimension of the inliers’ subspace c .
Next we prove that by initializing with a ˆ B 0 such that rank( ˆ B 0 )
=
c (cid:48) (i.e., ˆ B 0 is initialized to be full-rank), we can guarantee that we can solve the continuous version of DPCP using PSGM and converge to a ˆ B such that rank( ˆ B ) =
c thus getting span( ˆ B ) ≡ S ⊥ (along with recovering the true subspace dimension).
By projecting the PSGM iterates given in (10) onto S ⊥ we have, P [Equation] [Equation]
and P S
[Equation]
ˆ b k + [Equation] [Equation] We hence observe that the projections of successive iterates of PSGM are scaled versions of the corresponding projections of the previous iterates.
We can now state Lemma 4.
Lemma 4 The PSGM iterates ˆ b ki , i = 1 , 2 , . . .
, c (cid:48) , k = 1 , 2 , . .
given in (10), when randomly initialized with ˆ b 0 i s, i = 1 , 2 , . . .
, c (cid:48) that are independently drawn from a spherical distribution with unit (cid:96) 2 norm converge almost surely to c (cid:48) normal vectors of the inliers subspace S denoted as ˆ b ∗ i , i = 1 , 2 , . . .
, c (cid:48) that are given by ˆ b ∗ i = P S⊥ (ˆ b 0 i
[Equation]
S⊥ [Equation] [Equation] [Equation]
Lemma 4 shows that the initialization of PSGM plays a pivotal role since it determines the direction of the recovered normal vectors { ˆ b ∗ i } c (cid:48) i =1 . Lemmas 3 and 4 pave the way for Theorem 5.
Theorem 5 Let ˆ B 0 ∈ R D
×
c (cid:48) where c (cid:48) ≥ c with c denoting the true codimension of the inliers subspace S , consisting of unit (cid:96) 2 norm column vectors ˆ b 0 i ∈ S D − 1 , i = 1 , 2 , . . .
, c (cid:48) that are independently drawn from uniform distribution over the sphere S D − 1 .
A PSGM algorithm initialized with ˆ B 0 will almost surely converge to a matrix ˆ B ∗ such that span( ˆ B ∗ ) ≡ S ⊥ .
From Theorem 5 we observe that in the benign scenario where inliers and outliers are distributed under continuous measures, we can recover the correct orthogonal complement of the inlier’s subspace even when we are oblivious to its true codimension.
Remarkably, this is achieved by exploiting the implicit bias induced by multiple random initializations of the PSGM algorithm for solving the DPCP formulation given in (3), which is free of orthogonality constraints.
4. PSGM’ S ITERATES CONVERGENCE IN THE DISCRETE VERSION OF DPCP
From this analysis of the continuous version of DPCP we now extend to the the discrete version, which is of more practical relevance for ﬁnite data, yet also presents more challenges.
To begin, we reformulate the DPCP objective as follows
[Equation]
where x b i and o b i are called as average inliers and average outliers terms, deﬁned as x b i =
N
In Algorithm 1, we give the projected subgradient method (DPCP-PSGM) applied on the DPCP problem given in (3). Result: ˆ B = [ˆ b k 1 , ˆ b k 2 , . . . , ˆ b kc (cid:48) ] Initialize:
{+Randomly sample ˆ b 10 , ˆ b 20 , . . . , ˆ b c (cid:48) 
 from a uniform distribution on S D − 1 ; for k = 1 , 2 , . . . do for i = 1 , 2 , . . .+}
, c (cid:48) do
Update the step-size according to a speciﬁc rule; [Equation] end end Note that the average outliers and inliers terms are discrete versions of the corresponding continuous average terms c D b i and c d ˆ s i where ˆ s i =
P S ( b i ) , respectively, Tsakiris & Vidal (2018).
We now express the sub-gradient step of Algorithm 1 as
[Equation]
13) where the quantities e i,k
O =
o
ˆ b ki − c D ˆ b ki − and e i,k X
= x
ˆ b ki −
c D ˆ s ki account for the error between the continuous and discrete versions of the average outliers and the average inliers terms, respectively.
Following a similar path as in the continuous case we next project the iterates of (13) onto S ⊥ ,
P S ⊥ ( b {+k +
+}
i
[Equation] [Equation]
Remark.
Eq.
(14) reveals that DPCP-PSGM, applied on the discrete problem, gives rise to updates whose projections to S ⊥ are scaled and perturbed versions of the previous estimates. The magnitude of perturbation depends on the discrepancy between the continuous and the discrete problem.
Recall that P S ⊥ (ˆ b ki ) =
P S⊥ ( b ki ) (cid:107) b ki (cid:107) 2 , and we can rewrite the update of the 2nd iteration of DPCP-PSGM,
[Equation]
15) where we have assumed that (cid:107) b
i (cid:107) 2 = 1 .
By repeatedly applying the same steps
[Equation]
we can reach to the following recursive expression for P S ⊥ ( b Ki ) ,
[Equation]
(16) where for j > K − 1 we set (cid:81) K − 1 j = k +1 (1 − µ ji Mc D ) (cid:107) b ji
(cid:107)= 1 .
By dividing (16) with (cid:81) K − 1 k =0 (1 − µ ki Mc D ) (cid:107) b ki (cid:107)and by projecting onto the sphere S D − 1 we get
[Equation]
(17) where δ i is deﬁned as δ K i = (cid:80) K
[Equation]
kj
[Equation]
Assumption 1.
We assume that the principal angles θ i 0 for all b
i s satisfy the inequality θ i 0 <
[Equation]
Assumption 1 essentially assumes that the sufﬁcient condition given in eq.
(7) required by PSGM algorithm for converging to a normal vector is satisﬁed which is the same condition for success in
Zhu et al.
Under Assumption 1 we can invoke the convergence properties of PSGM given in Theorem 1 and get as K → ∞ , P S D − 1 ( P S ⊥ ( b K ))
→ ˆ b ∗ ∈
S ⊥ ∩ S D − 1 .
That being said, we denote ˆ b ∗ i =
P S D − 1 (cid:16) P S ⊥ ( b 0 i )
− P S ⊥ ( ˆ δ i ) (cid:17) , where ˆ δ i
=
lim K →∞ δ
Ki 3 .
Following the same steps as in Section 4.1 and by deﬁning matrices B ∗ =
[ b ∗ 1 , b ∗ 2 , . . .
, b ∗ c (cid:48) ] , B 0 =
[ b 01 , b 02 , . . .
, b 0 c (cid:48) ] , ˆ ∆ =
[ˆ δ 1 , ˆ δ 2 , . .
, ˆ δ c (cid:48) ] we can express the matrix B ∗ as ˆ B ∗ =
P S D − 1 (cid:16) P S ⊥ (cid:16) B 0 − ˆ ∆ (cid:17)(cid:17) where B ∗ will now consist of normal vectors of the inliers’ subspace.
In order to guarantee that span( B ∗ ) ≡ S ⊥ it thus sufﬁces to ensure that rank ( B ∗ ) =
c .
Here we show that a sufﬁcient condition for this to hold is that the matrix A
= B 0 − ˆ ∆ is full-rank.
Lemma
If σ c (cid:48)
( B 0 ) > (cid:107) ˆ ∆ (cid:107) 2 then matrix A
= B 0 − ˆ ∆ is full-rank.
From Lemma 6 we can see that the success of DPCP-PSGM hinges on how well-conditioned the matrix B 0 is.
Speciﬁcally, it says that if a lower-bound on the smallest singular is satisﬁed then
DPCP-PSGM is guaranteed to converge to the correct complement of the inlier without knowledge of the correct codimension c .
From this, we can prove the following Theorem.
Theorem 7 Let B 0 ∈ R D
×
c (cid:48) with columns randomly sampled from a unit (cid:96) 2 norm spherical distribution where c (cid:48) ≥ c with c denoting the true codimension of the inliers subspace S that satisﬁes
Assumption 1.
If
[Equation]
where κ = max i
[-Mµ 0 i β K 0 /K ∗ (1 − r i ) and r i = ( 1+ µ 
-][Equation]
r
[Equation]
[-) 1 − µ 
 i Mc D-] β 1 /K ∗ then with probability at least 1 − 2 exp( − (cid:15) 2 C 2 ) (where C 1 , C 2 are absolute constants), Algorithm 1 with a piecewise geometrically diminishing step size rule will converge to a matrix ˆ B ∗ such that span( ˆ B ∗ ) ≡ S ⊥ .
Note that quantities β, K ∗ , K 0 are used in the step-size update rule that is used as deﬁned in (8) (See also full version of Theorem 1 in Appendix)).
Theorem 7 shows that we can randomly initialize DPCPPSGM, with a matrix ˆ B 0 whose number of columns c (cid:48) is an overestimate of the true codimension c of the inliers’ subspace and with columns sampled independently by a uniform distribution over the unit sphere and recover a matrix that will span the orthogonal complement of S .
The probability of success depends on the geometry of the problem since condition (18) is trivially satisﬁed (RHS of (18) tends to 0 since η O → 0 and c O , max → c d ) in the continuous case which incurs a benign geometry.
Moreover, the a less benign geometry would increase the value of ( η O + c O , max − c d ) thus requiring a smaller initial codimension c (cid:48) that would lead to larger values the LHS of (18).
N UMERICAL S IMULATIONS
In this section we demonstrate the effectiveness of the proposed DPCP formulation and the derived
DPCP-PSGM algorithm in recovering orthogonal complements of subspace of unknown codimension.
We compare the proposed algorithm with previously developed methods i.e., DPCP-IRLS Tsakiris & Vidal (2018) and the Riemannian Subgradient Method (RSGM)
Zhu et al.
Recall that both DPCP-IRLS and RSGM address DPCP problem by enforcing orthogonality constraints, and thus both algorithms are quite sensitive if the true codimension of S is not known a priori . Further, they are both initialized using of spectral initialization i.e., ˆ B 0 ∈ R D × c (cid:48) which contains the ﬁrst c (cid:48) eigenvectors of matrix ˜ X
˜ X (cid:62) as its columns, as proposed in Tsakiris & Vidal (2018); Zhu et al.
Robustness to outliers in the unknown codimension regime.
In this experiment we set the dimension of the ambient space to D = 200 .
We randomly generate N inliers uniformly distributed with unit (cid:96) 2 norm in a d = 195 dimensional subspace (hence for its codimension we have c
= D − d = 5 ).
Following a similar process we generate M outliers that live in the ambient space and are sampled from a uniform distribution over the unit sphere.
Fig. 2, illustrates the distances (see Appendix) of the recovered matrix ˆ B as obtained by the proposed DPCP-PSGM algorithm initialized with an overestimate c (cid:48)
= 10 of the true codimension c codimension and two versions of RSGM i.e., RSGM when it is given as input true c = 5 and RSGM when being incognizant of c and hence it initialized with a c (cid:48)
= 10 of c As is shown in Fig.
2 (right), RSGM fails to recover the correct orthogonal complement of S when it is provided with an overestimate of the true c which is attributed to spectral initialization and the imposed orthogonality constraints.
On the contrary, DPCP-PSGM displays a remarkably robust behavior (Fig. 2(middle)) even without knowing the true value of c , performing similarly to RSGM when the latter knows beforehand the correct codimension (Fig. 2(left)).
Recovery of the true codimension.
Here we test DPCP-PSGM on the recovery of the true codimension c of the inliers’ subspace S .
Again, we set D = 200 and follow the same process described above for generating N = 1500 inliers.
We vary the true codimension of S from c = 10 to 20 and consider two different outlier’s ratios r , deﬁned as r = MM + N , namely r = 0 .
6 and r = 0 .
In both cases, DPCP-PSGM is initialized with the same overestimate of c i.e., c (cid:48) = 30 .
In Fig.
3 we report the estimated codimensions obtained by DPCP-PSGM for 10 independent trials of the experiments.
It can be observed that DPCP-PSGM achieves 100% for all different codimensions for r = 0 .
Moreover, it shows a remarkable performance in estimating the correct c ’s even in the more challenging case corresponding to outliers’ ratios equal to 0.7.
with the estimated codimensions being close to the true values even in the cases that it fails to exactly compute c .
The results corroborate the theory showing that the DPCP-PSGM with random initialization biases the solutions of ˆ B towards matrices with rank c .
A hyperspectral imaging experiment See Appendix B.1.
C ONCLUSIONS
We proposed a simple framework which allows us to perform robust subspace recovery without requiring a priori knowledge of the subspace codimension.
This is based on Dual Principal Component Pursuit (DPCP) and thus is amenable to handling subspaces of high relative dimensions.
We observed that a projected subgradient methods (PSGM) induces implicit bias and converges to a matrix that spans a basis of the orthogonal complement of the inliers subspace even as long as a) we overestimate it codimension, b) lift orthogonality constraints enforced in previous DPCP formulations and c) use random initialization.
We provide empirical results that corroborate the developed theory and showcase merits
Ethics Statement This work focuses on theoretical aspects of robust subspace recovery problem which is a well-established topic in machine learning research.
The research conducted in the framework of this work raises no ethical issues or any violations vis-a-vis the ICLR Code of Ethics.
A A PPENDIX
Theorem 1 (Theorem 3 of Zhu et al.
Let { ˆ b k } the sequence generated by the projected subgradient method in Zhu et al.
(2018), with initialization ˆ b 0 such that
[Equation]
where θ 0 denotes the principal angle of b 0 from S ⊥ , and
[Equation]
Let µ (cid:48) :
=
[Equation]
If µ
[Equation]
≤ µ
[Equation]
and the step size µ k is updated according to a piece-wise geometrically diminishing rule given as
[Equation]
where β < 1 , (cid:98)·(cid:99) is the ﬂoor function, and K 0 , K ∗ ∈ N are chosen such that
[Equation]
where,
K
[Equation]
[Equation]
) then for the angle θ k between ˆ b k and S ⊥ it holds
[Equation]
A. P ROOF OF L EMMA
Lemma
In the continuous case, the discrete DPCP problem given in (3) is reformulated as, min [Equation]
[Equation]
where f b : S D − 1 → R , f b ( z ) = | z (cid:62) b | , φ i is the principal angle of b i from the inliers subspace S and p is the probability of occurrence of an outlier.
Proof We deﬁne the discrete measures µ X , µ O associated with the inliers and outliers, respectively as,
[Equation]
where δ ( · ) is the Dirac function.
Recall that,
[Equation]
(26) where g : S D − 1 → R and µ S D − 1 is the uniform measure on S D − 1 .
The DPCP objective for the discrete version of the problem divided by M + N can be written as,
[Equation]
Note that µ X , µ O arise by discretizing the continuous uniform measures µ S D − 1 and µ S D − 1 ∩ S respectively ( µ S D − 1 ∩ S denotes the uniform measure on S D − 1 ∩ S ) and p is the probability of occurrence of an outlier i.e., MM + N → p as M, N → ∞ ( 1 − p corresponds to the probability of occurrence of an inlier).
That being said, the continuous version of DPCP can be simply stated by replacing µ X , µ O with µ S D − 1 and µ S D − 1 ∩ S in equation 27 as follows,
[Equation]
The RHS of (9) immediately shows up by invoking Proposition 4 in Tsakiris & Vidal (2018).
A. P ROOF OF L EMMA
Lemma 3 : A projected subgradient algorithm consisting of the steps described in (10) using a piecewise geometrically diminishing step size rule (see (8) in Theorem 1) will almost surely asymptotically converge to a matrix ˆ B ∗ ∈ R D
× c (cid:48) whose columns ˆ b ∗ i , i = 1 , 2 , . . .
, c (cid:48) will be normal vectors of the inliers’ subspace when randomly initialized with vectors b 0 i ∈ S D − 1 , i = 1 , 2 , . .
c (cid:48) uniformly distributed over the sphere S D − 1 .
Proof : The proof can be trivially obtained by noticing a) that the condition for convergence i.e., inequality (7) of the projected subgradient algorithm given in Theorem 1 becomes θ
i < π 2 in the continuous case (since η X → 0 , η O → 0 , c X , min → c d > 0 ) and b) the set of unit (cid:96) 2 -norm vectors b
i s, i = 1 , 2 , . . .
, c (cid:48) sampled independently by a uniform distribution over the sphere and whose principal angle θ
i is π 2 form the inliers’ subspace has measure 0 .
[Equation]
A. P ROOF OF T HEOREM
Lemma 4 The PSGM iterates ˆ b ki , i = 1 , 2 , . . .
, c (cid:48) , k = 1 , 2 , . .
given in (10), when randomly initialized with ˆ b 0 i s, i = 1 , 2 , . . .
, c (cid:48) that are independently drawn from a spherical distribution with unit (cid:96) 2 norm converge almost surely to c (cid:48) normal vectors of the inliers subspace S denoted as ˆ b ∗ i , i = 1 , 2 , . . .
, c (cid:48) that are given by
[Equation]
Proof Let us assume b 0 i
=
ˆ b 0 i .
The iterates of subgradients steps of PSGM can be written in the following form,
[Equation]
By projecting each update of PSGM onto S ⊥ and since P S ⊥ ( b ki ) = (cid:107) b ki (cid:107)P S ⊥
(ˆ b ki ) we have,
[Equation]
We can thus easily derive the following form for P S ⊥ ( b Ki ) ,
[Equation]
We know from Theorem 1 and Lemma 3 when DPCP-PSGM is initialized with b [-
-]0 i , i = 1 , 2 , . . .
, c (cid:48) s randomly drawn according according to a spherical distribution then it will almost surely converge as K → ∞ to vectors ˆ b ∗ i , i = 1 , 2 , . .
c (cid:48)
i.e., ˆ b Ki → ˆ b ∗ i where ˆ b ∗ i ∈ S ⊥ .
Hence P S ⊥ (ˆ b Ki )
→ ˆ b ∗ i as K → ∞ .
Note that from Theorem 1 we have that µ ki (cid:54)
= 1 pc D ∀ k = { 1 , 2 , . .
, K } hence (cid:81) K −
[Equation]
(cid:54) = 0 .
From 31 and after projecting on the unit sphere
and we thus have
[Equation]
Theorem 5 Let ˆ B 0 ∈ R D
×
c (cid:48) where c (cid:48) ≥ c with c denoting the true codimension of the inliers subspace S , consisting of unit (cid:96) 2 norm column vectors ˆ b 0 i ∈ S D − 1 , i = 1 , 2 , . . .
, c (cid:48) that are independently drawn from uniform distribution over the sphere S D − 1 .
A PSGM algorithm initialized with ˆ B 0 will almost surely converge to a matrix ˆ B ∗ such that span( ˆ B ∗ ) ≡ S ⊥ .
Proof From Lemma 4 we have that for each initial unit norm vector b 0 i which corresponds to the i th column of B 0 will almost surely converge to ˆ b ∗ i =
P
S⊥
(ˆ b 0 i )
(cid:107)P S⊥ (ˆ b 0 i ) (cid:107) 2 .
We can thus write B ∗ =
P S ⊥ ( B 0 ) Γ where Γ is a full-rank diagonal matrix given be Γ = diag( [Equation] S⊥
[Equation]
b [Equation] Note that P S ⊥ is a linear projection and thus we can write P S ⊥ ( B 0 ) =
B S ⊥ B (cid:62)S ⊥ where B S ⊥ R D
× c is an orthonormal matrix which spans S ⊥ .
Note that the probability of sampling a low-rank matrix
B 0 =
[ b 01 , b 02 , . . .
, b 0 c (cid:48) ] when columns b
i s are randomly and independently drawn from a spherical distribution is zero.
We thus have B ∗ =
B S ⊥ B (cid:62)S ⊥ B 0 Γ with rank( B ∗ )
=
c .
Lemma
If σ c (cid:48)
( B 0 ) > (cid:107) ˆ ∆ (cid:107) [-
-]2 then the rank of matrix B 0 − ˆ ∆ equals c (cid:48) .
Proof Let A =
B 0 − ˆ ∆ .
From singular value perturbation inequalities we have | σ i ( A ) − σ i ( B 0 ) | ≤ (cid:107) ˆ ∆ (cid:107) 2 , for i = 1 , 2 , . . .
, c (cid:48) .
Hence it holds,
[Equation]
If σ c (cid:48)
( B 0 ) > [-
 the-](cid:107) ˆ ∆ (cid:107) 2 then from equation 32 we get σ c (cid:48) ( A
[Equation]
(33) hence the matrix B 0 − ˆ ∆ will be full-rank.
A. P ROOF OF T HEOREM
We ﬁrst give the following Lemmas:
Lemma
For the (cid:96) 2 norm of e i,k O for any k = 1 , 2 , . . .
, K and i = 1 , 2 , . .
, c [Equation] it holds,
[Equation]
Proof
[Equation]
where we have used the fact that (cid:107) b (cid:107) 2 = 1 .
Lemma 8 Let the step size of Algorithm 1 (DPCP-PSGM) µ ki being updated following the piecewise geometrically diminishing step size rule with
[Equation]
For the spectral norm of ˆ ∆ it holds (cid:107) ˆ
[Equation]
(cid:107) 2 < √ c (cid:48) κ ( η O + c O , max − c d ) where κ = max i
[Equation]
K
[Equation]
and r i = ( 1+ µ
[Equation]
N
[Equation]
η X + c X
[Equation]
max
[Equation] + M [Equation] η O + c O
[Equation]
max [Equation] ) ) 1 − µi Mc D β 1 /K ∗ . Proof We ﬁrst bound the (cid:96) 2 norm of vectors b ji s.
We have that ∀ i = 1 , 2 , . . .
, c (cid:48) and j = 1 , 2 , . .
, K it holds
[Equation]
We deﬁne the quantities
[Equation]
ˆ b ji (cid:107) 2 = 1 hence
[Equation]
Due to equation 39 and since µ ji follows a non-increasing path as j → k , the scalar term (cid:81) kj =
[Equation]
b ji
[Equation]
1 − µ ji Mc D ) is bounded above as follows,
[Equation]
We now focus on the geometrically diminishing step size rule given in equation 8.
We have µ ki = µ
[Equation] β [Equation] k − K 0 /K ∗
[Equation] +1 <
[-µ 
-][Equation]
β ( k − K 0 ) /K
∗ for k ≥ K 0 and µ 0 i β ( k − K 0 ) /K
∗ > µ 0 i for k < K 0 .
Hence we can get the following upper bound
[Equation]
The series
[Equation]
is geometric and if
[Equation]
it converges as K → ∞ to 1 1 − r i where r
i
[Equation]
Let us now bound the (cid:96) 2 norms of the columns of ˆ ∆ .
From Lemma 7 we have (cid:107) e i,k O (cid:107) 2 ≤ η O
+ c O , max − c d .
We can easily thus derive that (cid:107) ˆ δ i (cid:107) 2 ≤ κ ( η O + c
O , max − c d ) .
For the spectral norm of ˆ ∆ we thus have
[Equation]
Where the last inequality arises since (cid:107) x (cid:107) 1 ≤ √ c (cid:48) (cid:107) x (cid:107) [Equation]
We then give the Theorem.
Theorem 9 (Theorem 5.58 of Vershynin (2010)) Let B be a D × d matrix ( D ≥ d ) whose columns b i are independent sub-gaussian isotropic random vectors in R D with (cid:107) b
i (cid:107) 2 = √ D almost surely.
Then for every t ≥
the inequality
[Equation]
with probability at least 1 − 2 exp( − ct 2 ) , where C = C (cid:48)
k , c = c (cid:48) K > 0 depend only on the subgaussian norm K = max j (cid:107)
b i (cid:107) ψ 2 of the columns.
The proof of Theorem 7 follows next.
Theorem 7 Let B 0 ∈ R D
×
c (cid:48) with columns randomly sampled from a unit (cid:96) 2 norm spherical distribution where c (cid:48) ≥ c with c denoting the true codimension of the inliers subspace S that satisﬁes
Assumption 1.
If
[Equation]
where κ = max i
[Equation]
r
[Equation]
β 1 /K ∗ then with probability at least 1 − 2 exp( − (cid:15) 2 C 2 ) (where C 1 , C 2 are absolute constants), Algorithm 1 with a geometrically diminishing step size rule will converge to a matrix ˆ B ∗ such that span( ˆ B ∗ ) ≡ S ⊥ .
[Table]
Proof By Assumption 1 we have that all columns of B 0 will satisfy the sufﬁcient condition for convergence of DPCP-PSGM (Algorithm 1) to a normal vector of S .
From Lemma 6 and we use the inequality σ c (cid:48)
( B 0 ) > (cid:107) ˆ ∆ (cid:107) 2 which ensures full-rankness of ˆ B ∗ , which is the key ingredient in order to prove that span( ˆ B ∗ ) =
S ⊥ .
We can then Use Theorem 9 for matrix B 0 .
Note that columns of Bare drawn independently and are uniformly distributed on the unit sphere.
Hence, columns of B 0 are sampled by subgaussian distribution and the LHS of the inequality of the theorem appears if we scale with 1 √ D so that to create unit-norm columns and use LHS of the inequality of Theorem 7.
The RHS of the inequality is due to the upper bound of (cid:107) ˆ ∆ (cid:107) 2 as stated in Lemma 8.
The absolute constants
C 1 , C 2 depend only the subgaussian norm of the uniform distribution (they is no dependency on the dimensions of the problem).
B E XPERIMENTAL DETAILS AND ADDITIONAL MATERIAL
All experiments were conducted on a MacBook Pro 2.6GhZ 6-Core Intel Core i7, memory [-
-]16GB
Mhz DDR using Matlab2019B. For computational purposes and in order to avoid ﬁne-tuning of the piecewise geometrically diminishing (PGD) step size, the modiﬁed backtracking line-search (MBLS) step-size rule was adopted for DPCP-PSGM as proposed in Zhu et al. We deﬁne the distance between two subspaces spanned by matrices B and A as dist ( B , A ) = min Q ∈ O ( D,c ) (cid:107) B − AQ (cid:107) F
where O
{+( D, c ) denotes the Stiefel manifold of orthogonal matrices of rank c . Note that dist ( B , A ) = 
 ⇐⇒ span( B ) ≡ span( A ) (see Zhu et al.+} B. O UTLIERS P URSUIT IN W ASHINGTON DC M ALL AVIRIS HSI
Hyperspectral images (HSIs) provide rich spectral information as compared to RGB images capturing a wide range of the electromagnetic spectrum.
Washington DC Mall AVIRIS HSI contains contiguous spectral bands captured at 0.4 to 2.4 µ m region of visible and infrared spectrum, Giampouras et al.
In this experiments we randomly choose 10 out of its 210 spectral bands.
Due to high coherence in the both the spectral and the spatial domain, pixels of HSIs admit representations in low-dimensional subspaces.
Here, we use a 100 × 100 segment of the hyperspectral image selecting randomly 10 out of its D = 210 spectral bands.
We form a matrix ˜ X of size 10 × 10000 whose columns correspond to different points in the 10 -dimensional ambient space.
Then we corrupt columns of ˜ X by replacing them with outliers that are generated uniformly at random with unit (cid:96)norm for two different outliers’ ratios i.e., r = 0 .
8 and r = 0 .
In the corrupted ˜ X , the remaining clear pixels are considered as the inliers.
Table 1 displays the F1 scores obtained by DPCP-PSGM,
RSGM and DPCP-IRLS algorithm.
The latter two algorithms are evaluated in two scenarios: a) codimension is initialized c (cid:48)
= 5 and b) c (cid:48)
= 10 .
Given the singular value distribution of the initial image, we infer that the dimension d of the inliers’ subspace is less or equal than 5.
Hence, c (cid:48)
= 5 (recall c = D − d ) is close to the true codimension value while c (cid:48)
= 10 is an overestimate thereof.
From Table 1, we can see that the proposed DPCP-PSGM succeeds in both outliers’ ratios regardless its unawareness of the true codimension value.
On the other hand, DPCPIRLS and RSGM fail when initialized with c
=
and this is attributed to the restrictions induced [Figure] due to the orthogonality constraints they both impose.
In Fig.
4 we provide annotated versions of the clean HSI, its corrupted by outliers version for outliers’ ratio r = 90% , and the annotated outliers as recovered by the proposed DPCP PSGM RSGM RSGM with c
= 5 and DPCP- with