Original: Frank-Wolfe-based Algorithms for Approximating Tyler’s M-estimator
Revised: Frank-Wolfe-based Algorithms for Approximating Tyler’s M-estimator

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Tyler’s M-estimator is a well known procedure for robust and heavy-tailed covariance estimation.
Revised: Tyler’s M-estimator is a well known procedure for robust and heavy-tailed covariance estimation.

Original: Tyler himself suggested an iterative ﬁxed-point algorithm forcomputing his estimator however, it requires super-linear (in the size of the data)runtime per iteration, which maybe prohibitive in large scale.
Revised: Tyler himself suggested an iterative ﬁxed-point algorithm for computing his estimator however, it requires super-linear (in the size of the data) runtime per iteration, which maybe prohibitive in large scale.

Original: In this work wepropose, to the best of our knowledge, the ﬁrst Frank-Wolfe-based algorithms forcomputing Tyler’s estimator.
Revised: In this work we propose, to the best of our knowledge, the ﬁrst Frank-Wolfe-based algorithms for computing Tyler’s estimator.

Original: One variant uses standard Frank-Wolfe steps, thesecond also considers away-steps (AFW), and the third is a geodesic version of
Revised: One variant uses standard Frank-Wolfe steps, the second also considers away-steps (AFW), and the third is a geodesic version of

Original: AFW (GAFW).
Revised: AFW (GAFW).

Original: AFW provably requires, up to a log factor, only linear time periteration, while GAFW runs in linear time (up to a log factor) in a large n (numberof data-points) regime.
Revised: AFW provably requires, up to a log factor, only linear time per iteration, while GAFW runs in linear time (up to a log factor) in a large n (number of data-points) regime.

Original: All three variants are shown to provably converge to theoptimal solution with sublinear rate, under standard assumptions, despite the factthat the underlying optimization problem is not convex nor smooth.
Revised: All three variants are shown to provably converge to the optimal solution with sublinear rate, under standard assumptions, despite the fact that the underlying optimization problem is not convex nor smooth.

Original: Under an additional fairly mild assumption, that holds with probability 1 when the (normalized)data-points are i.i.d.
Revised: Under an additional fairly mild assumption, that holds with probability 1 when the (normalized) data-points are i.i.d.

Original: samples from a continuous distribution supported on the entireunit sphere, AFW and GAFW are proved to converge with linear rates.
Revised: samples from a continuous distribution supported on the entire unit sphere, AFW and GAFW are proved to converge with linear rates.

Original: Importantly,all three variants are parameter-free and use adaptive step-sizes.
Revised: Importantly, all three variants are parameter-free and use adaptive step-sizes.

Original:  Introduction
Revised:  Introduction

Original: Given n data-points in R p , x 1 , . . .
Revised: Given n data-points in R p , x 1 , . . .

Original: , x n 2 R p different from zero, Tyler’s M-estimator (TME),originally proposed by Tyler in his seminal work [27], is a p ⇥ p positive deﬁnite matrix Q ⇤ whichsatisﬁes:
Revised: , x n ∈ R p different from zero, Tyler’s M-estimator (TME), originally proposed by Tyler in his seminal work [27], is a p × p positive deﬁnite matrix Q ∗ which satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: While the TME is not guaranteed to exist for any set of points x 1 , . .
Revised: While the TME is not guaranteed to exist for any set of points x 1 , . .

Original: , x n , it is well known that underthe following assumption proposed in [27], it does.
Revised: , x n , it is well known that under the following assumption proposed in [27], it does.

Original: Assumption 1.
Revised: Assumption 1.

Original: The data { x 1 , . . . , x n } ⇢ R p satisﬁes that x i 6 = 0 for all i = 1 , . . . , n , and that forany proper subspace L ⇢ R p , denoting by N ( L ) the number of data-points lying in L , it holds thatn > N ( L ) dim( L ) p .
Revised: The data { x 1 , . . . , x n } ⊂ R p satisﬁes that x i (cid:54) = 0 for all i = 1 , . . . , n , and that for any proper subspace L ⊂ R p , denoting by N ( L ) the number of data-points lying in L , it holds that n > N ( L ) dim( L ) p .

Original: The TME is a well known distribution-free robust covariance estimator.
Revised: The TME is a well known distribution-free robust covariance estimator.

Original: It is also a maximumlikelihood estimator for the shape matrix of angular and compound Gaussian distributions, and oftenused for estimation of heavy-tailed distributions [28].
Revised: It is also a maximum likelihood estimator for the shape matrix of angular and compound Gaussian distributions, and often used for estimation of heavy-tailed distributions [28].

Original: Tyler’s M-estimator has received notableinterest within the statistics and signal processing communities, see for instance the excellent survey[28] (including the many important references therein), as well as the recent works [30, 11, 29, 25, 6,16, 20, 24, 2] that provide additional analysis results, related estimators, as well as structured versions of Tyler’s estimator (e.g., convex constraints, low-rank structure, sparse structure, and more), to nameonly a few.
Revised: Tyler’s M-estimator has received notable interest within the statistics and signal processing communities, see for instance the excellent survey [28] (including the many important references therein), as well as the recent works [30, 11, 29, 25, 6, 16, 20, 24, 2] which provide additional analysis results, related estimators, as well as structured versions of Tyler’s estimator (e.g., convex constraints, low-rank structure, sparse structure, and more), to name only a few.

Original: It is also well known that when the TME exists, it is given by the optimal solution to the followingoptimization problem:
Revised: It is also well known that when the TME exists, it is given by the optimal solution to the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where S p denotes the set of real-valued symmetric p ⇥ p matrices, and Q  0{+(cid:31) 
+} ( Q ⌫ 0 ) denotes that36 Q is positive deﬁnite (positive semideﬁnite).
Revised: where S p denotes the set of real-valued symmetric p × p matrices, and Q {+(cid:31) 
+} ( Q (cid:23) 0 ) denotes that Q is positive deﬁnite (positive semideﬁnite).

Original: Indeed the connection between (1) and (2) becomes more apparent when examining the gradientgiven by
Revised: Indeed the connection between (1) and (2) becomes more apparent when examining the gradient given by

Original: [Equation]
Revised: [Equation]

Original: Thus, a matrix Q  0 satisfying (1), also satisﬁes Q r f ( Q ) Q = 0 , meaning r f ( Q ) = 0 , and theother way around.
Revised: Thus, a matrix Q (cid:31) 0 satisfying (1), also satisﬁes Q ∇ f ( Q ) Q = 0 , meaning ∇ f ( Q ) = 0 , and the other way around.

Original: In particular, it can be shown that points Q  0 for which r f ( Q ) = 0  are theonly stationary points of Problem (2) (i.e., do not have descent directions).
Revised: In particular, it can be shown that matrices Q (cid:31) 0 for which ∇ f ( Q ) = 0 , are the only stationary points of Problem (2) (i.e., do not have descent directions).

Original: Theorem 1 (see for instance [28]) .
Revised: Theorem 1 (see for instance [28]) .

Original: Under Assumption 1, Problem (2) admits a unique optimalsolution Q ⇤  0 .
Revised: Under Assumption 1, Problem (2) admits a unique optimal solution Q ∗ (cid:31) 0 .

Original: This solution is also the unique solution to Eq.
Revised: This solution is also the unique solution to Eq.

Original: (1) with trace equals p .
Revised: (1) with trace equals p .

Original: Throughout this paper we assume that Assumption 1 indeed holds true.
Revised: Throughout this paper we assume that Assumption 1 indeed holds true.

Original: Note also that (1) isinvariant to scaling of each data-point x i , i = 1 , . . . , n , and the matrix Q ⇤ .
Revised: Note also that (1) is invariant to scaling of each data-point x i , i = 1 , . . . , n , and the matrix Q ∗ .

Original: Hence, throughoutthis paper, as often customary, we assume that x 1 , . . .
Revised: Hence, throughout this paper, as often customary, we assume that x 1 , . . .

Original: , x n are normalized to have unit-length, i.e.,k x i k 2 = 1 , i = 1 , . .
Revised: , x n are normalized to have unit-length, i.e., (cid:107) x i (cid:107) 2 = 1 , i = 1 , . .

Original: . , p , and Q ⇤ is normalized to have trace equals p , i.e., Tr ( Q ⇤ )
Revised: . , p , and Q ∗ is normalized to have trace equals p , i.e., Tr ( Q ∗ )

Original: = p .
Revised: = p .

Original: The subject of this paper are efﬁcient algorithms for approximating the TME in large-scale, i.e.,when both n, p are large.
Revised: The subject of this paper are efﬁcient algorithms for approximating the TME in large-scale, i.e., when both n, p are large.

Original: In [27] Tyler proposed a simple algorithm for computing the TME whichperforms ﬁxed-point iterations, as we deﬁne next.
Revised: In [27] Tyler proposed a simple algorithm for computing the TME which performs ﬁxed-point iterations, as we deﬁne next.

Original: Deﬁnition 1 (Fixed-point iterations for computing the TME) .
Revised: Deﬁnition 1 (Fixed-point iterations for computing the TME) .

Original: The FPI method computes the followingiterations:
Revised: The FPI method computes the following iterations:

Original: [Equation]
Revised: [Equation]

Original: and the returned solution is given by Q = p Q T
Revised: and the returned solution is given by Q = p Q T

Original: Tr ( Q T ) , where Q T is the last iterate computed.
Revised: Tr ( Q T ) , where Q T is the last iterate computed.

Original: Note that each iteration (4) requires O ( p 3 + np 2 ) runtime:
Revised: Note that each iteration (4) requires O ( p 3 + np 2 ) runtime:

Original: O ( p 3 ) time in order to compute theinverse matrix Q  1 t from the previous iterate Q t , and additional O ( np 2 ) time to compute the sum ofrank-one matrices.
Revised: O ( p 3 ) time in order to compute the inverse matrix Q − 1 t from the previous iterate Q t , and additional O ( np 2 ) time to compute the sum of rank-one matrices.

Original: In particular, the runtime is super-linear in the size of the input which is np , andthus can be prohibitive when n, p are both very large (note that under Assumption 1 we always haven > p ).
Revised: In particular, the runtime is super-linear in the size of the input which is np , and thus can be prohibitive when n, p are both very large (note that under Assumption 1 we always have n > p ).

Original: In [27] Tyler also proved the convergence of the iteration (4) (under Assumption 1) but withouta rate.
Revised: In [27] Tyler also proved the convergence of the iteration (4) (under Assumption 1) but without a rate.

Original: Recently, it was proved in [6] that when the data x 1 , . . .
Revised: Recently, it was proved in [6] that when the data x 1 , . . .

Original: , x n are i.i.d.
Revised: , x n are i.i.d.

Original: samples from anelliptical distribution, the FPI method produces a matrix b Q  0 with Tr ( b Q ) = p , such that withhigh probability, k I  Q 
Revised: samples from an elliptical distribution, the FPI method produces a matrix (cid:98) Q (cid:31) 0 with Tr ( (cid:98) Q ) = p , such that with high probability, (cid:107) I − Q ∗ 1 / 2 (cid:98) Q − 1 Q ∗ 1 / 2 (cid:107) F ≤ (cid:15) , after O ( | log det Σ | + p + log(1 /(cid:15) ))

Original: ⇤ 1 /b Q  1 Q ⇤ 1 / 2 k F  ✏ , after O ( | log det ⌃ | + p + log(1 / ✏ )) iterations, forany error tolerance ✏ , where Q ⇤ is the solution to 1 with trace equals p , ⌃ is the shape matrix ofthe underlying elliptical distribution, and k · k F denotes the Frobenius (Euclidean) norm.
Revised: iterations, for any error tolerance (cid:15) , where Q ∗ is the solution to (1) with trace equals p , Σ is the shape matrix of the underlying elliptical distribution, and (cid:107)·(cid:107) F denotes the Frobenius (Euclidean) norm.

Original: The proof65 of this result is highly involved and relies on deep mathematical concepts such as strong geodesicconvexity and quantum expanders.
Revised: The proof of this result is highly involved and relies on deep mathematical concepts such as strong geodesic convexity and quantum expanders.

Original: Note that taking into account the runtime of a single iteration (4)and the bound on number of iterations, the total runtime to reach ✏ error (according to the measure ofconvergence in [6]) is ⌦ ( p 4 + np 3 ) .
Revised: Note that taking into account the runtime of a single iteration (4) and the bound on number of iterations, the total runtime to reach (cid:15) error (according to the measure of convergence in [6]) is Ω( p 4 + np 3 ) .

Original: The goal of this paper is to present new simple and efﬁcient ﬁrst-order methods for solving Problem(2) which avoid the super-linear runtime barrier of np 2 , and even the matrix inversion barrier ( O ( p 3 ) in practical implementations), required by each iteration of (4).
Revised: The goal of this paper is to present new simple and efﬁcient ﬁrst-order methods for solving Problem (2) which avoid the super-linear runtime barrier of np 2 , and even the matrix inversion barrier ( O ( p 3 ) in practical implementations), required by each iteration of (4).

Original: Our algorithms are based on the wellknown Frank-Wolfe method (aka conditional gradient method) for constrained smooth minimization[5, 15].
Revised: Our algorithms are based on the well known Frank-Wolfe method (aka conditional gradient method) for constrained smooth minimization [5, 15].

Original: To the best of our knowledge this is the ﬁrst time that the Frank-Wolfe method has beenconsidered for computing the TME.
Revised: To the best of our knowledge this is the ﬁrst time that the Frank-Wolfe method has been considered for computing the TME.

Original: We provide three such variants with the following properties:
Revised: We provide three such variants with the following properties:

Original: • All three variants provably converge to the optimal solution (the one satisfying Eq.
Revised: • All three variants provably converge to the optimal solution (the one satisfying Eq.

Original: (1))under Assumption 1 with a sublinear rate — O (1 / ✏ 2 ) iterations for approximation error ✏ .•
Revised: (1)) under Assumption 1 with a sublinear rate — O (1 /(cid:15) 2 ) iterations for approximation error (cid:15) .

Original:  Two variants, AFW and GAFW, are also proved to converge with a linear rate (i.e,.
Revised: • Two variants, AFW and GAFW, are also proved to converge with a linear rate (i.e,.

Original: log(1 / ✏ )dependence on the error tolerance ✏ ), under an additional fairly mild assumption that holdswith probability 1 when the (normalized) data-points are i.i.d.
Revised: log(1 /(cid:15) ) dependence on the error tolerance (cid:15) ), under an additional fairly mild assumption that holds with probability 1 when the (normalized) data-points are i.i

Original:  samples from a continuousdistribution supported on the entire unit sphere (note that the O (log(1 / ✏ )) rate proved in [6]also holds under an assumption that the data is sampled i.i.d.
Revised: d. samples from a continuous distribution supported on the entire unit sphere (note that the O (log(1 /(cid:15) )) rate proved in [6] also holds under an assumption that the data is sampled i.i.d.

Original: from an elliptical distribution).• All three variants are parameter-free and do not require any tuning of parameters, and applyadaptive step-sizes (i e., not ﬁxed beforehand).
Revised: from an elliptical distribution). • All three variants are parameter-free and do not require any tuning of parameters, and apply adaptive step-sizes (i.e., not ﬁxed beforehand).

Original: • The AFW variant requires only ˜ O ( np ) runtime per iteration which is linear in the size ofthe data, up to a single logarithmic factor, while another variant GAFW runs in the same time when n  p 2 .
Revised: • The AFW variant requires only ˜ O ( np ) runtime per iteration which is linear in the size of the data, up to a single logarithmic factor, while another variant GAFW runs in the same time when n ≥ p 2 .

Original: This is in contrast to FPI which requires O ( np 2 + p 3 ) time per iteration.
Revised: This is in contrast to FPI which requires O ( np 2 + p 3 ) time per iteration.

Original: In fact, even if processing the data could be ideally distributed / done in parallel , FPI stillrequires an
Revised: In fact, even if processing the data could be ideally distributed / done in parallel, FPI still requires an

Original: expensive matrix inversionon
Revised: expensive matrix inversion

Original:  each iteration ( which practically runs in O ( p 3 ) 
Revised: on each iteration (which practically runs in O ( p 3 ) time), while

Original: time ), while our AFW variant only requires additional O ( p 2 ) runtime per iteration.
Revised: our AFW variant only requires additional O ( p 2 ) runtime per iteration.

Original: A quick summary of our results is brought in Table 1.
Revised: A quick summary of our results is brought in Table 1.

Original: Importantly and quite pleasingly, our derivations and proofs rely mostly on standard arguments forcontinuous optimization in Euclidean spaces, and we believe that as such, are quite accessible.
Revised: Importantly and quite pleasingly, our derivations and proofs rely mostly on standard arguments for continuous optimization in Euclidean spaces, and we believe that as such, are quite accessible.

Original: Beyond our contribution to efﬁcient algorithms for computing the TME, our work also contributesmore broadly to the theory of Frank-Wolfe-type methods, which have received signiﬁcant interest inrecent years in the context of obtaining faster rates for convex and smooth problems (e.g., [10, 19, 9,7, 1]), dealing with nonsmooth objectives (e.g., [26, 4, 3]), and dealing with non-convex objectives(e.g.
Revised: Beyond our contribution to efﬁcient algorithms for computing the TME, our work also contributes more broadly to the theory of Frank-Wolfe-type methods, which have received signiﬁcant interest in recent years in the context of obtaining faster rates for convex and smooth problems (e.g., [10, 19, 9, 7, 1]), dealing with nonsmooth objectives (e.g., [26, 4, 3]), and dealing with non-convex objectives (e.g., [18]).

Original: While Problem (2) is nonsmooth and nonconvex, our Frank-Wolfe variants, with ourtailored-designed adaptive step-sizes, are guaranteed to converge to the global minimum, and thisis the ﬁrst result of its kind for Frank-Wolfe-based methods that we are aware of.
Revised: While Problem (2) is nonsmooth and nonconvex, our Frank-Wolfe variants, with our tailored-designed adaptive step-sizes, are guaranteed to converge to the global minimum, and this is the ﬁrst result of its kind for Frank-Wolfe-based methods that we are aware of.

Original: Moreover, whileto the best of our knowledge, so called away-steps in Frank-Wolfe algorithms were only shown tofacilitate linear convergence rates when optimizing over polytopes [19], here we establish that alsofor the feasible set of positive semideﬁnite matrices in Problem (2), such away-steps can lead tolinear convergence rates (under proper assumptions), as we prove for our AFW and GAFW variants.
Revised: Moreover, while to the best of our knowledge, so called away-steps in Frank-Wolfe algorithms were only shown to facilitate linear convergence rates when optimizing over polytopes [19], here we establish that also for the feasible set of positive semideﬁnite matrices in Problem (2), such away-steps can lead to linear convergence rates (under proper assumptions), as we prove for our AFW and GAFW variants.

Original: An additional contribution is our connection between Frank-Wolfe methods and optimization overmanifolds as captured by our GAFW variant which applies Frank-Wolfe steps and away-steps w.r.t.
Revised: An additional contribution is our connection between Frank-Wolfe methods and optimization over manifolds as captured by our GAFW variant which applies Frank-Wolfe steps and away-steps w.r.t.

Original: tothe geodesic gradient of Problem (2).
Revised: to the geodesic gradient of Problem (2).

Original: It is thus our hope that our work will lead to additional efﬁcient
Revised: It is thus our hope that our work will lead to additional efﬁcient

Original: Frank-Wolfe-based algorithms for other important and well-structured non-convex and potentiallynonsmooth problems.
Revised: Frank-Wolfe-based algorithms for other important and well-structured non-convex and potentially nonsmooth problems.

Original: We introduce the following notation.
Revised: We introduce the following notation.

Original: We let S p + and S p ++ denote the set of positive semideﬁniteand positive deﬁnite matrices in S p , respectively.
Revised: We let S p + and S p ++ denote the sets of positive semideﬁnite and positive deﬁnite matrices in S p , respectively.

Original: We denote S p
Revised: We denote S p

Original: = { Q 2 S p + |
Revised: = { Q ∈ S p + |

Original: Tr ( Q ) = p } ,
Revised: Tr ( Q ) = p } ,

Original: S p + = { Q 2 S p ++ | Tr ( Q ) = p } .
Revised: S p + = { Q ∈ S p ++ | Tr ( Q ) = p } .

Original: For a matrix M 2 S p , we let  i ( M ) denote the i th largest112 signed eigenvalue of M .
Revised: For a matrix M ∈ S p , we let λ i ( M ) denote the i th largest signed eigenvalue of M .

Original: We also denote at times by  max ( M ) and  min ( M ) the largest and smallest(signed) eigenvalues, respectively.
Revised: We also denote at times by λ max ( M ) and λ min ( M ) the largest and smallest (signed) eigenvalues, respectively.

Original: For real matrices we let k · k 2 , k · k F , k · k 1 denote the spectral114 norm (largest singular value), Frobenius (Euclidean) norm, and trace norm (sum of singular values),respectively.
Revised: For real matrices we let (cid:107)·(cid:107) 2 , (cid:107)·(cid:107) F , (cid:107)·(cid:107) 1 denote the spectral norm (largest singular value), Frobenius (Euclidean) norm, and trace norm (sum of singular values), respectively.

Original: We also denote by h · , · i the standard inner-product in S p .
Revised: We also denote by (cid:104)· , ·(cid:105) the standard inner-product in S p .

Original: It is known that for every Q 2 S p + , f ( Q ) is ﬁnite, while for every sequence ( Q k ) k  1 ✓ S p  with 117  min ( Q k ) ! k [-!1 
-] it holds that f ( Q k ) !
Revised: It is known that for every Q ∈ S p + , f ( Q ) is ﬁnite, while for every sequence ( Q k ) k ≥ 1 ⊆ S p + with λ min ( Q k ) −→ k [-!1 
-]→∞ 0 , it holds that f ( Q k ) −→ k →∞ ∞ , see [28].

Original: k !1 1 , see [28].

Original: This implies the following lemma that willbe used throughout the paper.
Revised: This implies the following lemma that will be used throughout the paper.

Original: Lemma 1.
Revised: Lemma 1.

Original: Fix Q 0 2 S p
Revised: Fix Q 0 ∈ S p

Original: + .
Revised: + .

Original: There exists a constant  > 0 such that for every Q 2 S p + satisfying120 f ( Q )  f ( Q 0 ) , it holds that  min ( Q )   .
Revised: There exists a constant λ > 0 such that for every Q ∈ S p + satisfying f ( Q ) ≤ f ( Q 0 ) , it holds that λ min ( Q ) ≥ λ .

Original: [Table] 
Revised: [Table] [Table]

Original:  Frank-Wolfe-based Algorithms for Approximating Tyler’s M-estimator
Revised:  Frank-Wolfe-based Algorithms for Approximating Tyler’s M-estimator

Original: We consider three Frank-Wolfe variants for Problem (2), all described below in Algorithm 1.
Revised: We consider three Frank-Wolfe variants for Problem (2), all described below in Algorithm 1.

Original: Allthree variants require to solve (approximately) a certain eigenvalue problem, a different one for eachvariant, see more details below.
Revised: All three variants require to solve (approximately) a certain eigenvalue problem, a different one for each variant, see more details below.

Original: All three variants have the following two properties: 1.
Revised: All three variants have the following two properties: 1.

Original: they alluse the same adaptive scheme to compute the step-size µ t , which in particular does not require anyknowledge of the parameters of the problem (this choice will become clearer in the convergenceanalysis), and
Revised: they all use the same adaptive scheme to compute the step-size µ t , which in particular does not require any knowledge of the parameters of the problem (this choice will become clearer in the convergence analysis) [Table]

Original:  since the gradient of the objective f ( · ) requires the inverse matrix Q  1 t , where Q t isthe current iterate of the algorithm, they all explicitly and efﬁciently maintain the inverse Q  1 t usingthe well-known Sherman-Morrison formula and capitalizing on the fact that the algorithm performs arank-one update.
Revised: and 2. since the gradient of the objective f ( · ) requires the inverse matrix Q − 1 t , where Q t is the current iterate of the algorithm, they all explicitly and efﬁciently maintain the inverse Q − 1 t using the well-known Sherman-Morrison formula and capitalizing on the fact that the algorithm performs a rank-one update.

Original: Hence, the inverse could be updated in only O ( p 2 ) time per iteration (as opposed tothe standard O ( p 3 ) matrix inversion time).
Revised: Hence, the inverse could be updated in only O ( p 2 ) time per iteration (as opposed to the standard O ( p 3 ) matrix inversion time).

Original: Below we expand on each of the eigenvalue problems in
Revised: Below we expand on each of the eigenvalue problems in

Original: Algorithm 1 and how they correspond to familiar and new Frank-Wolfe-type methods.
Revised: Algorithm 1 and how they correspond to familiar and new Frank-Wolfe-type methods.

Original: We note that Algorithm 1 is completely independent of any parameter of Problem (2).
Revised: We note that Algorithm 1 is completely independent of any parameter of Problem (2).

Original: The onlyparameter the algorithm accepts is the approximation parameter  2 [0 , 1) , however this choice is notvery important as will be evident from our convergence theorems, and only brought for convinience.
Revised: The only parameter the algorithm accepts is the approximation parameter β ∈ [0 , 1) , however this choice is not very important as will be evident from our convergence theorems, and only brought for convinience.

Original: In particular one can always simply choose some universal constant, for instance  = 1 / 2 .
Revised: In particular one can always simply choose some universal constant, for instance β = 1 / 2 .

Original: Thealgorithm also requires a feasible initialization point Q 0 2 S p + and its inverse, and a convenientchoice is to simply take Q 

Revised: The algorithm also requires a feasible initialization point Q 0 ∈ S p + and its inverse, and a convenient choice is to simply take Q

Original: = I =
Revised: = I =

Original: Q  1 0 .
Revised: Q − 1 0 .

Original: Algorithm 
Revised: Algorithm 1 Frank-Wolfe variants for approximating Tyler’s M-estimator

Original: Input: Q 0 , Q  1 0 such that Q 0  0 ,  2 [0 , 1) for t = 0, , ... do
Revised: Input: Q 0 , Q − 1 0 such that Q 0 (cid:31) 0 , β ∈ [0 , 1) for t = 0, , ... do

Original: Solve an approximate eigenvalue problem: let v t 2 R p be such that k v t k = p p and satisﬁes one of the following:
Revised: Solve an approximate eigenvalue problem: let v t ∈ R p be such that (cid:107) v t (cid:107) = √ p and satisﬁes one of the following:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: end for
Revised: end for

Original: Standard Frank-Wolfe update: For the feasible set S p , it is well known that the solution to the
Revised: Standard Frank-Wolfe update: For the feasible set S p , it is well known that the solution to the

Original: FW linear optimization problem min V 2 S p h V , r f ( Q t ) i at the current point Q t is given w.l.o.g.
Revised: FW linear optimization problem min V ∈S p (cid:104) V , ∇ f ( Q t ) (cid:105) at the current point Q t is given w.l.o.g.

Original: as
Revised: as

Original: V + := vv > , where v = p p u and u is a unit-length eigenvector of r f ( Q t ) corresponding to thesmallest eigenvalue, i.e., u > r f ( Q ) u
Revised: V + := vv (cid:62) , where v = √ p u and u is a unit-length eigenvector of ∇ f ( Q t ) corresponding to the smallest eigenvalue, i.e., u (cid:62)

Original:  =  min ( r f ( Q t )) , see for instance [14, 15].
Revised: ∇ f ( Q ) u = λ min ( ∇ f ( Q t )) , see for instance [14, 15].

Original: This is exactly the step in Eq.
Revised: This is exactly the step in Eq.

Original: (5) in Algorithm 1, only that in Eq.
Revised: (5) in Algorithm 1, only that in Eq.

Original: (5) we do not require exact computation of theeigenvector, but allow for a (1   ) multiplicative approximation w.r.t.
Revised: (5) we do not require exact computation of the eigenvector, but allow for a (1 − β ) multiplicative approximation w.r.t.

Original: the smallest eigenvalue.
Revised: the smallest eigenvalue.

Original: Frank-Wolfe with Away-Steps (AFW) update: The AFW update (see [13, 19]) applies one oftwo types of updates.
Revised: Frank-Wolfe with Away-Steps (AFW) update: The AFW update (see [13, 19]) applies one of two types of updates.

Original: One is the standard FW step discussed above.
Revised: One is the standard FW step discussed above.

Original: The other, called away-steptakes the form Q t + Q t +  ( Q t  V  ) ,  > 0 , where V  2 S p is ideally chosen so that148 V  maximizes the inner product h V  , r f ( Q t ) i , subject to the constraint that there indeed exists149 a corresponding  > 0 , so that Q t +1 is feasible [13, 19].
Revised: The other, called away-step , takes the form Q t +1 ← Q t + µ ( Q t − V − ) , µ > 0 , where V − ∈ S p is ideally chosen so that V − maximizes the inner product (cid:104) V − , ∇ f ( Q t ) (cid:105) over S p , subject to the constraint that there indeed exists a corresponding µ > 0 , so that Q t +1 is feasible [13, 19].

Original: For Q t  0 , all points in S p give rise to150 such positive  , and so, V  is  the solution to max V 2 S p h V , r f ( Q t ) i , which w.l.o.g. is of the form 
Revised: For Q t (cid:31) 0 , all points in S p give rise to such positive µ , and so, V − is simply the solution to max V ∈S p (cid:104) V , ∇ f ( Q t ) (cid:105) , which w.l.o.g. is of the form V − =

Original: V  = vv > , where v = p p u and u is a unit-length eigenvector of r f ( Q t ) corresponding to thelargest (signed) eigenvalue.
Revised: vv (cid:62) , where v = √ p u and u is a unit-length eigenvector of ∇ f ( Q t ) corresponding to the largest (signed) eigenvalue.

Original: If h Q t  V + , r f ( Q t ) i  h V   Q t , r f ( Q t ) i , AFW performs a standard FW update with[-V + , 
-]
Revised: If (cid:104) Q t − V + , ∇ f ( Q t ) (cid:105) ≥ (cid:104) V − − Q t , ∇ f ( Q t ) (cid:105) , AFW performs a standard FW update with[-V + , 
-]

Original:  and otherwise, it performs an away-step with V  .
Revised: V + , and otherwise, it performs an away-step with V − .

Original: In case of Problem (2), a straightforward calculation shows that h Q t , r f ( Q t ) i = 0 .
Revised: In case of Problem (2), a straightforward calculation shows that (cid:104) Q t , ∇ f ( Q t ) (cid:105) = 0 .

Original: Thus, in our case, A FW step is taken if h V + , r f ( Q t ) i h V  , r f ( Q t ) i , i.e., if  p  min ( r f ( Q t ))  p  max ( r f ( Q t )) , and otherwise an away-step is taken.
Revised: Thus, in our case, a FW step is taken if −(cid:104) V + , ∇ f ( Q t ) (cid:105) ≥ (cid:104) V − , ∇ f ( Q t ) (cid:105) , i.e., if − pλ min ( ∇ f ( Q t )) ≥ pλ max ( ∇ f ( Q t )) , and otherwise an away-step is taken.

Original: Thus, taking both cases into account, AFW performs a step of the form Q t + Q t +  ( vv >  Q t ) ,where | v > r f ( Q t ) v |  p max {   min ( r f ( Q t )) ,  max ( r f ( Q t )) }=
Revised: Thus, taking both cases into account, AFW performs a step of the form Q t +1 ← Q t + µ ( vv (cid:62) − Q t ) (here µ can also be negative), where | v (cid:62) ∇ f ( Q t ) v | ≥ p max {− λ min ( ∇ f ( Q t )) , λ max ( ∇ f ( Q t )) }

Original:  p kr f ( Q t ) k 2 , which is ex- 159 actly what we have in Eq.
Revised: = p (cid:107)∇ f ( Q t ) (cid:107) 2 , which is exactly what we have in Eq.

Original: (6) in Algorithm 1, only that in (6) we again do not require precisecomputation, but allow for a (1   ) multiplicative approximation.
Revised: (6) in Algorithm 1, only that in (6) we again do not require precise computation, but allow for a (1 − β ) multiplicative approximation.

Original: Geodesic Frank-Wolfe with Away-Steps (GAFW) update: GAFW performs the same updatesas AFW, but not w.r.t.
Revised: Geodesic Frank-Wolfe with Away-Steps (GAFW) update: GAFW performs the same updates as AFW, but not w.r.t.

Original: to the (standard) gradient r f ( Q t ) , but with respect to the so-called geodesicgradient given by Q 1 / 2 t r f ( Q t ) Q 1 / 2 t (see for instance [6]).
Revised: to the (standard) gradient ∇ f ( Q t ) , but with respect to the so-called geodesic gradient given by Q 1 / 2 t ∇ f ( Q t ) Q 1 / 2 t (see for instance [6]).

Original: Note that k Q 1 / 2 t r f ( Q t ) Q 1 / 2 t k 2 =max u : k u k 2 =1 | u > Q 1 / 2 t r f ( Q t ) Q 1 / 2 t u | 2 .
Revised: Note that (cid:107) Q 1 / 2 t ∇ f ( Q t ) Q 1 / 2 t (cid:107) 2 = max u : (cid:107) u (cid:107) 2 =1 | u (cid:62) Q 1 / 2 t ∇ f ( Q t ) Q 1 / 2 t u | 2 .

Original: Consider the parametrization u =
Revised: Consider the parametrization u =

Original: Q  1 / 2 t v k Q  1 / 2 t v k 2 (note it isinvariant to the norm of v ). 
Revised: Q − 1 / 2 t v (cid:107) Q − 1 / 2 t v (cid:107) 2 (note it is invariant to the norm of v ). This gives (cid:107) Q 1 / 2 t ∇ f ( Q t ) Q [Equation]

Original: This gives k Q 1 / 2 t r f ( Q t ) Q 1 / 2 t k 2 = max v | v > r f ( Q t ) v | v > Q  1 t v , which is exactlythe update step in Eq.
Revised: [Equation] (cid:107) 2 = max [Equation] , which is exactly the update step in Eq.

Original: (7) in Algorithm 1, only that in Eq.
Revised: (7) in Algorithm 1, only that in Eq.

Original: (7) it sufﬁces to ﬁnd a (1   ) -multiplicativeapproximation.
Revised: (7) it sufﬁces to ﬁnd a (1 − β ) -multiplicative approximation.

Original: As will be evident from our analysis, and in particular in the error reduction argumentin Lemma 3, this step maximizes the decrease in function value on each iteration.
Revised: As will be evident from our analysis, and in particular in the error reduction argument in Lemma 3, this step maximizes the decrease in function value on each iteration.

Original: The following lemma establishes that the step-sizes of Algorithm 1 indeed always produce feasibleiterates in S p + .
Revised: The following lemma establishes that the step-sizes of Algorithm 1 indeed always produce feasible iterates in S p + .

Original: The proof is given in the appendix.
Revised: The proof is given in the appendix.

Original: Lemma 2 (Feasibility of Algorithm 1) .
Revised: Lemma 2 (Feasibility of Algorithm 1) .

Original: Suppose that Tr ( Q 0 ) = p and Q 0  0 .
Revised: Suppose that Tr ( Q 0 ) = p and Q 0 (cid:31) 0 .

Original: Then, for everyiteration t  1 of Algorithm 1 it holds that Tr ( Q t ) = p and Q t  0 .
Revised: Then, for every iteration t ≥ 1 of Algorithm 1 it holds that Tr ( Q t ) = p and Q t (cid:31) 0 .

Original: 2.1 Efﬁcient implementation of the eigenvalue oracles in Algorithm1
Revised: 2.1 Efﬁcient implementation of the eigenvalue oracles in Algorithm

Original: We now discuss how the various eigenvalue problems in Algorithm 1 could be solved (to sufﬁcientapproximation) efﬁciently via simple and well-known iterative algorithms for leading eigenvectorcomputation, such as the power method or the Lanczos algorithm [12, 23].
Revised: We now discuss how the various eigenvalue problems in Algorithm 1 could be solved (to sufﬁcient approximation) efﬁciently via simple and well-known iterative algorithms for leading eigenvector computation, such as the power method or the Lanczos algorithm [12, 23].

Original: These implementationsrely on three simple ideas:
Revised: These implementations rely on three simple ideas:

Original: Use of the Sherman-Morrison formula for rank-one updates to update the inverse  Q  1 t + from the previous one Q  1 t  in O ( p 2 ) time.
Revised: Use of the Sherman-Morrison formula for rank-one updates to update the inverse matrix Q − 1 t +1 from the previous one Q − 1 t , given the vector v t , in O ( p 2 ) time.

Original: Explicitly maintaining and updating the vectors Q  1 t x i , i = 1 , . .
Revised: Explicitly maintaining and updating the vectors Q − 1 t x i , i = 1 , . .

Original: . , n , which can be done in overall O ( np ) time per iteration.
Revised: . , n , which can be done in overall O ( np ) time per iteration.

Original: This allows to compute a matrix-vector product of the form r f ( Q t ) v , for some v 2 R p , in only O ( np ) time (see expression for r f in Eq.
Revised: This allows to compute a matrix-vector product of the form ∇ f ( Q t ) v , for some v ∈ R p , in only O ( np ) time (see expression for ∇ f in Eq.

Original: Apply the power method or the Lanczos algorithm (or any other method for leading eigenvector computation) to solve the leading eigenvalue problems (5), (6), (7), relying on the fact that due to the pervious two items, each iteration of the power method requires only O ( p 2 + np ) time.
Revised: Apply the power method or the Lanczos algorithm (or any other method for leading eigenvector computation) to solve the leading eigenvalue problems (5), (6), (7), relying on the fact that due to the pervious two items, each iteration of the power method requires only O ( p 2 + np ) time.

Original: The complete proof of the following theorem is given in the appendix.
Revised: The complete proof of the following theorem is given in the appendix.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let  2 [0 , 1) be some universal constant (e.g.,  = 1 / 2 ).
Revised: Let β ∈ [0 , 1) be some universal constant (e.g., β = 1 / 2 ).

Original: Algorithm 1 admitsimplementations based on fast algorithms for leading eigenvector computation (e.g., the powermethod / Lanczos [12, 23]), such that each iteration of Algorithm 1 could be implemented in: 
Revised: Algorithm 1 admits implementations based on fast algorithms for leading eigenvector computation (e.g., the power method / Lanczos [12, 23]), such that each iteration of Algorithm 1 could be implemented in: ˜

Original: ˜ O ( np + p 2 ) time when using AFW steps (Eq.
Revised: O ( np + p 2 ) time when using AFW steps (Eq.

Original: O ( p 3 ) + ˜ O
Revised: O ( p 3 ) + ˜ O

Original: ( np + p 2 ) time when using GAFWsteps (Eq.
Revised: ( np + p 2 ) time when using GAFW steps (Eq.

Original: (7) ), and ˜ O ⇣q kr f ( Q t ) k|  p ( r f ( Q t ))
Revised: (7) ), and ˜ O (cid:16)(cid:113) (cid:107)∇ f ( Q t ) (cid:107) 2 | λ p ( ∇ f ( Q t ))

Original: | ( np + p 2 ) ⌘ when using FW steps (Eq.
Revised: | ( np + p 2 ) (cid:17) when using FW steps (Eq.

Original: (5) ), where in all casesthe ˜ O notation hides a logarithmic factor in the dimension p and the probability of failure  .
Revised: (5) ), where in all cases the ˜ O notation hides a logarithmic factor in the dimension p and the probability of failure δ .

Original: Remark 1.
Revised: Remark 1.

Original: Note that the worst-case time to solve the standard FW eigenvalue problem, in termsof the dependence on the size of the data np , is much worse than that of AFW and GAFW.
Revised: Note that the worst-case time to solve the standard FW eigenvalue problem, in terms of the dependence on the size of the data np , is much worse than that of AFW and GAFW.

Original: This isbecause, while the eigenvalue problems in AFW, GAFW require to approximate the largest eigenvaluein magnitude, FW requires to approximate the smallest signed eigenvalue.
Revised: This is because, while the eigenvalue problems in AFW, GAFW require to approximate the largest eigenvalue in magnitude, FW requires to approximate the smallest signed eigenvalue.

Original: Remark 2.
Revised: Remark 2.

Original: Since leading eigenvector algorithms such as the power method are usually initializedwith a random vector, their guarantees only hold with high probability, as captured in Theorem2.
Revised: Since leading eigenvector algorithms such as the power method are usually initialized with a random vector, their guarantees only hold with high probability, as captured in Theorem 2.

Original: However, since the dependence on the probability of failure is only logarithmic, for the clarityof presentation, henceforth we neglect such considerations and treat these computations as if theyalways succeed.
Revised: However, since the dependence on the probability of failure is only logarithmic, for the clarity of presentation, henceforth we neglect such considerations and treat these computations as if they always succeed.

Original:  Sublinear Convergence of Algorithm1
Revised:  Sublinear Convergence of Algorithm

Original: While the measure of convergence in our sublinear rates for AFW and GAFW will be simply thedistance in spectral norm from satisfying the TME equation (1) , our measure of convergence for thestandard FW variant is slightly less obvious and is motivated by the following observation, the proofof which is given in the appendix.
Revised: While the measure of convergence in our sublinear rates for AFW and GAFW will be simply the distance in spectral norm from satisfying the TME equation (1) , our measure of convergence for the standard FW variant is slightly less obvious and is motivated by the following observation, the proof of which is given in the appendix.

Original: Observation 1.
Revised: Observation 1.

Original: For any Q2 S
Revised: For any Q

Original: [Equation]
Revised: [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: and min ⇣
Revised: and

Original: [Equation]
Revised: [Equation]

Original: 0 if and only if Q = Q ⇤ .
Revised: if and only if Q = Q ∗ .

Original: Theorem 3 ( O (1 / ✏ 2 ) convergence of Algorithm 1) .
Revised: Theorem 3 ( O (1 /(cid:15) 2 ) convergence of Algorithm 1) .

Original: Consider the iterates of Algorithm 1 and deﬁnethe function T (˜ ✏ )=
Revised: Consider the iterates of Algorithm 1 and deﬁne the function T (˜ (cid:15) )

Original: d 4( f ( Q 0 )  f ( Q ⇤ ))
Revised: = (cid:100) 4( f ( Q 0 ) − f ( Q ∗ ))

Original: (1 + ˜ ✏  2 ) e .
Revised: (1 + ˜ (cid:15) − 2 ) (cid:101) .

Original: Fix ✏ > 0 and deﬁne
Revised: Fix (cid:15) > 0 and deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then, when using FW steps (Eq. 5), it holds that for all t  T (˜ ✏ FW) ,
Revised: Then, when using FW steps (Eq. 5), it holds that for all t ≥ T [Equation] FW

Original: [Equation]
Revised: [Equation]

Original: When using AFW steps (Eq.
Revised: When using AFW steps (Eq.

Original: or GAFW steps (Eq.
Revised: or GAFW steps (Eq.

Original:  it holds for all t  T (˜ ✏ AFW ) or t  T (˜ ✏ GAFW ) , respectively, that
Revised: , it holds for all t ≥ T (˜ (cid:15) AFW ) or t ≥ T (˜ (cid:15) GAFW ) , respectively, that

Original: [Equation]
Revised: [Equation]

Original: Remark 3.
Revised: Remark 3.

Original: Note that one motivation for considering the use of the GAFW variant in light of
Revised: Note that one motivation for considering the use of the GAFW variant in light of

Original: Theorem 3, is that it enjoys better conditioning than AFW in terms of the maximal condition number max ( Q )/
Revised: Theorem 3, is that it enjoys better conditioning than AFW in terms of the maximal condition number λ max ( Q )

Original:  min ( Q ) over the initial level set { Q 2 S p | f ( Q )  f ( Q 0 ) } .
Revised: /λ min ( Q ) over the initial level set { Q ∈ S p | f ( Q ) ≤ f ( Q 0 ) } .

Original: The complete proof of Theorem 3 is given in the appendix.
Revised: The complete proof of Theorem 3 is given in the appendix.

Original: Below we state and prove the keytechnical step — a bound on the improvement in function value that Algorithm 1 makes on a singleiteration.
Revised: Below we state and prove the key technical step — a bound on the improvement in function value that Algorithm 1 makes on a single iteration.

Original: In particular, the lemma is independent of the way the vector v t is generated and henceapplies to all three variants described in Algorithm 1.
Revised: In particular, the lemma is independent of the way the vector v t is generated and hence applies to all three variants described in Algorithm 1.

Original: Lemma 3.
Revised: Lemma 3.

Original: Fix some iteration t of Algorithm 1 and deﬁne L t = v > t r f ( Q t) v t v
Revised: Fix some iteration t of Algorithm 1 and deﬁne L t = v [Equation] t ∇ f ( Q t

Original: [Equation]
Revised: [Equation]

Original: 1 t 
Revised: v t v

Original: [Equation]
Revised: [Equation]

Revised: t Q − 1 t v t .

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Remark 4.
Revised: Remark 4.

Original: An immediate important consequence ofthe Lemma 3 is that Algorithm 1 is a decentmethod, i.e., the function value never increases from one iteration to the next.
Revised: An immediate important consequence of Lemma 3 is that Algorithm 1 is a decent method, i [Equation] e., the function value never increases from one iteration to the next.

Original: Lemma 3 also showsthe logic behind the GAFW step in Eq.
Revised: Lemma 3 also reveals one motivation for the GAFW step in Eq.

Original: 7, since it maximizes the error reduction.
Revised: (7) : (up to a (1 − β ) factor) it maximizes the reduction in function value.

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: Using the deﬁnition of Q t
Revised: Using the deﬁnition of Q t

Original: +

Revised: +

Original: we have that,
Revised: we have that,

Original: [Equation]
Revised: [Equation]

Original: Using the Sherman-Morrison formula we have that,
Revised: Using the Sherman-Morrison formula we have that,

Original: [Equation]
Revised: [Equation]

Original: where we recall that  t = µ t 1  µ t .
Revised: where we recall that γ t = µ t 1 − µ t .

Original: Using the well-known matrix determinant lemma for rank-one updates we have,
Revised: Using the well-known matrix determinant lemma for rank-one updates we have,

Original: [Equation]
Revised: [Equation]

Original: Plugging (12) and (13) into (11)
Revised: Plugging (12) and (13) into (11)

Original: [Equation]
Revised: [Equation]

Original: we obtainf ( Q
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Revised: where (a) follows from the inequality log(1 + x ) ≤ x .

Original: We now consider two cases.
Revised: We now consider two cases.

Original: If  t  0 , using the inequality log(1 + x )  x 2 2+ x 1+ x = x  x 2 2(1+ x ) for allx  0 , we have that
Revised: If γ t ≥ 0 , using the inequality log(1 + x ) ≤ x 2 2+ x 1+ x = x − x 2 2(1+ x ) for all x ≥ 0 , we have that

Original: [Equation]
Revised: [Equation]

Original: If  t < 0 , using the inequality log(1 + x )  2 x 2+ x = x  x 2 2+ x for all 0  x >  1 , we have that
Revised: If γ t < 0 , using the inequality log(1 + x ) ≤ 2 x 2+ x = x − x 2 2+ x for all 0 ≥ x > − 1 , we have that

Original: [Equation]
Revised: [Equation]

Original: It is easily veriﬁed that for our choice  t = 
Revised: It is easily veriﬁed that for our choice γ t = − v

Original:  v > t r f ( Q t ) v t ( v > t Q  1 t v t ) 2 , it holds that
Revised: [Equation] 2 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Thus, considering both options for  t (  0 or < 0 ), we have that
Revised: Thus, considering both options for γ t ( ≥ 0 or < 0 ), we have that

Original: [Equation]
Revised: [Equation]

Original: The Lemma follows from considering the two cases | L t |  1 and | L t | < 1 , and simplifying.
Revised: The Lemma follows from considering the two cases | L t | ≥ 1 and | L t | < 1 , and simplifying.

Original: Corollary 1.
Revised: Corollary 1.

Original: Fix ✏ > 0 .
Revised: Fix (cid:15) > 0 .

Original: For any t  ⌃ [Equation] ⌥ iterations of Algorithm 1 itholds that, min ⌧ =0 ,...,t  1 | L ⌧ |  ✏ .
Revised: For any t ≥ (cid:6) 4( f ( Q 0 ) − f ( Q ∗ )) (cid:0) 1 + (cid:15) − 2 (cid:1)(cid:7) iterations of Algorithm 1 it holds that, min τ =0 ,...,t − 1 | L τ | ≤ (cid:15) .

Original: Proof.
Revised: Proof.

Original: Fix some iteration t of Algorithm 1.
Revised: Fix some iteration t of Algorithm 1.

Original: Using Lemma 3 we have that
Revised: Using Lemma 3 we have that

Original: [Equation]
Revised: [Equation]

Original: Thus, for t  ⌃

Original: [Equation]

Original: ( f

Original: [Equation]

Original: Q 0 )

Original: [Equation]

Original: Q

Original: [Equation]  [Equation] ⌥ iterations there must exist some ⌧ 2 { 0 , . . .
Revised: Thus, for t ≥ (cid:6) 4( f ( Q 0 ) − f ( Q ∗ )) (cid:0) 1 + (cid:15) − 2 (cid:1)(cid:7) iterations there must exist some τ ∈ { 0 , . . .

Original: , t  1 }such that | L ⌧ |  ✏ .
Revised: , t − 1 } such that | L τ | ≤ (cid:15) .

Original:  Linear Convergence of AFW and GAFW
Revised:  Linear Convergence of AFW and GAFW

Original: In this section we prove that under an additional (to Assumption 1) mild assumption, the AFW and
Revised: In this section we prove that under an additional (to Assumption 1) mild assumption, the AFW and

Original: GAFW variants converge linearly in function value.
Revised: GAFW variants converge linearly in function value.

Original: Assumption 2.
Revised: Assumption 2.

Original: The data-points x 1 , . .
Revised: The data-points x 1 , . .

Original: , x n satisfy that n  2 p , and for any subset S ✓{ x 1 , . . .
Revised: , x n satisfy that n ≥ 2 p , and for any subset S ⊆ { x 1 , . . .

Original: , x n } such that |S|  n/ 2  it holds that span ( S ) 
Revised: , x n } such that |S| ≥ n/ 2 , it holds that span ( S ) =

Original: = R p .
Revised: R p .

Original: Remark 5.
Revised: Remark 5.

Original: Note that for any ﬁxed n , if the (normalized) data-points are i.i.d.
Revised: Note that for any ﬁxed n , if the (normalized to have unit norms) data-points are i.i.d.

Original: samples from acontinuous distribution which is supported on the entire unit sphere, Assumption 2 holds withprobability 1 .
Revised: samples from a continuous distribution which is supported on the entire unit sphere, Assumptionholds with probability 1 .

Original: Theorem 4 (Linear convergence of AFW and GAFW) .
Revised: Theorem 4 (Linear convergence of AFW and GAFW) .

Original: Denote h t = f
Revised: Denote h t = f

Original: ( Q t )  f ( Q ⇤ ) for all t  0 .
Revised: ( Q t ) − f ( Q ∗ ) for all t ≥ 0 .

Original: Suppose Assumption 2 holds.
Revised: Suppose Assumption 2 holds.

Original: Then, Algorithm 1 when run with AFW steps, satisﬁes
Revised: Then, Algorithm 1 when run with AFW steps, satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where ⇢ > 0 is the constant implied by Theorem 5, and  0 :=
Revised: where ρ > 0 is the constant implied by Theorem 5, and κ 0 :

Original:  max Q 2 S p : f ( Q )  f ( Q 0 )
Revised: = max Q ∈S p : f ( Q ) ≤ f ( Q 0 )

Original:  max ( Q )  min ( Q ) .
Revised: λ max ( Q ) λ min ( Q ) .

Original: When using GAFW steps, Algorithm 1 satisﬁes
Revised: When using GAFW steps, Algorithm 1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Remark 6.
Revised: Remark 6.

Original: As with Theorem 3, we see that GAFW enjoys better conditioning than AFW w.r.t.
Revised: As with Theorem 3, we see that GAFW enjoys better conditioning than AFW w.r.t.

Original: themaximal condition number  max ( Q ) /  min ( Q ) over the initial level set.
Revised: the maximal condition number λ max ( Q ) /λ min ( Q ) over the initial level set.

Original: The complete proof of Theorem 4 is given in the appendix.
Revised: The complete proof of Theorem 4 is given in the appendix.

Original: The key part in the proof is to establishthat, under Assumption 2, Problem (2) satisﬁes a Polyak- Ł ojasiewicz (PL) condition w.r.t.
Revised: The key part in the proof is to establish that, under Assumption 2, Problem (2) satisﬁes a Polyak-Łojasiewicz (PL) condition w.r.t.

Original: boththe standard gradient r f ( Q ) and the geodesic gradient Q 1 / 2 r f ( Q ) Q 1 / 2 , which is a propertywell-known to facilitate linear convergence rates, see for instance [22, 17, 9].
Revised: both the standard gradient ∇ f ( Q ) and the geodesic gradient Q 1 / 2 ∇ f ( Q ) Q 1 / 2 , which is a property well-known to facilitate linear convergence rates for ﬁrst-order methods, see for instance [22, 17, 9].

Original: A (standard, i.e., nongeodesic) PL condition at some query point Q takes the form kr f ( Q ) k 2  C ( f ( Q )  f ( Q ⇤ )) , forsome constant C > 0 independent of Q .
Revised: A (standard, i.e., non geodesic) PL condition at some query point Q takes the form (cid:107)∇ f ( Q ) (cid:107) 2 ≥ C ( f ( Q ) − f ( Q ∗ )) , for some constant C > 0 independent of Q .

Original: The proof of our PL condition is highly non-trivial and is very much inspired by the analysis in[6].
Revised: The proof of our PL condition is highly non-trivial and is very much inspired by the analysis in [6].

Original: However, while the proof in [6] relies heavily on geodesic strong convexity and quantumexpansion arguments, we give a different proof (in particular, our Assumption 2 is different fromtheir explicit assumption that the data is sampled from an elliptical distribution) that does not usesuch considerations and, we believe, is more straightforward and accessible.
Revised: However, while the proof in [6] relies heavily on geodesic strong convexity and quantum expansion arguments, we give a different proof (in particular, our Assumption 2 is different from their explicit assumption that the data is sampled from an elliptical distribution) that does not use such considerations and, we believe, is more straightforward and accessible.

Original: Theorem 5 (Polyak- Ł ojasiewicz condition) .
Revised: Theorem 5 (Polyak-Łojasiewicz condition) .

Original: Suppose Assumption 2 holds, and let Q 0 2 S p + .
Revised: Suppose Assumption 2 holds, and let Q 0 ∈ S p + .

Original: Then,there exists a constant ⇢ > 0 such that for any Q 2 S p + satisfying f ( Q )  f ( Q 0 )  it holds that, [Equation]
Revised: Then, there exists a constant ρ > 0 such that for any Q ∈ S p + satisfying f ( Q ) ≤ f ( Q 0 ) , it holds that, (cid:107) Q 1 / 2 ∇ f ( Q ) Q 1 / 2 (cid:107) 22 ≥ ρ ( f ( Q ) − f ( Q ∗ )) .

Original: Remark 7.
Revised: Remark 7.

Original: While Theorem 5 establishes that the PL parameter ⇢ is strictly positive, its precise value(i.e. , its dependence on the data) is quite intricate and does not admit a simple formula (see proof inappendix for more details).
Revised: While Theorem 5 establishes that the PL parameter ρ is strictly positive, its precise value (i.e., its dependence on the data) is quite intricate and does not admit a simple formula (see proof in appendix for more details).

Original:  Numerical Simulations
Revised:  Numerical Simulations

Original: In order to give some demonstration for the empirical performance of our Frank-Wolfe-basedalgorithms, we conducted two types of experiments that closely follow those in [28] (Chapter 3)with some minor changes, which consider a Gaussian distribution with outlier contamination and aheavy-tailed multivariate t-distribution.
Revised: In order to give some demonstration for the empirical performance of our Frank-Wolfe-based algorithms, we conducted two types of experiments that closely follow those in [28] (Chapter 3) with some minor changes, which consider a Gaussian distribution with outlier contamination, and a heavy-tailed multivariate t-distribution.

Original: We consider a large sample regime in which n = p 2 .
Revised: We consider a large sample regime in which n = p 2 .

Original: Werecall that in this regime each iteration of the Fixed-point Iterations method (FPI) takes O ( np 2 ) time,while our AFW, GAFW variants require only O ( np ) time, up to a single log term, per-iteration.
Revised: We recall that in this regime each iteration of the Fixed-point Iterations method (FPI) takes O ( np 2 ) time, while our AFW, GAFW variants require only O ( np ) time, up to a single log term, per-iteration.

Original: Data generation:
Revised: Data generation:

Original: Following the experiments conducted in [28] with some minor changes, weconsider i. a Gaussian distribution with outliers contamination in which, each Gaussian-distributedvector is replaced with probability 0 .
Revised: Following the experiments conducted in [28] with some minor changes, we consider i. a Gaussian distribution with outliers contamination in which, each Gaussian-distributed vector is replaced with probability 0 .

Original: 9 /p with the eigenvector associated with the smallest eigenvalueof the covariance, and ii.
Revised: 9 /p with the eigenvector associated with the smallest eigenvalue of the covariance, and ii.

Original: a heavy-tailed multivariate t-distribution with two degrees of freedom.
Revised: a heavy-tailed multivariate t-distribution with two degrees of freedom.

Original: Inboth experiments we set p = 50 , n = [-
-] and we take the true unknown covariance to be a Toeplitzmatrix with the elements Q i,j = 0 .[-
-]
Revised: In both experiments we set p = 50 , n = [-
-]2500 , and we take the true unknown covariance to be a Toeplitz matrix with the elements Q i,j = 0 .[-
-]

Original: | i  j | .
Revised: | i − j | .

Original: Methodology:
Revised: Methodology:

Original: To present results that are implementation and scale independent as possible, foreach of the considered methods we estimate the required running time per number of iterations normalized by the data-size np .
Revised: To present results that are implementation and scale independent as possible, for each of the considered methods we estimate the required running time per number of iterations, normalized by the data-size np .

Original: Since we consider the regime n = p 2 , each iteration of FPI takes O ( np 2 ) .
Revised: Since we consider the regime n = p 2 , each iteration of FPI takes O ( np 2 ) time.

Original: For all Frank-Wolfe variants (FW, AFW, and GAFW) we use Python’s SCIPY .
Revised: For all Frank-Wolfe variants (FW, AFW, and GAFW) we use Python’s SCIPY .

Original: SPARSE .
Revised: SPARSE .

Original: LINALG .
Revised: LINALG .

Original: EIGSHprocedure, which is based on the Lanczos algorithm, to solve the corresponding eigenevalue problemto low-accuracy (which corresponds to the approximation parameter  in Algorithm 1).
Revised: EIGSH procedure, which is based on the Lanczos algorithm, to solve the corresponding eigenevalue problem to low-accuracy (which corresponds to the approximation parameter β in Algorithm 1).

Original: We veriﬁedthat in all of our experiments and for all FW variants, this procedure makes at most 2 iterations, andthus the runtime per iteration of each of these variants is estimated by O ( np ) .
Revised: We veriﬁed that in all of our experiments and for all FW variants, this procedure makes at most 2 iterations, and thus, per the discussion in Section 2.1, the runtime per iteration of each of these variants is estimated by O ( np ) .

Original: Thus, normalizingthe estimated runtime by the data-size np , in the ﬁgures below each iteration of FPI is estimated totake p times more than that of any of the FW variants.
Revised: Thus, normalizing the estimated runtime by the data-size np , in the ﬁgures below each iteration of FPI is estimated to take p times more than that of any of the FW variants.

Original: All methods are initialized from the samplecovariance and each ﬁgure is the average of 20 i.i.d.
Revised: All methods are initialized from the sample covariance (normalized to have trace equals p ) and each ﬁgure is the average of 20 i.i.d.

Original: runs.
Revised: experiments.

Original: It can be seen from Figure 1 that for the considered datasets, FPI converges within a few iterationsand thus it is very difﬁcult to beat.
Revised: Results: For all experiments we compute Tyler’s estimator to high accuracy by runningiterations of the Fixed-point method and we denote the resulting matrix by Q ∗ . In Figure 1 we report the distance in spectral norm (in log scale) of the iterates of the different methods form Q ∗ , and in Figure 2 we report the approximation error f ( Q t ) − f ( Q ∗ ) of the iterates (also in log scale).

Original: Nevertheless, we see thatour GAFW variant, in particular in thecontaminated Gaussian model, is signiﬁcantly faster than FPI, and reduces the  error substentiallybefore FPI completes even a single iteration.
Revised: It can be seen that in both setups and with respect to both measures, GAFW converges faster than FPI, and that in the contaminated Gaussian distribution setup it is signiﬁcantly faster. Moreover, looking at the approximation error in log-scale, it indeed seems to exhibit a linear convergence rate. We also observe that the FW and AFW variants converge signiﬁcantly slower than GAFW, which demonstrates how the better conditioning of GAFW (as captured also in our convergence theorems,

Original: In both experiments we also see, perhaps surprisingly,that while AFW is faster at ﬁrst than FW,  inthe long run, FW seems to converge faster, which couldbe due to the small feasible step-sizes when using “away directions” (i.e., when the step-size µ t isnegative).
Revised: Theorems 3 and 4) may be signiﬁcant. Additionally, and perhaps surprisingly, AFW converges slower than FW, which suggests a situation in which AFW takes many away-steps , since they are better descent directions than the standard FW step, but in turn results in much smaller step-sizes, and thus overall, the convergence is slower.

Original:  Conclusions
Revised:  Conclusions

Original: We have presented the ﬁrst Frank-Wolfe-based variants for approximating Tyler’s M-estimator forrobust and heavy-tailed covariance estimation.
Revised: We have presented the ﬁrst Frank-Wolfe-based variants for approximating Tyler’s M-estimator for robust and heavy-tailed covariance estimation.

Original: In particular these include parameter-free and globallyconvergent variants with nearly linear runtime per-iteration and, under a mild assumption, withlinear convergence rates, despite the fact that the underlying optimization problem is not convexnor smooth.
Revised: In particular these include parameter-free and globallyconvergent variants with nearly linear runtime per-iteration and, under a mild assumption, with linear convergence rates, despite the fact that the underlying optimization problem is not convex nor smooth.

Original: We hope these results will pave the way to nearly linear-time algorithms for additionalhighly-structured nonconvex and nonsmooth problems. 
Revised: We hope these results will pave the way to nearly linear-time algorithms for additional highly-structured nonconvex and nonsmooth problems. A Proofs Missing from SectionA.1 Proof of Lemma

Revised: Before proving the lemma we need a simple observation.

Original: Checklist
Revised: Observation 2.

Original: For all authors...
Revised: For any Q ∈ S p + and any vector v ∈ R p such that (cid:107) v (cid:107) = √ p , we have that [Equation]

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: Proof. Since Q (cid:31) 0 , using the expression for ∇ f ( Q ) in Eq. (3), we have that

Original: contributions and scope?
Revised: [Equation] Thus, for the condition stated in the observation to hold true, it sufﬁces that v (cid:62) Q − 1 v > 1 . Since (cid:107) v (cid:107) = √ p we have that

Original: [Yes]
Revised: [Equation]

Original: (b) Did you describe the limitations of your work?
Revised: p , where the last inequality follows since Tr ( Q ) = p and Q (cid:31) 0 , which imply that λ max ( Q ) < p . Thus, the observation is indeed correct. Proof of Lemma 2. The fact that for all t it holds that Tr ( Q t )

Original: [Yes]
Revised: = p follows immediately from the deﬁnition of Q t +1 in Algorithm 1 and the fact that Tr ( v t v (cid:62) t ) = p . In order to show that for all t , Q t (cid:31) 0 , we consider two cases. First, suppose that v (cid:62) t ∇ f ( Q t ) v t < 0 .

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: In this case we have from the deﬁnition of µ t that µ t ∈ [0 , 1] . In particular, if Q t (cid:31) 0 we have that µ t ∈ [0 , 1) and so, by the deﬁnition of Q t +1 it follows that Q t +1 (cid:31) 0 as well. For the second case in which v (cid:62) t ∇ f ( Q t ) v t > 0 , using the assumption that Q t (cid:31) 0 and Observation 2, we ﬁrst note that µ t < 0 , and so (1 − µ t ) Q t (cid:31) 0 . Denoting A t = (1 − µ t ) Q t , we make the following observations:

Original: [N/A]
Revised: [Equation]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: Note that µ t 1 − µ t = − v [Equation] and [Equation]

Original: them?
Revised: The RHS is non-negative and it is equal to zero if and only if for all i = 1 , . . . , n it holds that x i ⊥ Q − 1 t v t , however this implies that there exists a subspace of dimension p − 1 containing all n data-points, which is in contrast to Assumption 1. A.2 Proof of Theorem Proof.

Original: [Yes]
Revised: We ﬁrst focus on the efﬁcient implementation of the AFW step (Eq. and then show how it relates to computing the GAFW step (Eq. Then, we discuss the efﬁcient implementation of the FW step (Eq. Fix some iteration t of Algorithm 1.

Revised: We reduce the computation of v t which satisﬁes (6) to three approximate leading eigenvalue computations.

Revised: Fix some ˜ β ∈ [0 , 1) to be determined later.

Revised: First, we compute a unit vector u such that u (cid:62) ( ∇ f ( Q t )) 2 u ≥ (1 − ˜ β ) 2 λ 1 (( ∇ f ( Q t )) 2 ) =

Revised: (1 − ˜ β ) 2 (cid:107)∇ f ( Q t ) (cid:107) 22 , and we compute C =

Revised: (cid:112) u (cid:62) ( ∇ f ( Q t )) 2 u .

Revised: Next we deﬁne the matrices

Revised: M + = (1 − ˜ β ) − 1 C I + ∇ f ( Q t ) , M − = (1 − ˜ β ) − 1 C I − ∇ f ( Q t ) .

Revised: Note that by deﬁnition, both

Revised: M + , M − are positive semideﬁnite.

Revised: Next we compute unit vectors u + , u − , which are approximate leading eigenvectors, with a factor (1 − ˜ β ) multiplicative approximation, of the matrices M + , M − , respectively.

Revised: That is,

Revised: [Equation]

Original: If you are including theoretical results...
Revised: , which using the deﬁnition of M + , M − implies that [Equation] Thus, [Equation]

Revised: where in (a) we have used the fact that by deﬁnition C ≤ (cid:107)∇ f ( Q t ) (cid:107) 2 .

Revised: Thus, setting ˜ β such that 1 − 3˜ β 1 − ˜ β ≥ 1 − β and taking v

Revised: [Equation]

Revised: we obtain the result required by (6).

Revised: Since for a universal constant β , ˜ β as deﬁned above is also a universal constant, computing each approximated eigenvector u , u + , u − using the well-known power method with random initialization requires O (cid:0) log pδ (cid:1) iterations, where δ is the desired failure probability, see for instance [8] (Theorem

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: A.1). Each such iteration of the power method requires to compute a matrix-vector product with a matrix of the form c I ± ∇ f ( Q t ) , where c is a given scalar. Thus, it remains to detail how to compute fast matrix-vector products with the gradient ∇ f ( Q t ) .

Revised: Fix some vector v .

Revised: It holds that

Original: [Yes]
Revised: [Equation]

Revised: Note that if the vectors y t,i := Q − 1 t x

Revised: i , i = 1 , . . .

Revised: , n are stored explicitly in memory, and recalling that

Revised: Q − 1 t is also computed and stored explicitly, computing ∇ f ( Q t ) v requires overall only O ( np + p 2 ) time.

Revised: It thus remains to show how given the vectors y t,i , i = 1 , . . . , n , and the vector v t , we can quickly compute the new vectors y t

Original: (b) Did you include complete proofs of all theoretical results?
Revised: +1 , i := Q − 1 t +1 x i , i = 1 , . . . , n .

Revised: Using the Sherman-Morrison formula we have that,

Original: [Yes]
Revised: [Equation]

Revised: where we recall that γ t = µ t 1 − µ t .

Revised: Thus, for all i = 1 , . . .

Revised: , n we have that,

Original: If you ran experiments...
Revised: [Equation] Thus, after computing Q − 1 t v t , we can indeed compute y t +1 ,i form y t,i in O ( p ) time, and overall O ( np + p 2 ) to compute all vectors y t +1 ,i , i = 1 , . . . , n .

Revised: We now turn to discuss the efﬁcient computation of the GAFW step in Eq.

Revised: As discussed in Section 2, computing v t in this case amounts to ﬁnding a unit vector u such that | u (cid:62) Q 1 / 2 ∇ f ( Q t ) Q 1 / 2 u | ≥ (1 − β ) (cid:107) Q 1 / 2 t ∇ f ( Q t ) Q 1 / 2 t (cid:107) 2 , and returning v t = √ p Q 1 / 2 t u (cid:107) Q 1 / 2 t u (cid:107) .

Revised: Thus, given the matrix Q 1 / 2 t explicitly, computing such unit vector u , and then the corresponding vector v t , could be carried out in time ˜ O ( p 2 + np ) using the same reasoning as that in the computation of the AFW step.

Revised: In particular, each matrix-vector product of the form Q 1 / 2 t ∇ f ( Q t ) Q 1 / 2 t v could be carried out in O ( p 2 + np ) times as explained above.

Original: (a) Did you include the code, data, and instructions needed toreproduce the main experi-
Revised: Thus, using additional O ( p 3 ) to explicitly compute the matrix Q 1 / 2 t , we obtain the result of the theorem for the GAFW updates. Finally, for the FW updates (Eq. , we compute the constant C and deﬁne the matrix M − as before, but this time we compute the vector u − (there is no need for u + in the standard FW step) so that u (cid:62)− M − u − ≥ (1 − ˆ β ) λ 1 ( M − ) , for some ˆ β ∈ [0 , 1) , to be determined shortly. Similarly to the analysis for AFW, this yields,

Original: mental results (either in the supplemental material or as a URL)?
Revised: [Equation] Thus, choosing for instance ˜ β = 1 / 2 and ˆ β = | λ p ( ∇ f ( Q t )) | 3 (cid:107)∇ f ( Q t ) (cid:107) 2 β , we indeed have that u (cid:62)− ∇ f ( Q t ) u − ≤ (1 − β ) λ p ( ∇ f ( Q t )) , and so, returning v t = √ p u − indeed satisﬁes (5).

Original: {+Using standard results for the power method (see for instance Theorem A.1 in [8]), as discusses above, computing such u − with failure probability at most δ requires O (cid:16) (cid:107)∇ f ( Q t ) (cid:107) 
+}
Revised: {+Using standard results for the power method (see for instance Theorem A.1 in [8]), as discusses above, computing such u − with failure probability at most δ requires O (cid:16) (cid:107)∇ f ( Q t ) (cid:107) 
+}

Revised: | λ p ( ∇ f ( Q t )) | β log( pδ − 1 ) (cid:17) matrixvector products.

Original: {+This could be improved to only O (cid:16)(cid:113) (cid:107)∇ f ( Q t ) (cid:107) 
+}
Revised: {+This could be improved to only O (cid:16)(cid:113) (cid:107)∇ f ( Q t ) (cid:107) 
+}

Revised: | λ p ( ∇ f ( Q t )) | β log( pδ − 1 ) (cid:17) matrix-vector products using the faster Lanczos method [12, 23, 21].

Revised: B Proofs Missing from SectionB.1 Proof of Observation

Revised: Proof.

Original: [Yes] The details in{+We ﬁrst establish that λ min ( ∇ f ( Q )) ≤ 0 and λ min ( ∇ f ( Q )) = 
 if and only if Q = Q ∗ . Then, the observation follows since Q (cid:31) 0 and Q ∇ f ( Q ) Q = Q − p n (cid:80) n i =+}
Revised: {+We ﬁrst establish that λ min ( ∇ f ( Q )) ≤ 0 and λ min ( ∇ f ( Q )) = 
 if and only if Q = Q ∗ . Then, the observation follows since Q (cid:31) 0 and Q ∇ f ( Q ) Q = Q − p n (cid:80) n i =+}

Revised: x i x (cid:62) i x (cid:62)

Revised: [Equation]

Revised: −

Revised: [Equation]

Original:  the experimental section allow to reproduce the results exactly{+eigen-decomposition of ∇ f ( Q ) as ∇ f ( Q ) = (cid:80) pi =
+}
Revised: A straight-forward calculation shows that for any Q ∈ S p + it holds that (cid:104) Q , ∇ f ( Q ) (cid:105) = 0 . Writing the {+eigen-decomposition of ∇ f ( Q ) as ∇ f ( Q ) = (cid:80) pi =
+}

Original: (b) Did you specify all the training details (e
Revised: λ i u i u (cid:62) i , this implies that 0 = (cid:104) Q , ∇ f ( Q ) (cid:105) = p (cid:88) i

Revised: =

Revised: λ i u (cid:62)

Revised: i Qu i .

Revised: Since Q ∈ S p + implies that for all i = 1 , . .

Original: {+, p , u (cid:62) i Qu i > 0 , it follows that λ p ≤ 0 , and moreover, λ p = 
+}
Revised: {+, p , u (cid:62) i Qu i > 0 , it follows that λ p ≤ 0 , and moreover, λ p = 
+}

Original: {+if and only if λ i = 
+}
Revised: {+if and only if λ i = 
+}

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: for all i = 1 , . . . , p , meaning ∇ f ( Q ) = 0 , which holds if and only if Q satisﬁes (1), i.e., Q = Q ∗ . B.2 Proof of Theorem Proof.

Revised: Fix ˜ (cid:15) > 0 .

Revised: Corollary 1 establishes that for all variants and for all t ≥ T (˜ (cid:15) ) , it holds that min τ =0 ,...,t − 1 | L τ | ≤ ˜ (cid:15) .

Revised: Let us denote by t ∗ an index of such iteration for which it holds that | L t ∗ | ≤ ˜ (cid:15) .

Revised: Let us begin with FW steps.

Revised: By deﬁnition we have that

Revised: [Equation]

Revised: Thus, for ˜ (cid:15) =

Revised: ˜ (cid:15) FW and t

Revised: = t ∗ we indeed have that (8) holds.

Revised: We now turn to prove (9) holds for AFW and GAFW updates.

Revised: For AFW updates, using similar arguments as before, it holds that

Revised: [Equation]

Revised: Thus, for ˜ (cid:15) =

Revised: ˜ (cid:15) AFW and t = t ∗ we indeed have that (9) holds for AFW.

Revised: We turn to prove that (9) holds for GAFW updates.

Revised: According to the update rule we have that,

Revised: [Equation]

Revised: Thus, for ˜ (cid:15) =

Revised: ˜ (cid:15) GAFW and t = t ∗ we have that (9) holds for GAFW.

Revised: C Proofs Missing from SectionC.1 Proof of Theorem

Revised: Before we can prove Theorem 5 we need to establish several auxiliary results.

Revised: Lemma 4.

Revised: Under Assumption 2, there exist a constant α > 0 such that for any pair of unit vectors u , v in R p , it holds that 1 n (cid:80) ni =

Original: [Yes]
Revised: [Equation]

Original: (c) Did you report error bars (e
Revised: u [Equation] x i ) [Equation]

Original: g., with respect to the random seed after running experi-
Revised: v [Equation] x i ) 2 ≥ α . Proof.

Original: ments multiple times)?
Revised: Let u ∗ , v ∗ be the optimal solutions to the optimization problem: [Equation] and suppose by way of contradiction that α = 0 . Thus, for all i = 1 , . . . , n , x i ⊥ u ∗ or x i ⊥ v ∗ , which in particular implies that there exists

Original: [No]
Revised: S ⊆ { x 1 , . . . , x n } such that |S| ≥ n/ 2 and either u ∗ or v ∗ are orthogonal to all vectors in S , which implies orthogonality to span ( S ) . However, according to Assumption 2, span ( S ) = R p , and thus we arrive at a contradiction.

Revised: Lemma 5.

Revised: Suppose Assumption 2 holds.

Revised: Consider the function

Revised: [Equation]

Revised: for some Z (cid:31) 0 , and W ∈ S p such that Tr ( W ) = 0 and (cid:107) W (cid:107) 1 = 1 .

Revised: Then,

Revised: [Equation]

Revised: where α > 0 is the constant implied by Lemma 4.

Revised: Proof.

Original: (d) Did you include the total amount of compute and the type of resources used (e{+W as W = (cid:80) pj =
 λ j u j u (cid:62) j . Calculations give [Equation]+}
Revised: Write the eigen-decomposition of {+W as W = (cid:80) pj =
 λ j u j u (cid:62) j . Calculations give [Equation]+}

Revised: Thus, g

Revised: [Equation]

Revised: which implies that,

Revised: [Equation]

Revised: Let us introduce the notation C ij = exp( − λ j t )( x i (cid:62) Z − 12 u j ) 2 for all i ∈ { 1 , . . . , n } , j ∈ { 1 , . .

Revised: . , p } .

Revised: We have that

Revised: [Equation]

Revised: Re-arranging we get,

Revised: [Equation]

Revised: Let us ﬁx some i, j , and recall that C ij = exp( − λ j t )

Revised: ( x i (cid:62) Z − 1 2 u j ) 2 .

Revised: Since (cid:107) W (cid:107) 2 ≤ (cid:107) W (cid:107)

Revised: [Equation]

Revised: = 1 , we have that

Revised: [Equation]

Revised: Thus, for any j, k = 1 , . . .

Revised: , p we have that,

Revised: [Equation]

Revised: where the last inequality is due to Lemma 4.

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Introducing the notation δ ( Z ) = λ max ( Z ) λ min ( Z ) [Equation] we ﬁnally we get,

Revised: [Equation]

Revised: where in (a) we have used the assumption that Tr ( W ) = 0 , and in (b) we have used the assumption that (cid:107) W (cid:107) 1 = 1 .

Revised: Lemma 6.

Revised: Suppose Assumption 2 holds.

Revised: Let Z ∈ S ++ such that det( Z ) = 1 , and let Z ∗ (cid:31) 0 be a minimizer of f ( · ) such that det( Z ∗ ) =

Revised: It holds that

Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: Z ) − f

Revised: ( Z

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: where κ ( Z ) := λ max ( Z ) /λ min ( Z ) , D ( Z , Z ∗ ) :

Revised: =

Revised: || log( Z − 12 Z ∗ Z − 12 )

Revised: || 1 ,

Revised: and α > 0 is the constant implied by Lemma 4.

Revised: Proof.

Revised: Let W = log

Revised: ( Z − 1 / 2 Z ∗ Z − 1 / 2 ) D , where D =

Revised: D

Revised: ( Z , Z ∗ ) =

Revised: || log( Z − 1 / 2 Z ∗

Revised: Z − 1 / 2 )

Revised: || 1 .

Revised: Note that

Revised: [Equation]

Revised: where the last equality is due to the fact that det( Z ) = det( Z ∗ ) = 1 .

Revised: Note also that by deﬁnition (cid:107) W (cid:107) 1 = 1 .

Revised: Denote the function

Original: [N/A]
Revised: [Equation]

Original: If you are using existing assets (e
Revised: and note that g (0) = f ( Z ) , g ( D ) = f ( Z ∗ ) .

Original: g., code, data, models) or curating/releasing new assets...
Revised: Thus, using Lemma 5, and denoting the constant A = κ − 2 ( Z ) p − 1 exp( − 4 D ( Z , Z ∗ )) α , we have that, [Equation]

Original: If your work uses existing assets, did you cite the creators?
Revised: Integrating on both sides we have that, [Equation] Integrating again we have,

Original: [N/A]
Revised: [Equation]

Original: ) Did you mention the license of the assets?
Revised: It holds that [Equation] where the last inequality is due to Hölder’s inequality and the fact that || W || 1 = 1 . Plugging-back we have that,

Original: [N/A]
Revised: [Equation]

Original: Did you include any new assets either in the supplemental material or as a URL?
Revised: The right hand side obtains its minimal value for t ∗ = (cid:107) Z 1 / 2 ∇ f ( Z ) Z

Original: [N/A]
Revised: [Equation]

Original: Did you discuss whether and how consent was obtained from people whose data you’re
Revised: 2 (cid:107) 2 A , which yields,

Original: using/curating?
Revised: [Equation] and speciﬁcally,

Original: [N/A]
Revised: [Equation]

Original: Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: which concludes the proof. We can now prove Theorem

Original: information
Revised: Proof of Theorem 5.

Original: or offensive content?
Revised: Given Q 0 and Q , deﬁne their unit-determinant normalizations Z 0 :=

Original: [N/A
Revised: det − 1 /p ( Q 0 ) Q 0 , Z := det − 1 /p ( Q ) Q . Denote also Z ∗ = det − 1 /p ( Q ∗ ) Q ∗ (recall that since f ( · ) is invariant to scaling, Z ∗ also minimizes f ( · ) ). From Lemma 6 we have that,

Revised: [Equation]

Revised: where κ ( Z ) := λ max ( Z ) /λ min ( Z ) , D ( Z , Z ∗ ) :

Revised: =

Revised: || log( Z − 12 Z ∗ Z − 12 )

Revised: || 1 ,

Revised: and α > 0 is the constant implied by Lemma 4.

Revised: Using the fact that the functions f ( Z ) , κ ( Z ) are invariant to scaling of Z , and so is the matrix

Revised: Z 1 / 2 ∇ f ( Z ) Z 1 / 2 , we have that

Revised: [Equation]

Revised: Recall that according to Lemma 1, for every Q in the level set { Q ∈ S p | f ( Q ) ≤ f ( Q 0 ) } , λ min ( Q ) > λ , for some λ > 0 .

Revised: This further implies that

Original: If you
Revised: [Equation] is upper-bounded by some constant on the level-set { M ∈ S p | f ( M ) ≤ f ( Q 0 ) } , which yields the corollary. C.2 Proof of Theorem Proof.

Original: used crowdsourcing or conducted research with human subjects..
Revised: Let us ﬁrst consider the case of AFW steps.

Original: Did you include the full text of instructions given to
Revised: From Lemma 3 we have that out of t iterations

Original: participants and screenshots ifapplicable
Revised: Algorithm 1 has executed, the maximum number of iterations τ in which it holds that L 2 τ > 1 , is at most (cid:100) 4( f ( Q 0 ) − f ( Q ∗ )) (cid:101) = (cid:100) 4 h 0 (cid:101) , where we recall that L τ :=

Original: [N/A
Revised: [Table]

Original: participant Institutional Review approvals, applicable
Revised: On any other iteration τ we have that,

Original: [N A
Revised: [Equation]

Original:  the estimated hourly wage paid
Revised: where (a) follows from the deﬁnition of L τ and the AFW update step (Eq. (6)), and (b) follows from Theorem 5. Rearranging, we have that

Original: participants
Revised: [Equation]

Original: the
Revised: Thus, we can conclude that,

Original: amount
Revised: [Equation]

Original: spent on participant compensation
Revised: The result for GAFW steps follows from the same reasoning, only now using the GAFW update step (Eq. (7)) which gives the improved bound:

Original: A
Revised: [Equation]