Original: S UBGRADIENT D ESCENT L EARNS O RTHOGONAL D ICTIONARIES
Revised: S UBGRADIENT D ESCENT L EARNS O RTHOGONAL D ICTIONARIES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper concerns dictionary learning, i e., sparse coding, a fundamental representation learning problem.
Revised: This paper concerns dictionary learning, i.e., sparse coding, a fundamental representation learning problem.

Original: We show that a subgradient descent algorithm, with random initialization, can recover orthogonal dictionaries on a natural nonsmooth, nonconvex (cid:96) 1 minimization formulation of the problem, under mild statistical assumption on the data.
Revised: We show that a subgradient descent algorithm, with random initialization, can recover orthogonal dictionaries on a natural nonsmooth, nonconvex (cid:96) 1 minimization formulation of the problem, under mild statistical assumption on the data.

Original: This is in contrast to previous provable methods that require either expensive computation or delicate initialization schemes.
Revised: This is in contrast to previous provable methods that require either expensive computation or delicate initialization schemes.

Original: Our analysis develops several tools for characterizing landscapes of nonsmooth functions, which might be of independent interest for provable training of deep networks with nonsmooth activations (e g., ReLU), among other applications.
Revised: Our analysis develops several tools for characterizing landscapes of nonsmooth functions, which might be of independent interest for provable training of deep networks with nonsmooth activations (e.g., ReLU), among other applications.

Original: Preliminary synthetic and real experiments corroborate our analysis and show that our algorithm works well empirically in recovering orthogonal dictionaries
Revised: Preliminary synthetic and real experiments corroborate our analysis and show that our algorithm works well empirically in recovering orthogonal dictionaries.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Dictionary learning (DL), i.e. , sparse coding, concerns the problem of learning compact representations, i.e., given data Y , one tries to ﬁnd a representation basis A and coefﬁcients X , so that
Revised: Dictionary learning (DL), i e. , sparse coding, concerns the problem of learning compact representations, i e., given data Y , one tries to ﬁnd a representation basis A and coefﬁcients X , so that

Original: Y ≈ AX where X is most sparse.
Revised: Y ≈ AX where X is most sparse.

Original: DL has numerous applications especially in image processing and computer vision (Mairal et al., 2014).
Revised: DL has numerous applications especially in image processing and computer vision (Mairal et al., 2014).

Original: When posed in analytical form, DL seeks a transformation
Revised: When posed in analytical form, DL seeks a transformation

Original: Q such that QY is sparse; in this sense DL can be considered as an (extremely!)
Revised: Q such that QY is sparse; in this sense DL can be considered as an (extremely!)

Original: primitive “deep” network (Ravishankar & Bresler, 2013).
Revised: primitive “deep” network (Ravishankar & Bresler, 2013).

Original: Many heuristic algorithms have been proposed to solve DL since the seminal work of Olshausen &
Revised: Many heuristic algorithms have been proposed to solve DL since the seminal work of Olshausen &

Original: Field (1996), most of them surprisingly effective in practice (Mairal et al., 2014; Sun et al., 2015).
Revised: Field (1996), most of them surprisingly effective in practice (Mairal et al., 2014; Sun et al., 2015).

Original: However, understandings on when and how DL is solvable have only recently started to emerge.
Revised: However, understandings on when and how DL is solvable have only recently started to emerge.

Original: Under appropriate generating models on A and X , Spielman et al.
Revised: Under appropriate generating models on A and X , Spielman et al.

Original: (2012) showed that complete (i.e., square, invertible)
Revised: (2012) showed that complete (i e., square, invertible)

Original: A can be recovered from Y , provided that X is ultra-sparse.
Revised: A can be recovered from Y , provided that X is ultra-sparse.

Original: Subsequent works (Agarwal et al., 2017; Arora et al., 2014; 2015; Chatterji & Bartlett, 2017; Awasthi & Vijayaraghavan, 2018) provided similar guarantees for overcomplete (i.e. fat)
Revised: Subsequent works (Agarwal et al., 2017; Arora et al., 2014; 2015; Chatterji & Bartlett, 2017; Awasthi & Vijayaraghavan, 2018) provided similar guarantees for overcomplete (i.e. fat)

Original: A , again in the ultra-sparse regime.
Revised: A , again in the ultra-sparse regime.

Original: The latter methods are invariably based on nonconvex optimization with model-dependent initialization, rendering their practicality on real data questionable.
Revised: The latter methods are invariably based on nonconvex optimization with model-dependent initialization, rendering their practicality on real data questionable.

Original: The ensuing developments have focused on breaking the sparsity barrier and addressing the practicality issue.
Revised: The ensuing developments have focused on breaking the sparsity barrier and addressing the practicality issue.

Original: Convex relaxations based on the sum-of-squares (SOS)
Revised: Convex relaxations based on the sum-of-squares (SOS)

Original: SDP hierarchy can recover overcomplete A when X has linear sparsity (Barak et al., 2015; Ma et al., 2016; Schramm &
Revised: SDP hierarchy can recover overcomplete A when X has linear sparsity (Barak et al., 2015; Ma et al., 2016; Schramm &

Original: Steurer, 2017), while incurring expensive computation (solving large-scale SDP’s or large-scale tensor decomposition).
Revised: Steurer, 2017), while incurring expensive computation (solving large-scale SDP’s or large-scale tensor decomposition).

Original: By contrast, Sun et al. (2015) showed that complete A can be recovered in the linear sparsity regime by solving a certain nonconvex problem with arbitrary initialization.
Revised: By contrast, Sun et al. (2015) showed that complete A can be recovered in the linear sparsity regime by solving a certain nonconvex problem with arbitrary initialization.

Original: However, the second-order optimization method proposed there is still expensive.
Revised: However, the second-order optimization method proposed there is still expensive.

Original: This problem is partially addressed by (Gilboa et al., 2018) which proved that the ﬁrst-order gradient descent with random initialization enjoys a similar performance guarantee.
Revised: This problem is partially addressed by (Gilboa et al., 2018) which proved that the ﬁrst-order gradient descent with random initialization enjoys a similar performance guarantee.

Original: A standing barrier toward practicality is dealing with nonsmooth functions.
Revised: A standing barrier toward practicality is dealing with nonsmooth functions.

Original: To promote sparsity in the coefﬁcients, the (cid:96) 1 norm is the function of choice in practical DL, as is common in modern signal processing and machine learning (Cand`es, 2014): despite its nonsmoothness, this choice often admits highly scalable numerical methods, such as proximal gradient method and alternating direction method (Mairal et al., 2014).
Revised: To promote sparsity in the coefﬁcients, the (cid:96) 1 norm is the function of choice in practical DL, as is common in modern signal processing and machine learning (Cand`es, 2014): despite its nonsmoothness, this choice often admits highly scalable numerical methods, such as proximal gradient method and alternating direction method (Mairal et al., 2014).

Original: The analyses in Sun et al. (2015); Gilboa et al.
Revised: The analyses in Sun et al. (2015); Gilboa et al.

Original: (2018), however, focused on characterizing the algorithm-independent function landscape of a certain nonconvex formulation of DL, which takes a smooth surrogate to (cid:96) 1 to get around the nonsmoothness.
Revised: (2018), however, focused on characterizing the algorithm-independent function landscape of a certain nonconvex formulation of DL, which takes a smooth surrogate to (cid:96) 1 to get around the nonsmoothness.

Original: The tactic smoothing there introduced substantial analysis difﬁculty, and broke the practical advantage of computing with the simple (cid:96) 1 function.
Revised: The tactic smoothing there introduced substantial analysis difﬁculty, and broke the practical advantage of computing with the simple (cid:96) 1 function.

Original: In this paper, we show that working directly with a natural (cid:96) 1 norm formulation results in neat analysis and a practical algorithm.
Revised: In this paper, we show that working directly with a natural (cid:96) 1 norm formulation results in neat analysis and a practical algorithm.

Original: We focus on the problem of learning orthogonal dictionaries: given data { y i } i ∈ [ m ] generated as y i
Revised: We focus on the problem of learning orthogonal dictionaries: given data { y i } i ∈ [ m ] generated as y i

Original: =
Revised: =

Original: Ax i , where A ∈ R n × n is a ﬁxed unknown orthogonal matrix and each x i ∈ R n is an iid Bernoulli-Gaussian random vector with parameter θ ∈ (0 , 1) , recover A .
Revised: Ax i , where A ∈ R n × n is a ﬁxed unknown orthogonal matrix and each x i ∈ R n is an iid Bernoulli-Gaussian random vector with parameter θ ∈ (0 , 1) , recover A .

Original: This statistical model is the same as in previous works (Spielman et al., 2012; Sun et al., 2015).
Revised: This statistical model is the same as in previous works (Spielman et al., 2012; Sun et al., 2015).

Original: Write Y .
Revised: Write Y .

Original: =
Revised: =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y m ] and similarly X .
Revised: , y m ] and similarly X .

Original: =
Revised: =

Original: [ x 1 , . . .
Revised: [ x 1 , . . .

Original: , x m ] .
Revised: , x m ] .

Original: We propose to recover A by solving the following nonconvex (due to the constraint), nonsmooth (due to the objective) optimization problem: minimize 
Revised: We propose to recover A by solving the following nonconvex (due to the constraint), nonsmooth (due to the objective) optimization problem: minimize q ∈ R n f ( q

Original: [Equation]
Revised: [Equation]

Original: subject to (cid:107) q (cid:107)
Revised: m [Equation] m m (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original:  = 1 .
Revised: | q (cid:62) y i | subject to (cid:107) q (cid:107) 2 = 1 .

Original: Based on the statistical model, q (cid:62) Y = q (cid:62) AX has the highest sparsity when q is a column of A (up to sign) so that q (cid:62) A is 1 -sparse.
Revised: Based on the statistical model, q (cid:62) Y = q (cid:62) AX has the highest sparsity when q is a column of A (up to sign) so that q (cid:62) A is 1 -sparse.

Original: Spielman et al.
Revised: Spielman et al.

Original: (2012) formalized this intuition and optimized the same objective as Eq.
Revised: (2012) formalized this intuition and optimized the same objective as Eq.

Original: (1.1) with a (cid:107) q (cid:107) ∞ = 1 constraint, which only works when θ ∼ O (1 / √ n ) .
Revised: (1.1) with a (cid:107) q (cid:107) ∞ = 1 constraint, which only works when θ ∼ O (1 / √ n ) .

Original: Sun et al.
Revised: Sun et al.

Original: (2015) worked with the sphere constraint but replaced the (cid:96) 1 objective with a smooth surrogate, introducing substantial analytical and computational deﬁciencies as alluded above.
Revised: (2015) worked with the sphere constraint but replaced the (cid:96) 1 objective with a smooth surrogate, introducing substantial analytical and computational deﬁciencies as alluded above.

Original: In constrast, we show that with sufﬁciently many samples, the optimization landscape of formulation (1.1) is benign with high probability (over the randomness of X ), and a simple Riemannian subgradient descent algorithm can provably recover A in polynomial time.
Revised: In constrast, we show that with sufﬁciently many samples, the optimization landscape of formulation (1.1) is benign with high probability (over the randomness of X ), and a simple Riemannian subgradient descent algorithm can provably recover A in polynomial time.

Original: Theorem 1.1 (Main result, informal version of Theorem 3.1) .
Revised: Theorem 1.1 (Main result, informal version of Theorem 3.1) .

Original: Assume θ ∈
Revised: Assume θ ∈

Original: [1 /n, 1 / 2] .
Revised: [1 /n, 1 / 2] .

Original: For m ≥ Ω( θ − 2 n 4 log 4 n ) , the following holds with high probability: there exists a poly( m, (cid:15) − 1 ) - time algorithm, which runs Riemannian subgradient descent on formulation (1.1) from at most
Revised: For m ≥ Ω( θ − 2 n 4 log 4 n ) , the following holds with high probability: there exists a poly( m, (cid:15) − 1 ) - time algorithm, which runs Riemannian subgradient descent on formulation (1.1) from at most

Original: O ( n log n ) independent, uniformly random initial points, and outputs a set of vectors { (cid:98) a 1 , . .
Revised: O ( n log n ) independent, uniformly random initial points, and outputs a set of vectors { (cid:98) a 1 , . .

Original: , (cid:98) a n } such that up to permutation and sign change, (cid:107) (cid:98) a
Revised: , (cid:98) a n } such that up to permutation and sign change, (cid:107) (cid:98) a

Original: i − a i (cid:107) 2 ≤ (cid:15) for all i ∈ [ n ] .
Revised: i − a i (cid:107) 2 ≤ (cid:15) for all i ∈ [ n ] .

Original: In words, our algorithm works also in the linear sparsity regime, the same as established in Sun et al.
Revised: In words, our algorithm works also in the linear sparsity regime, the same as established in Sun et al.

Original: ; Gilboa et al.
Revised: ; Gilboa et al.

Original: (2018), at a lower sample complexity O ( n 4 ) in contrast to the existing
Revised: (2018), at a lower sample complexity O ( n 4 ) in contrast to the existing

Original: O ( n 5 . 5 ) in Sun et al.
Revised: O ( n 5 . 5 ) in Sun et al.

Original: 1 As for the landscape, we show that (Theorems 3.4 and 3.6) each of the desired solutions {± a i } i ∈
Revised: 1 As for the landscape, we show that (Theorems 3.4 and 3.6) each of the desired solutions {± a i } i ∈

Original: [ n ] is a local minimizer of formulation (1.1) with a sufﬁciently large basin of attraction so that a random initialization will land into one of the basins with at least constant probability.
Revised: [ n ] is a local minimizer of formulation (1.1) with a sufﬁciently large basin of attraction so that a random initialization will land into one of the basins with at least constant probability.

Original: To obtain the result, we integrate and develop elements from nonsmooth analysis (on
Revised: To obtain the result, we integrate and develop elements from nonsmooth analysis (on

Original: Riemannian manifolds), set-valued analysis, and random set theory, which might be valuable to studying other nonconvex, nonsmooth optimization problems.
Revised: Riemannian manifolds), set-valued analysis, and random set theory, which might be valuable to studying other nonconvex, nonsmooth optimization problems.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Dictionary learning Besides the many results sampled above, we highlight similarities of our result to Gilboa et al.
Revised: Dictionary learning Besides the many results sampled above, we highlight similarities of our result to Gilboa et al.

Original: Both propose ﬁrst-order optimization methods with random initialization, and several quantities we work with in the proofs are the same.
Revised: Both propose ﬁrst-order optimization methods with random initialization, and several quantities we work with in the proofs are the same.

Original: A deﬁning difference is we work with the nonsmooth (cid:96) 1 objective directly, while Gilboa et al.
Revised: A deﬁning difference is we work with the nonsmooth (cid:96) 1 objective directly, while Gilboa et al.

Original: (2018) built on the smoothed objective from Sun et al.
Revised: (2018) built on the smoothed objective from Sun et al.

Original: We put considerable emphasis on practicality: the subgradient of the nonsmooth objective is considerably cheaper to evaluate than that of the smooth objective in Sun et al.
Revised: We put considerable emphasis on practicality: the subgradient of the nonsmooth objective is considerably cheaper to evaluate than that of the smooth objective in Sun et al.

Original: (2015), and
Revised: (2015), and

Original: in the algorithm we use Euclidean projection rather than exponential mapping to remain feasible—again, the former is much lighter for computation.
Revised: in the algorithm we use Euclidean projection rather than exponential mapping to remain feasible—again, the former is much lighter for computation.

Original: General nonsmooth analysis While nonsmooth analytic tools such as subdifferential for convex functions are now well received in machine learning and relevant communities, that for general functions are much less so.
Revised: General nonsmooth analysis While nonsmooth analytic tools such as subdifferential for convex functions are now well received in machine learning and relevant communities, that for general functions are much less so.

Original: The Clarke subdifferential and relevant calculus developed for the family of locally Lipschitz functions seem to be particularly relevant, and cover several families of functions of interest, such as convex functions, differentiable functions, and many forms of composition(Clarke, 1990; Aubin, 1998;
Revised: The Clarke subdifferential and relevant calculus developed for the family of locally Lipschitz functions seem to be particularly relevant, and cover several families of functions of interest, such as convex functions, differentiable functions, and many forms of composition

Original:  Bagirov et al., 2014).
Revised: (Clarke, 1990; Aubin, 1998; Bagirov et al., 2014).

Original: Remarkably, majority of the tools and results can be generalized to locally Lipschitz functions on Riemannnian manifolds (Ledyaev & Zhu, 2007;
Revised: Remarkably, majority of the tools and results can be generalized to locally Lipschitz functions on Riemannnian manifolds (Ledyaev & Zhu, 2007;

Original: Hosseini & Pouryayevali, 2011).
Revised: Hosseini & Pouryayevali, 2011).

Original: Our formulation (1.1) is exactly optimization of a locally Lipschitz function (as it is convex) on a Riemannian manifold (the sphere).
Revised: Our formulation (1.1) is exactly optimization of a locally Lipschitz function (as it is convex) on a Riemannian manifold (the sphere).

Original: For simplicity, we try to avoid the full manifold language, nonetheless.
Revised: For simplicity, we try to avoid the full manifold language, nonetheless.

Original: Nonsmooth optimization on Riemannian manifolds or with constraints Equally remarkable is many of the smooth optimization techniques and convergence results can be naturally adapted to optimization of locally Lipschitz functions on Riemannian manifolds (Grohs & Hosseini, 2015; Hosseini, 2015; Hosseini & Uschmajew, 2017; Grohs & Hosseini, 2016).
Revised: Nonsmooth optimization on Riemannian manifolds or with constraints Equally remarkable is many of the smooth optimization techniques and convergence results can be naturally adapted to optimization of locally Lipschitz functions on Riemannian manifolds (Grohs & Hosseini, 2015; Hosseini, 2015; Hosseini & Uschmajew, 2017; Grohs & Hosseini, 2016).

Original: New optimization methods such as gradient sampling and variants have been invented to solve general nonsmooth problems (Burke et al., 2005; 2018;
Revised: New optimization methods such as gradient sampling and variants have been invented to solve general nonsmooth problems (Burke et al., 2005; 2018;

Original: Bagirov et al., 2014; Curtis & Que, 2015; Curtis et al., 2017).
Revised: Bagirov et al., 2014; Curtis & Que, 2015; Curtis et al., 2017).

Original: Almost all available convergence results pertain to only global convergence, which is too weak for our purpose.
Revised: Almost all available convergence results pertain to only global convergence, which is too weak for our purpose.

Original: Our speciﬁc convergence analysis gives us a local convergence result (Theorem 3.8).
Revised: Our speciﬁc convergence analysis gives us a local convergence result (Theorem 3.8).

Original: Nonsmooth landscape characterization Nonsmoothness is not a big optimization barrier if the problem is convex; here we review some recent work on analyzing nonconvex nonsmooth problems.
Revised: Nonsmooth landscape characterization Nonsmoothness is not a big optimization barrier if the problem is convex; here we review some recent work on analyzing nonconvex nonsmooth problems.

Original: Loh & Wainwright (2015) study the regularized empirical risk minimization problem with nonsmooth regularizers and show results of the type “all stationary points are within statistical error of ground truth” under certain restricted strong convexity of the smooth risk.
Revised: Loh & Wainwright (2015) study the regularized empirical risk minimization problem with nonsmooth regularizers and show results of the type “all stationary points are within statistical error of ground truth” under certain restricted strong convexity of the smooth risk.

Original: Duchi & Ruan (2017); Davis et al.
Revised: Duchi & Ruan (2017); Davis et al.

Original: (2017) study the phase retrieval problem with (cid:96) 1 loss, characterizing its nonconvex nonsmooth landscape and providing efﬁcient algorithms.
Revised: (2017) study the phase retrieval problem with (cid:96) 1 loss, characterizing its nonconvex nonsmooth landscape and providing efﬁcient algorithms.

Original: There is a recent surge of work on analyzing one-hidden-layer ReLU networks, which are nonconvex and nonsmooth.
Revised: There is a recent surge of work on analyzing one-hidden-layer ReLU networks, which are nonconvex and nonsmooth.

Original: Algorithm-independent characterizations of the landscape are mostly local and require strong initialization procedures (Zhong et al., 2017), whereas stronger global results can be established via designing new loss functions
Revised: Algorithm-independent characterizations of the landscape are mostly local and require strong initialization procedures (Zhong et al., 2017), whereas stronger global results can be established via designing new loss functions

Original: (Ge et al., 2017), relating to PDEs (Mei et al., 2018), or problem-dependent analysis of the SGD (Li & Yuan, 2017; Li & Liang, 2018).
Revised: (Ge et al., 2017), relating to PDEs (Mei et al., 2018), or problem-dependent analysis of the SGD (Li & Yuan, 2017; Li & Liang, 2018).

Original: Our result provides an algorithm-independent chacaterization of the landscape of non-smooth dictionary learning, and is “almost global” in the sense that the initialization condition is satisifed by random initialization with high probability.
Revised: Our result provides an algorithm-independent chacaterization of the landscape of non-smooth dictionary learning, and is “almost global” in the sense that the initialization condition is satisifed by random initialization with high probability.

Original: Other nonsmooth problems in application Prevalence of nonsmooth problems in optimal control and economics is evident from all monographs on nonsmooth analysis (Clarke, 1990; Aubin, 1998;
Revised: Other nonsmooth problems in application Prevalence of nonsmooth problems in optimal control and economics is evident from all monographs on nonsmooth analysis (Clarke, 1990; Aubin, 1998;

Original: Bagirov et al., 2014).
Revised: Bagirov et al., 2014).

Original: In modern machine learning and data analysis, nonsmooth functions are often taken to encode structural information (e.g., sparsity, low-rankness, quantization), or whenever robust estimation is desired.
Revised: In modern machine learning and data analysis, nonsmooth functions are often taken to encode structural information (e.g., sparsity, low-rankness, quantization), or whenever robust estimation is desired.

Original: In deep learning, the optimization problem is nonsmooth when nonsmooth activations are in use, e.g., the popular ReLU.
Revised: In deep learning, the optimization problem is nonsmooth when nonsmooth activations are in use, e.g., the popular ReLU.

Original: The technical ideas around nonsmooth analysis, set-valued analysis, and random set theory that we gather and develop here are particularly relevant to these applications.
Revised: The technical ideas around nonsmooth analysis, set-valued analysis, and random set theory that we gather and develop here are particularly relevant to these applications.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Problem setup Given an unknown orthogonal dictionary
Revised: Problem setup Given an unknown orthogonal dictionary

Original: A =
Revised: A =

Original: [ a 1 , . .
Revised: [ a 1 , . .

Original: . , a n ] ∈ R n × n , we wish to recover A through m observations of the form 
Revised: . , a n ] ∈ R n × n , we wish to recover A through m observations of the form y i

Original: [Equation] or Y = AX in matrix form, where X =
Revised: = Ax i , (2.1) or Y = AX in matrix form, where X =

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x m ] and Y =
Revised: , x m ] and Y =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y m ] .
Revised: , y m ] .

Original: The coefﬁcient vectors x i are sampled from the Bernoulli-Gaussian distribution with parameter θ ∈ (0 , 1) , denoted as BG( θ ) : each entry x ij is independently drawn from a standard Gaussian with probability θ and zero otherwise.
Revised: The coefﬁcient vectors x i are sampled from the Bernoulli-Gaussian distribution with parameter θ ∈ (0 , 1) , denoted as BG( θ ) : each entry x ij is independently drawn from a standard Gaussian with probability θ and zero otherwise.

Original: The Bernoulli-Gaussian is a good prototype distribution for sparse vectors, as x i will be on average θ -sparse.
Revised: The Bernoulli-Gaussian is a good prototype distribution for sparse vectors, as x i will be on average θ -sparse.

Original: For any z ∼ iid Ber( θ ) , we let Ω denote the set of non-zero indices, which is a random set itself.
Revised: For any z ∼ iid Ber( θ ) , we let Ω denote the set of non-zero indices, which is a random set itself.

Original: We assume that n ≥ 3 and θ ∈ [1 /n, 1 / 2] .
Revised: We assume that n ≥ 3 and θ ∈ [1 /n, 1 / 2] .

Original: In particular, θ ≥ 1 /n is to require that each x i has at least one non-zero entry on average.
Revised: In particular, θ ≥ 1 /n is to require that each x i has at least one non-zero entry on average.

Original: First-order geometry We will focus on the ﬁrst-order geometry of the non-smooth objective Eq.
Revised: First-order geometry We will focus on the ﬁrst-order geometry of the non-smooth objective Eq.

Original: (1.1): f ( q ) = 1 m (cid:80) mi =1 | q (cid:62)
Revised: (1.1): f ( q ) = 1 m (cid:80) mi =1 | q (cid:62)

Original: y
Revised: y

Original: i | .
Revised: i | .

Original: In the whole Euclidean space R n , f is convex withsub-differential set
Revised: In the whole Euclidean space R n , f is convex with

Original: [Equation]
Revised: [Equation]

Original: where sign( · ) is the set-valued sign function (i.e. sign(0)
Revised: where sign( · ) is the set-valued sign function (i.e. sign(0)

Original: =
Revised: =

Original: [ − 1 , 1] ).
Revised: [ − 1 , 1] ).

Original: As we minimize f subject to the constraint (cid:107) q (cid:107) 2 = 1 , our problem is no longer convex.
Revised: As we minimize f subject to the constraint (cid:107) q (cid:107) 2 = 1 , our problem is no longer convex.

Original: The Riemannian sub-differential of f on S n −is deﬁned as (Hosseini & Uschmajew, 2017):
Revised: The Riemannian sub-differential of f on S n −is deﬁned as (Hosseini & Uschmajew, 2017):

Original: [Equation]
Revised: [Equation]

Original: A point q is stationary for problem Eq.
Revised: A point q is stationary for problem Eq.

Original: (1.1) if 0 ∈ ∂ R f ( q ) .
Revised: (1.1) if 0 ∈ ∂ R f ( q ) .

Original: We will not distinguish between local maxima and saddle points—we call a stationary point q a saddle point if there is a descent direction (i.e. direction along which the function is locally maximized at q ).
Revised: We will not distinguish between local maxima and saddle points—we call a stationary point q a saddle point if there is a descent direction (i.e. direction along which the function is locally maximized at q ).

Original: Set-valued analysis As the subdifferential is a set-valued mapping, analyzing it requires some setvalued analysis, which we brieﬂy present here.
Revised: Set-valued analysis As the subdifferential is a set-valued mapping, analyzing it requires some setvalued analysis, which we brieﬂy present here.

Original: The addition of two sets is deﬁned as the Minkowski summation: X + Y = { x + y : x ∈ X, y ∈ Y } .
Revised: The addition of two sets is deﬁned as the Minkowski summation: X + Y = { x + y : x ∈ X, y ∈ Y } .

Original: The expectation of random sets is a straightforward extension of the Minkowski sum allowing any measurable “selection” procedure; for the concrete deﬁnition see (Molchanov, 2013).
Revised: The expectation of random sets is a straightforward extension of the Minkowski sum allowing any measurable “selection” procedure; for the concrete deﬁnition see (Molchanov, 2013).

Original: The Hausdorff distance between two sets is deﬁned as
Revised: The Hausdorff distance between two sets is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Basic properties about the Hausdorff distance are provided in Appendix A.1.
Revised: Basic properties about the Hausdorff distance are provided in Appendix A.1.

Original: Notations Bold small letters (e.g., x ) are vectors and bold capitals are matrices (e.g., X ).
Revised: Notations Bold small letters (e.g., x ) are vectors and bold capitals are matrices (e.g., X ).

Original: The dotted equality .
Revised: The dotted equality .

Original: = is for deﬁnition.
Revised: = is for deﬁnition.

Original: For any positive integer k , [ k ] .
Revised: For any positive integer k , [ k ] .

Original: = { 1 , . . .
Revised: = { 1 , . . .

Original: , k } .
Revised: , k } .

Original: By default, (cid:107)·(cid:107) is the (cid:96)norm if applied to a vector, and the operator norm if applied to a matrix.
Revised: By default, (cid:107)·(cid:107) is the (cid:96)norm if applied to a vector, and the operator norm if applied to a matrix.

Original: C and c or any indexed versions are reserved for universal constants that may change from place to place.
Revised: C and c or any indexed versions are reserved for universal constants that may change from place to place.

Original:  M AIN R ESULT
Revised:  M AIN R ESULT

Original: We now state our main result, the recovery guarantee for learning orthogonal dictionary by solving formulation (1.1).
Revised: We now state our main result, the recovery guarantee for learning orthogonal dictionary by solving formulation (1.1).

Original: Theorem 3.1 (Recovering orthogonal dictionary via subgradient descent) .
Revised: Theorem 3.1 (Recovering orthogonal dictionary via subgradient descent) .

Original: Suppose we observe
Revised: Suppose we observe

Original: [Equation]
Revised: [Equation]

Original: samples in the dictionary learning problem and we desire an accuracy (cid:15) ∈ (0 , 1) for recovering the dictionary.
Revised: samples in the dictionary learning problem and we desire an accuracy (cid:15) ∈ (0 , 1) for recovering the dictionary.

Original: With probability at least 1 − exp (cid:0) − cmθ 3 n

Original: [Equation] [Equation] 3 m (cid:1) − exp ( − c (cid:48) R/n ) , an algorithm which runs Riemannian subgradient descent R = C (cid:48) n log n times with independent random initializations on S n − 1 outputs a set of vectors { (cid:98) a 1 , . . 
Revised: With probability at least 1 − exp (cid:0) − cmθ 3 n − 3 log − 3 m (cid:1) − exp ( − c (cid:48) R/n ) , an algorithm which runs Riemannian subgradient descent R = C (cid:48) n log n times with independent random initializations on S n − 1 outputs a set of vectors { (cid:98) a 1 , . . .

Original: , (cid:98) a n } such that up to permutation and sign change, (cid:107) (cid:98) a
Revised: , (cid:98) a n } such that up to permutation and sign change, (cid:107) (cid:98) a

Original: i − a i (cid:107) 2 ≤ (cid:15) for all i ∈ [ n ] .
Revised: i − a i (cid:107) 2 ≤ (cid:15) for all i ∈ [ n ] .

Original: The total number of subgradient descent iterations is bounded by
Revised: The total number of subgradient descent iterations is bounded by

Original: [Equation]
Revised: [Equation]

Original: Here C, C (cid:48) , C (cid:48)(cid:48) , c, c (cid:48) > 0 are universal constants.
Revised: Here C, C (cid:48) , C (cid:48)(cid:48) , c, c (cid:48) > 0 are universal constants.

Original: At a high level, the proof of Theorem 3.1 consists of the following steps, which we elaborate throughout the rest of this section.
Revised: At a high level, the proof of Theorem 3.1 consists of the following steps, which we elaborate throughout the rest of this section.

Original: Partition the sphere into 2 n symmetric “good sets” and show certain directional gradient is strong on population objective E [ f ] inside the good sets (Section 3.1).
Revised: Partition the sphere into 2 n symmetric “good sets” and show certain directional gradient is strong on population objective E [ f ] inside the good sets (Section 3.1).

Original: Show that the same geometric properties carry over to the empirical objective f with high probability.
Revised: Show that the same geometric properties carry over to the empirical objective f with high probability.

Original: This involves proving the uniform convergence of the subdifferential set ∂f to E [ ∂f ] (Section 3.2).
Revised: This involves proving the uniform convergence of the subdifferential set ∂f to E [ ∂f ] (Section 3.2).

Original: Under the benign geometry, establish the convergence of Riemannian subgradient descent to one of {± a i : i ∈ [ n ] } when initialized in the corresponding “good set” (Section 3.3).
Revised: Under the benign geometry, establish the convergence of Riemannian subgradient descent to one of {± a i : i ∈ [ n ] } when initialized in the corresponding “good set” (Section 3.3).

Original: Calling the randomly initialized optimization procedure O ( n log n ) times will recover all of { a 1 , . . .
Revised: Calling the randomly initialized optimization procedure O ( n log n ) times will recover all of { a 1 , . . .

Original: , a n } with high probability, by a coupon collector’s argument (Section 3.4).
Revised: , a n } with high probability, by a coupon collector’s argument (Section 3.4).

Original: Scaling and rotating to identity Throughout the rest of this paper, we are going to assume WLOG that the dictionary is the identity matrix, i.e. A = I n , so that Y =
Revised: Scaling and rotating to identity Throughout the rest of this paper, we are going to assume WLOG that the dictionary is the identity matrix, i.e. A = I n , so that Y =

Original: X , f ( q ) =
Revised: X , f ( q ) =

Original: (cid:13)(cid:13) q (cid:62) X
Revised: (cid:13)(cid:13) q (cid:62) X

Original: (cid:13)(cid:13) 1 , and the goal is to ﬁnd the standard basis vectors {± e 1 , . . .
Revised: (cid:13)(cid:13) 1 , and the goal is to ﬁnd the standard basis vectors {± e 1 , . . .

Original: , ± e n } .
Revised: , ± e n } .

Original: The case of a general orthogonal A can be reduced to this special case via rotating by A (cid:62) :
Revised: The case of a general orthogonal A can be reduced to this special case via rotating by A (cid:62) :

Original: q (cid:62) Y =
Revised: q (cid:62) Y =

Original: q (cid:62) AX =
Revised: q (cid:62) AX =

Original: ( q (cid:48) )
Revised: ( q (cid:48) )

Original: (cid:62) X where q (cid:48)
Revised: (cid:62) X where q (cid:48)

Original: =
Revised: =

Original: A (cid:62) q and applying the result on q (cid:48) .
Revised: A (cid:62) q and applying the result on q (cid:48) .

Original: We also scale the objective by (cid:112) π/ 2 for convenience of later analysis.
Revised: We also scale the objective by (cid:112) π/ 2 for convenience of later analysis.

Original: 3. P ROPERTIES OF THE POPULATION OBJECTIVE
Revised: 3. P ROPERTIES OF THE POPULATION OBJECTIVE

Original: We begin by characterizing the geometry of the expected objective E [ f ] .
Revised: We begin by characterizing the geometry of the expected objective E [ f ] .

Original: Recall that we have rotated
Revised: Recall that we have rotated

Original: A to be identity, so that we have
Revised: A to be identity, so that we have

Original: [Equation]
Revised: [Equation]

Original: Minimizers and saddles of the population objective We begin by computing the function value and subdifferential set of the population objective and giving a complete characterization of its stationary points, i [Equation] e. local minimizers and saddles.
Revised: Minimizers and saddles of the population objective We begin by computing the function value and subdifferential set of the population objective and giving a complete characterization of its stationary points, i.e. local minimizers and saddles.

Original: Proposition [Equation] 2 (Population objective value and gradient) .
Revised: Proposition 3.2 (Population objective value and gradient) .

Original: We have
Revised: We have

Original: E 
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.3 (Stationary points) .
Revised: Proposition 3.3 (Stationary points) .

Original: The stationary points of E [ f ] on the sphere are
Revised: The stationary points of E [ f ] on the sphere are

Original: [Equation]
Revised: [Equation]

Original: The case k = 1 corresponds to the 2 n global minimizers q = ±
Revised: The case k = 1 corresponds to the 2 n global minimizers q = ±

Original: e i , and all other values of k correspond to saddle points.
Revised: e i , and all other values of k correspond to saddle points.

Original: A consequence of Proposition 3.3 is that the population objective has no “spurious local minima”: each stationary point is either a global minimizer or a saddle point, though the problem itself is non-convex due to the constraint.
Revised: A consequence of Proposition 3.3 is that the population objective has no “spurious local minima”: each stationary point is either a global minimizer or a saddle point, though the problem itself is non-convex due to the constraint.

Original: Identifying 2 n “good” subsets We now deﬁne 2 n subsets on the sphere, each containing one of the global minimizers {± e i } and possessing benign geometry for both the population and empirical objective, following (Gilboa et al., 2018).
Revised: Identifying 2 n “good” subsets We now deﬁne 2 n subsets on the sphere, each containing one of the global minimizers {± e i } and possessing benign geometry for both the population and empirical objective, following (Gilboa et al., 2018).

Original: For any ζ ∈ [0 , ∞ ) and i ∈ [ n ] deﬁne
Revised: For any ζ ∈ [0 , ∞ ) and i ∈ [ n ] deﬁne

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Original: ζ
Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: For points in S ( i +) ζ ∪ S
Revised: For points in S ( i +) ζ ∪ S

Original: ( i − ) ζ , the i -th index is larger than all other indices (in absolute value) by a multiplicative factor of ζ .
Revised: ( i − ) ζ , the i -th index is larger than all other indices (in absolute value) by a multiplicative factor of ζ .

Original: In particular, for any point in these subsets, the largest index is unique, so by Proposition 3.3 all population saddle points are excluded from these 2 n subsets.
Revised: In particular, for any point in these subsets, the largest index is unique, so by Proposition 3.3 all population saddle points are excluded from these 2 n subsets.

Original: Intuitively, this partition can serve as a “tiebreaker”: points in S ( i +) ζis closer to e i than all the other 2 n − 1 signed basis vectors.
Revised: Intuitively, this partition can serve as a “tiebreaker”: points in S ( i +) ζis closer to e i than all the other 2 n − 1 signed basis vectors.

Original: Therefore, we hope that optimization algorithms initialized in this region could favor e i over the other standard basis vectors, which we are going to show is indeed the case.
Revised: Therefore, we hope that optimization algorithms initialized in this region could favor e i over the other standard basis vectors, which we are going to show is indeed the case.

Original: For simplicity, we are going to state our geometry results in S ( n +) ζ ; by symmetry the results will automatically carry over to all the other 2 n − 1 subsets.
Revised: For simplicity, we are going to state our geometry results in S ( n +) ζ ; by symmetry the results will automatically carry over to all the other 2 n − 1 subsets.

Original: Theorem 3.4 (Lower bound on directional subgradients) .
Revised: Theorem 3.4 (Lower bound on directional subgradients) .

Original: Fix any ζ 0 ∈ (0 , 1) .
Revised: Fix any ζ 0 ∈ (0 , 1) .

Original: We have (a) For all q ∈ S ( n +) ζand all indices j (cid:54)
Revised: We have (a) For all q ∈ S ( n +) ζand all indices j (cid:54)

Original: = n such that q j (cid:54) = 0 ,
Revised: = n such that q j (cid:54) = 0 ,

Original: [Equation]
Revised: [Equation]

Original: b) For all q ∈ S ( n +) ζ , we have that
Revised: (b) For all q ∈ S ( n +) ζ, we have that

Original: [Equation]
Revised: [Equation]

Original: These lower bounds verify our intuition: points inside S ( n +) ζhave subgradients pointing towards e n , both in a coordinate-wise sense and a combined sense: the direction e n − q n q is exactly the tangent direction of the sphere at q that points towards e n .
Revised: These lower bounds verify our intuition: points inside S ( n +) ζ [Equation] have subgradients pointing towards e n , both in a coordinate-wise sense and a combined sense: the direction e n − q n q is exactly the tangent direction of the sphere at q that points towards e n .

Original: 3. B ENIGN GEOMETRY OF THE EMPIRICAL OBJECTIVE
Revised: 3. B ENIGN GEOMETRY OF THE EMPIRICAL OBJECTIVE

Original: We now show that the benign geometry in Theorem 3.4 is carried onto the empirical objective f given sufﬁciently many samples, using a concentration argument.
Revised: We now show that the benign geometry in Theorem 3.4 is carried onto the empirical objective f given sufﬁciently many samples, using a concentration argument.

Original: The key result behind is the concentration of the empirical subdifferential set to the population subdifferential, where concentration is measured in the Hausdorff distance between sets.
Revised: The key result behind is the concentration of the empirical subdifferential set to the population subdifferential, where concentration is measured in the Hausdorff distance between sets.

Original: Proposition 3.5 (Uniform convergence of subdifferential) .
Revised: Proposition 3.5 (Uniform convergence of subdifferential) .

Original: For any t ∈ (0 , 1] , when
Revised: For any t ∈ (0 , 1] , when

Original: [Equation]

Original: 2 ( n/t ) , (

Original: [Equation]
Revised: [Equation]

Original: .10) with probability at least 1 − exp (cid:0) − cmθt 2 / log m (cid:1) , we have
Revised: with probability at least 1 − exp (cid:0) − cmθt 2 / log m (cid:1) , we have

Original: [Equation]
Revised: [Equation]

Original: for all q ∈ S n

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Here C, c ≥ 0 are universal constants.
Revised: Here C, c ≥ 0 are universal constants.

Original: The concentration result guarantees that the sub-differential set is close to its expectation given sufﬁciently many samples with high probability.
Revised: The concentration result guarantees that the sub-differential set is close to its expectation given sufﬁciently many samples with high probability.

Original: Choosing an appropriate concentration level t , the lower bounds on the directional subgradients carry over to the empirical objective f , which we state in the following theorem.
Revised: Choosing an appropriate concentration level t , the lower bounds on the directional subgradients carry over to the empirical objective f , which we state in the following theorem.

Original: Theorem 3.6 (Directional subgradient lower bound, empirical objective) .
Revised: Theorem 3.6 (Directional subgradient lower bound, empirical objective) .

Original: There exist universal constants C, c ≥ 

Revised: There exist universal constants C, c ≥

Original: so that the following holds: for all ζ 0 ∈ (0 , 1) , when m ≥ Cn 4 θ − 2 ζ − 2 0 log 2 ( n/ζ 0 ) , with probability at least 1 − exp (cid:0)− cmθ 3 ζ 20 n − 3 log − 1 m (cid:1) , the following properties hold simultaneously for all the n subsets (cid:110) S
Revised: so that the following holds: for all ζ 0 ∈ (0 , 1) , when m ≥ Cn 4 θ − 2 ζ − 2 0 log 2 ( n/ζ 0 ) , with probability at least 1 − exp (cid:0)

Original: ( i +)
Revised: [Equation]

Original:  ζ S ( i − ) ζ : i ∈ [ n ] (cid:111) : (stated only for S ( n +) ζ
Revised: 3 ζ 20 n − 3 log − 1 m (cid:1) , the following properties hold simultaneously for all the 2 n subsets (cid:110) S

Revised: [Equation]

Original: (a) For all q ∈ S ( n +) ζand all j ∈ [ n ] with q j (cid:54)
Revised: (a) For all q ∈ S ( n +) ζand all j ∈ [ n ] with q j (cid:54)

Original: = 0 and q 2 n /q 2 j ≤ 3 ,
Revised: = 0 and q 2 n /q 2 j ≤ 3 ,

Original: [Equation]
Revised: [Equation]

Original: [Equation] q ∈ S  n +) 
Revised: (b) For all q ∈ S ( n +) ζ

Original: [Equation]
Revised: [Equation]

Original: The consequence of Theorem 3.6 is two-fold.
Revised: The consequence of Theorem 3.6 is two-fold.

Original: First, it guarantees that the only possible stationary point of f in S ( n +) ζis e n : for every other point q (cid:54) =
Revised: First, it guarantees that the only possible stationary point of f in S ( n +) ζis e n : for every other point q (cid:54) =

Original: e n , property (b) guarantees that 0 / ∈ ∂ R f ( q ) , therefore q is non-stationary.
Revised: e n , property (b) guarantees that 0 / ∈ ∂ R f ( q ) , therefore q is non-stationary.

Original: Second, the directional subgradient lower bounds allow us to establish convergence of the Riemannian subgradient descent algorithm, in a way similar to showing convergence of unconstrained gradient descent on star strongly convex functions.
Revised: Second, the directional subgradient lower bounds allow us to establish convergence of the Riemannian subgradient descent algorithm, in a way similar to showing convergence of unconstrained gradient descent on star strongly convex functions.

Original: We now present an upper bound on the norm of the subdifferential sets, which is needed for the convergence analysis.
Revised: We now present an upper bound on the norm of the subdifferential sets, which is needed for the convergence analysis.

Original: Proposition 3.7.
Revised: Proposition 3.7.

Original: There exist universal constants C, c ≥ 0 such that
Revised: There exist universal constants C, c ≥ 0 such that

Original: [Equation]
Revised: [Equation]

Original: 14) with probability at least 1 − exp (cid:0) − cmθ log − 1 m (cid:1) , provided that m ≥ Cn log n .
Revised: with probability at least 1 − exp (cid:0) − cmθ log − 1 m (cid:1) , provided that m ≥ Cn log n .

Original: This particularly implies that
Revised: This particularly implies that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3. F INDING ONE BASIS VIA R IEMANNIAN SUBGRADIENT DESCENT
Revised: 3. F INDING ONE BASIS VIA R IEMANNIAN SUBGRADIENT DESCENT

Original: The benign geometry of the empirical objective allows a simple Riemannian subgradient descent algorithm to ﬁnd one basis vector a time.
Revised: The benign geometry of the empirical objective allows a simple Riemannian subgradient descent algorithm to ﬁnd one basis vector a time.

Original: The Riemannian subgradient descent algorithm with initialization q (0) and step size (cid:8) η ( k ) (cid:9) k ≥ 0 is as follows.
Revised: The Riemannian subgradient descent algorithm with initialization q (0) and step size (cid:8) η ( k ) (cid:9) k ≥ 0 is as follows.

Original: For an arbitrary v ∈ ∂ R f (cid:0) q ( k ) (cid:1) ,
Revised: For an arbitrary v ∈ ∂ R f (cid:0) q ( k ) (cid:1) ,

Original: [Equation]
Revised: [Equation]

Original: for k =
Revised: for k =

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Each iteration moves in an arbitrary Riemannian subgradient direction followed by a projection back onto the sphere.
Revised: Each iteration moves in an arbitrary Riemannian subgradient direction followed by a projection back onto the sphere.

Original: We show that the algorithm is guaranteed to ﬁnd one basis as long as the initialization is in the “right” region.
Revised: We show that the algorithm is guaranteed to ﬁnd one basis as long as the initialization is in the “right” region.

Original: To give a concrete result, we set ζ 0 = 1 / (5 log n ) .
Revised: To give a concrete result, we set ζ 0 = 1 / (5 log n ) .

Original: Theorem 3.8 (One run of subgradient descent recovers one basis) .
Revised: Theorem 3.8 (One run of subgradient descent recovers one basis) .

Original: Let m ≥ Cθ − 2 n 4 log 4 n and (cid:15) ∈ (0 , 3 / 100] .
Revised: Let m ≥ Cθ − 2 n 4 log 4 n and (cid:15) ∈ (0 , 3 / 100] .

Original: With probability at least 1 − exp (cid:0) − cmθ 3 n − 3 log − 3 m (cid:1) the following happens.
Revised: With probability at least 1 − exp (cid:0) − cmθ 3 n − 3 log − 3 m (cid:1) the following happens.

Original: If the initialization q (0) ∈ S ( n +) 1 / (5log n ) , and we run the projected Riemannian subgradient descent with step size η ( k ) =
Revised: If the initialization q (0) ∈ S ( n +) 1 / (5log n ) , and we run the projected Riemannian subgradient descent with step size η ( k ) =

Original: k − α / (100 √ n ) with α ∈ (0 , 1 / 2) , and keep track of the best function value so far until after iterate K is performed, producing q best .
Revised: k − α / (100 √ n ) with α ∈ (0 , 1 / 2) , and keep track of the best function value so far until after iterate K is performed, producing q best .

Original: Then, q best obeys
Revised: Then, q best obeys

Original: [Equation]
Revised: [Equation]

Original: , and(cid:13)(cid:13) [Equation]
Revised: , and

Original: [Equation]
Revised: [Equation]

Original: (3.17) provided that
Revised: 17) provided that

Original: [Equation]
Revised: [Equation]

Original: In particular, choosing α = 3 / 8 < 1 / 2 , it sufﬁces to let
Revised: In particular, choosing α = 3 / 8 < 1 / 2 , it sufﬁces to let

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here C, C (cid:48) , c ≥ 0 are universal constants.
Revised: Here C, C (cid:48) , c ≥ 0 are universal constants.

Original: The above optimization result (Theorem 3.8) shows that Riemannian subgradient descent is able to ﬁnd the basis vector e n when initialized in the associated region S ( n +) 1 / (5log n ) .
Revised: The above optimization result (Theorem 3.8) shows that Riemannian subgradient descent is able to ﬁnd the basis vector e n when initialized in the associated region S ( n +) 1 / (5log n ) .

Original: We now show that a simple uniformly random initialization on the sphere is guaranteed to be in one of these 2 n regions with at least probability 1 / 2 .
Revised: We now show that a simple uniformly random initialization on the sphere is guaranteed to be in one of these 2 n regions with at least probability 1 / 2 .

Original: Lemma 3.9 (Random initialization falls in “good set”) .
Revised: Lemma 3.9 (Random initialization falls in “good set”) .

Original: Let q (0) ∼ Uniform( S n − 1 ) , then with probability at least 1 / 2 , q (0) belongs to one of the 2 n sets (cid:110) S ( i +) 1 / (5log n ) , S ( i − ) 1 / (5log n ) : i ∈ [ n ] (cid:111) .
Revised: Let q (0) ∼ Uniform( S n − 1 ) , then with probability at least 1 / 2 , q (0) belongs to one of the 2 n sets (cid:110) S ( i +) 1 / (5log n ) , S ( i − ) 1 / (5log n ) : i ∈ [ n ] (cid:111) .

Original: 3. R ECOVERING ALL BASES FROM MULTIPLE RUNS
Revised: 3. R ECOVERING ALL BASES FROM MULTIPLE RUNS

Original: As long as the initialization belongs to S ( i +) 1 / (5log n ) or S ( i − ) 1 / (5log n ) , our ﬁnding-one-basis result in Theorem 3.8 guarantees that Riemannian subgradient descent will converge to e i or
Revised: As long as the initialization belongs to S ( i +) 1 / (5log n ) or S ( i − ) 1 / (5log n ) , our ﬁnding-one-basis result in Theorem 3.8 guarantees that Riemannian subgradient descent will converge to e i or

Original: − e i respectively.
Revised: − e i respectively.

Original: Therefore if we run the algorithm with independent, uniformly random initializations on the sphere multiple times, by a coupon collector’s argument, we will recover all the basis vectors.
Revised: Therefore if we run the algorithm with independent, uniformly random initializations on the sphere multiple times, by a coupon collector’s argument, we will recover all the basis vectors.

Original: This is formalized in the following theorem.
Revised: This is formalized in the following theorem.

Original: Theorem 3.10 (Recovering the identity dictionary from multiple random initializations) .
Revised: Theorem 3.10 (Recovering the identity dictionary from multiple random initializations) .

Original: Let m ≥
Revised: Let m ≥

Original: Cn 4 θ − 2 log 4 n and (cid:15) ∈ (0 , 1) , with probability at least 1 − exp (cid:0) − cmθ 3 n − 3 log − 3 m (cid:1) the following happens.
Revised: Cn 4 θ − 2 log 4 n and (cid:15) ∈ (0 , 1) , with probability at least 1 − exp (cid:0) − cmθ 3 n − 3 log − 3 m (cid:1) the following happens.

Original: Suppose we run the Riemannian subgradient descent algorithm independently for R times, each with a uniformly random initialization on S n − 1 , and choose the step size as η ( k ) =
Revised: Suppose we run the Riemannian subgradient descent algorithm independently for R times, each with a uniformly random initialization on S n − 1 , and choose the step size as η ( k ) =

Original: k − 3 / 8 / (100 √ n ) .
Revised: k − 3 / 8 / (100 √ n ) .

Original: Then, provided that R ≥ C (cid:48) n log n , all standard basis vectors will be recovered up to (cid:15) accuracy with probability at least 1 − exp ( − cR/n ) in C (cid:48)
Revised: Then, provided that R ≥ C (cid:48) n log n , all standard basis vectors will be recovered up to (cid:15) accuracy with probability at least 1 − exp ( − cR/n ) in C (cid:48)

Original: Rθ − 16 / 3 (cid:15) − 8 / 3 n 4 log 8 / 3 n iterations.
Revised: Rθ − 16 / 3 (cid:15) − 8 / 3 n 4 log 8 / 3 n iterations.

Original: Here C, C (cid:48) , c ≥ 0 are universal constants.
Revised: Here C, C (cid:48) , c ≥ 0 are universal constants.

Original: When the dictionary A is not the identity matrix, we can apply the rotation argument sketched in the beginning of this section to get the same result, which leads to our main result in Theorem 3.1.
Revised: When the dictionary A is not the identity matrix, we can apply the rotation argument sketched in the beginning of this section to get the same result, which leads to our main result in Theorem 3.1.

Original:  P ROOF HIGHLIGHTS
Revised:  P ROOF HIGHLIGHTS

Original: A key technical challenge is establishing the uniform convergence of subdifferential sets in Proposition 3.5, which we now elaborate.
Revised: A key technical challenge is establishing the uniform convergence of subdifferential sets in Proposition 3.5, which we now elaborate.

Original: Recall that the population and empirical subdifferentials are
Revised: Recall that the population and empirical subdifferentials are

Original: [Equation]
Revised: [Equation]

Original: and we wish to show that the difference between ∂f ( q ) and E [ ∂f ] ( q ) is small uniformly over q ∈ Q = S n − 1 .
Revised: and we wish to show that the difference between ∂f ( q ) and E [ ∂f ] ( q ) is small uniformly over q ∈ Q = S n − 1 .

Original: Two challenges stand out in showing such a uniform convergence:
Revised: Two challenges stand out in showing such a uniform convergence:

Original: The subdifferential is set-valued and random, and it is unclear a-priori how one could formulate and analyze the concentration of random sets.
Revised: The subdifferential is set-valued and random, and it is unclear a-priori how one could formulate and analyze the concentration of random sets.

Original: The usual covering argument won’t work here, as the Lipschitz gradient property does not hold: ∂f ( q ) and E [ ∂f ] ( q ) are not Lipschitz in q .
Revised: The usual covering argument won’t work here, as the Lipschitz gradient property does not hold: ∂f ( q ) and E [ ∂f ] ( q ) are not Lipschitz in q .

Original: Therefore, no matter how ﬁne we cover the sphere in Euclidean distance, points not in this covering can have radically different subdifferential sets.
Revised: Therefore, no matter how ﬁne we cover the sphere in Euclidean distance, points not in this covering can have radically different subdifferential sets.

Original: 4. C ONCENTRATION OF RANDOM SETS
Revised: 4. C ONCENTRATION OF RANDOM SETS

Original: We state and analyze concentration of random sets in the Hausdorff distance (deﬁned in Section 2).
Revised: We state and analyze concentration of random sets in the Hausdorff distance (deﬁned in Section 2).

Original: We now illustrate how the Hausdorff distance is the “right” distance to consider for concentration of subdifferentials—the reason is that the Hausdorff distance is closely related to the support function of sets, which for any set S ∈ R n is deﬁned as
Revised: We now illustrate how the Hausdorff distance is the “right” distance to consider for concentration of subdifferentials—the reason is that the Hausdorff distance is closely related to the support function of sets, which for any set S ∈ R n is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: For convex compact sets, the sup difference between their support functions is exactly the Hausdorff distance.
Revised: For convex compact sets, the sup difference between their support functions is exactly the Hausdorff distance.

Original: Lemma 4.1 (Section 1.3.2, Molchanov (2013)) .
Revised: Lemma 4.1 (Section 1.3.2, Molchanov (2013)) .

Original: For convex compact sets X, Y ⊂ R n , we have
Revised: For convex compact sets X, Y ⊂ R n , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.1 is convenient for us in the following sense.
Revised: Lemma 4.1 is convenient for us in the following sense.

Original: Suppose we wish to upper bound the difference of ∂f ( q ) and E [ ∂f ] ( q ) along some direction u ∈ S n − 1 (as we need in proving the key empirical geometry result Theorem 3.6).
Revised: Suppose we wish to upper bound the difference of ∂f ( q ) and E [ ∂f ] ( q ) along some direction u ∈ S n − 1 (as we need in proving the key empirical geometry result Theorem 3.6).

Original: As both subdifferential sets are convex and compact, by Lemma 4.
Revised: As both subdifferential sets are convex and compact, by Lemma 4.

Original: we immediately have
Revised: we immediately have

Original: (cid:12)(cid:12)(cid:12)(cid:12) inf

Original: [Equation]
Revised: [Equation]

Original: Therefore, as long as we are able to bound the Hausdorff distance, all directional differences between the subdifferentials are simultaneously bounded, which is exactly what we want to show to carry the benign geometry from the population to the empirical objective.
Revised: Therefore, as long as we are able to bound the Hausdorff distance, all directional differences between the subdifferentials are simultaneously bounded, which is exactly what we want to show to carry the benign geometry from the population to the empirical objective.

Original: 4. C OVERING IN THE d E METRIC
Revised: 4. C OVERING IN THE d E METRIC

Original: We argue that the absence of gradient Lipschitzness is because the Euclidean distance is not the “right” metric in this problem.
Revised: We argue that the absence of gradient Lipschitzness is because the Euclidean distance is not the “right” metric in this problem.

Original: Think of the toy example f ( x ) =
Revised: Think of the toy example f ( x ) =

Original: | x | , whose subdifferential set ∂f ( x ) = sign( x ) is not Lipschitz across x = 0 .
Revised: | x | , whose subdifferential set ∂f ( x ) = sign( x ) is not Lipschitz across x = 0 .

Original: However, once we partition R into R > 0 , R < 0 and { 0 } (i.e. according to the sign pattern), the subdifferential set is Lipschitz on each subset.
Revised: However, once we partition R into R > 0 , R < 0 and { 0 } (i.e. according to the sign pattern), the subdifferential set is Lipschitz on each subset.

Original: The situation with the dictionary learning objective is quie similar: we resolve the gradient nonLipschitzness by proposing a stronger metric d E on the sphere which is sign-pattern aware and averages all “subset angles” between two points.
Revised: The situation with the dictionary learning objective is quie similar: we resolve the gradient nonLipschitzness by proposing a stronger metric d E on the sphere which is sign-pattern aware and averages all “subset angles” between two points.

Original: Formally, we deﬁne d E as
Revised: Formally, we deﬁne d E as

Original: [Equation]
Revised: [Equation]

Original: (the second equality shown in Lemma C.1.
Revised: (the second equality shown in Lemma C.1.

Original: Our plan is to perform the covering argument in d E , which requires showing gradient Lipschitzness in d E and bounding the covering number.
Revised: Our plan is to perform the covering argument in d E , which requires showing gradient Lipschitzness in d E and bounding the covering number.

Original: Lipschitzness of ∂f and E [ ∂f ] in d E For the population subdifferential E [ ∂f ] , note that E [ ∂f ] ( q ) 
Revised: Lipschitzness of ∂f and E [ ∂f ] in d E For the population subdifferential E [ ∂f ] , note that E [ ∂f ] ( q ) =

Original: [Equation]
Revised: [Equation]

Revised: BG

Original: [Equation]
Revised: [Equation]

Original: modulo rescaling).
Revised: θ ) [sign( q (cid:62) x ) x ] (modulo rescaling).

Original: Therefore, to bound d H ( E [ ∂f ] ( p ) , E [ ∂f ] ( q )) by Lemma 4.1, we have the bound for all u ∈ S n −[Equation] [Equation]
Revised: Therefore, to bound d H ( E [ ∂f ] ( p ) , E [ ∂f ] ( q )) by Lemma 4.1, we have the bound for all u ∈ S n −

Original: [Equation]
Revised: [Equation]

Original: As long as d E ( p , q ) ≤ (cid:15) , the indicator is non-zero with probability at most (cid:15) , and thus the above expectation should also be small – we bound it by O ( (cid:15) (cid:112) log(1 /(cid:15) )) in Lemma F.5.
Revised: As long as d E ( p , q ) ≤ (cid:15) , the indicator is non-zero with probability at most (cid:15) , and thus the above expectation should also be small – we bound it by O ( (cid:15) (cid:112) log(1 /(cid:15) )) in Lemma F.5.

Original: To show the same for the empirical subdifferential ∂f , one only needs to bound the observed proportion of sign differences for all p , q such that d E ( p , q ) ≤ (cid:15) , which by a VC dimension argument is uniformly bounded by 2 (cid:15) with high probability (Lemma C.5).
Revised: To show the same for the empirical subdifferential ∂f , one only needs to bound the observed proportion of sign differences for all p , q such that d E ( p , q ) ≤ (cid:15) , which by a VC dimension argument is uniformly bounded by 2 (cid:15) with high probability (Lemma C.5).

Original: Bounding the covering number in d E Our ﬁrst step is to reduce d E to the maximum lengthangle (the d 2 metric) over any consistent support pattern.
Revised: Bounding the covering number in d E Our ﬁrst step is to reduce d E to the maximum lengthangle (the d 2 metric) over any consistent support pattern.

Original: This is achieved through the following vector angle inequality (Lemma C.2): for any p , q ∈ R d ( d ≥ 3 ), we have(cid:54) ( p , q
Revised: This is achieved through the following vector angle inequality (Lemma C.2): for any p , q ∈ R d ( d ≥ 3 ), we have

Original: [Equation]
Revised: [Equation]

Original:  ≤ (cid:88)Ω
Revised: ( p , q ) ≤ (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [ d

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: | = (cid:54) ( p Ω , q Ω ) provided (cid:54) ( p, q ) ≤ π/ 2 .
Revised: [Equation] provided (cid:54) [Equation] p

Original:  Therefore, as long as sign( p ) = sign( q ) (coordinate-wise) and max | Ω | =2 (cid:54) ( p Ω , q Ω ) ≤ (cid:15)/n 2 , we would have for all | Ω |≥ 3 that(cid:54) ( p Ω , q Ω ) ≤ π/ 2 and (cid:54) ( p Ω , q Ω ) ≤
Revised: [Equation] Therefore, as long as sign( p ) = sign( q ) (coordinate-wise) and max | Ω | =2 (cid:54) ( p Ω , q Ω ) ≤ (cid:15)/n 2 , we would have for all | Ω |≥ 3 that

Original: [Equation]
Revised: [Equation]

Original: ( p Ω (cid:48) , q Ω (cid:48) ) ≤ (cid:18) | Ω | (cid:19) · (cid:15) n 2 ≤ (cid:15).

Original: By Eq.
Revised: By Eq.

Original: (4.5), the above implies that d E ( p , q ) ≤ (cid:15)/π , the desired result.
Revised: (4.5), the above implies that d E ( p , q ) ≤ (cid:15)/π , the desired result.

Original: Hence the task reduces to constructing an η = (cid:15)/n 2 covering in d 2 over any consistent sign pattern.
Revised: Hence the task reduces to constructing an η = (cid:15)/n 2 covering in d 2 over any consistent sign pattern.

Original: Our second step is a tight bound on this covering number : the η -covering number in d 2 is bounded by exp( Cn log( n/η ))
Revised: Our second step is a tight bound on this covering number : the η -covering number in d 2 is bounded by exp( Cn log( n/η ))

Original: (Lemma C.3).
Revised: (Lemma C.3).

Original: For bounding this, a ﬁrst thought would be to take the covering in all size-2 angles (there are (cid:0) n 2 (cid:1) of them) and take the common reﬁnement of all their partitions, which gives covering number ( C/η ) O ( n 2 ) =
Revised: For bounding this, a ﬁrst thought would be to take the covering in all size-2 angles (there are (cid:0) n 2 (cid:1) of them) and take the common reﬁnement of all their partitions, which gives covering number ( C/η ) O ( n 2 ) =

Original: exp( Cn 2 log(1 /η )) .
Revised: exp( Cn 2 log(1 /η )) .

Original: We improve upon this strategy by sorting the coordinates in p and restricting attentions in the consecutive size-2 angles after the sorting (there are n − 1 of them).
Revised: We improve upon this strategy by sorting the coordinates in p and restricting attentions in the consecutive size-2 angles after the sorting (there are n − 1 of them).

Original: We show that a proper covering in these consecutive size-2 angles by η/n will yield a covering for all size-2 angles by η .
Revised: We show that a proper covering in these consecutive size-2 angles by η/n will yield a covering for all size-2 angles by η .

Original: The corresponding covering number in this case is thus ( Cn/η ) O ( n ) =
Revised: The corresponding covering number in this case is thus ( Cn/η ) O ( n ) =

Original: exp( Cn log( n/η )) , which modulo the log n factor is the tightest we can get.
Revised: exp( Cn log( n/η )) , which modulo the log n factor is the tightest we can get.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Setup We set the true dictionary
Revised: Setup We set the true dictionary

Original: A to be the identity and random orthogonal matrices, respectively.
Revised: A to be the identity and random orthogonal matrices, respectively.

Original: For each choice, we sweep the combinations of ( m, n ) with n ∈ { 30 , 50 , 70 , 100 } and m = 10 n { 0 .
Revised: For each choice, we sweep the combinations of ( m, n ) with n ∈ { 30 , 50 , 70 , 100 } and m = 10 n { 0 .

Original: 5 , 1 , 1 . 5 , 2 , 2 . 5 } , and ﬁx the sparsity level at θ = 0 . 1 , 0 .
Revised: 5 , 1 , 1 . 5 , 2 , 2 . 5 } , and ﬁx the sparsity level at θ = 0 . 1 , 0 .

Original: 3 , 0 . 5 , respectively.
Revised: 3 , 0 . 5 , respectively.

Original: For each ( m, n ) pair, we generate 10 problem instances, corresponding to re-sampling the coefﬁcient matrix X fortimes.
Revised: For each ( m, n ) pair, we generate 10 problem instances, corresponding to re-sampling the coefﬁcient matrix X fortimes.

Original: Note that our theoretical guarantee applies for m = (cid:101) Ω( n 4 ) , and the sample complexity we experiment with here is lower than what our theory requires.
Revised: Note that our theoretical guarantee applies for m = (cid:101) Ω( n 4 ) , and the sample complexity we experiment with here is lower than what our theory requires.

Original: To recover the dictionary, we run the Riemannian subgradient descent algorithm Eq.
Revised: To recover the dictionary, we run the Riemannian subgradient descent algorithm Eq.

Original: (3.16) with decaying step size η ( k ) = 1 / √ k , corresponding to the boundary case α = 1 / 2 in Theorem 3.8 with a much better base size.
Revised: (3.16) with decaying step size η ( k ) = 1 / √ k , corresponding to the boundary case α = 1 / 2 in Theorem 3.8 with a much better base size.

Original: Metric As Theorem 3.1 guarantees recovering the entire dictionary with R ≥ Cn log n independent runs, we perform R = round (5 n log n ) runs on each instance.
Revised: Metric As Theorem 3.1 guarantees recovering the entire dictionary with R ≥ Cn log n independent runs, we perform R = round (5 n log n ) runs on each instance.

Original: For each run, a true dictionary element a i is considered to be found if (cid:107) a i − q best (cid:107) ≤ 10 − 3 .
Revised: For each run, a true dictionary element a i is considered to be found if (cid:107) a i − q best (cid:107) ≤ 10 − 3 .

Original: For each instance, we regard it a successful recovery if the R = round (5 n log n ) runs have found all the dictionary elements, and we report the empirical success rate over the 10 instances.
Revised: For each instance, we regard it a successful recovery if the R = round (5 n log n ) runs have found all the dictionary elements, and we report the empirical success rate over the 10 instances.

Original: Result From our simulations, Riemannian subgradient descent succeeds in recovering the dictionary as long as m ≥ Cn 2 (Fig. 2), across different sparsity level θ .
Revised: Result From our simulations, Riemannian subgradient descent succeeds in recovering the dictionary as long as m ≥ Cn 2 (Fig. 2), across different sparsity level θ .

Original: The dependency on n is consistent with our theory and suggests that the actual sample complexity requirement for guaranteed recovery might be even lower than (cid:101) O
Revised: The dependency on n is consistent with our theory and suggests that the actual sample complexity requirement for guaranteed recovery might be even lower than (cid:101) O

Original: ( n 4 ) we established.
Revised: ( n 4 ) we established.

Original: 3 The (cid:101) O ( n 2 ) rate we observe also matches the results based on the SOS method (Barak et al., 2015; Ma et al., 2016; Schramm & Steurer, 2017).
Revised: 3 The (cid:101) O ( n 2 ) rate we observe also matches the results based on the SOS method (Barak et al., 2015; Ma et al., 2016; Schramm & Steurer, 2017).

Original: Moreover, the problem seems to become harder when θ grows, evident from the observation that the success transition threshold being pushed to the right.
Revised: Moreover, the problem seems to become harder when θ grows, evident from the observation that the success transition threshold being pushed to the right.

Original: 2.5 number of measurements (as power of n)0.1 0. 0.9f r ac t i on o f s u ccess f u l r ec o ve r y n=30n=50n=70n= 0. 2.5 number of measurements (as power of n)0.1 0. 0.9f r ac t i on o f s u ccess f u l r ec o ve r y n=30n=50n=70n= 0.
Revised: [Figure]

Original: 2.5 number of measurements (as power of n)0.1 0. 0.9f r ac t i on o f s u ccess f u l r ec o ve r y n=30n=50n=70n= Faster alternative  for large-scale instancesThe Riemannian subgradient descent is cheap per iteration but slow in overall convergence, similar to many other ﬁrst-order methods. We also test a faster quasi-Newton type method, GRANSO, 4 that employs BFGS for solving constrained nonsmooth problems based on sequential quadratic optimization (Curtis et al., 2017).
Revised: Additional experiments A faster alternative algorithm for large-scale instances is tested in Appendix H. A complementary experiment on real images is included as Appendix I.

Original: For a large dictionary of dimension n = 400 and sample complexity m = 10 n 2 (i.e., 1 . 6 × 10 6 ), GRANSO successfully identiﬁes a basis after 1500 iterations with CPU time 4 hours on a two-socket Intel Xeon E5-2640v processor (10-core Broadwell, 2.40 GHz)—this is approximately 10 × faster than the Riemannian subgradient descent method, showing the potential of quasi-Newton type methods for solving largescale problems.

Original: Experiment with image data A complementary experiment on real images is included as Appendix H.

Original:  C ONCLUSION AND F UTURE D IRECTIONS
Revised:  C ONCLUSION AND F UTURE D IRECTIONS

Original: This paper presents the ﬁrst theoretical guarantee for orthogonal dictionary learning using subgradient descent on a natural (cid:96) 1 minimization formulation.
Revised: This paper presents the ﬁrst theoretical guarantee for orthogonal dictionary learning using subgradient descent on a natural (cid:96) 1 minimization formulation.

Original: Along the way, we develop tools for analyzing the optimization landscape of nonconvex nonsmooth functions, which could be of broader interest.
Revised: Along the way, we develop tools for analyzing the optimization landscape of nonconvex nonsmooth functions, which could be of broader interest.

Original: For futute work, there is an O ( n 2 ) sample complexity gap between what we established in Theorem 3.1, and what we observed in the simulations alongside previous results based on the SOS method (Barak et al., 2015; Ma et al., 2016; Schramm & Steurer, 2017).
Revised: For futute work, there is an O ( n 2 ) sample complexity gap between what we established in Theorem 3.1, and what we observed in the simulations alongside previous results based on the SOS method (Barak et al., 2015; Ma et al., 2016; Schramm & Steurer, 2017).

Original: As our main geometric result Theorem 3.6 already achieved tight bounds on the directional derivatives, further sample complexity improvement could potentially come out of utilizing second-order information such as the strong negative curvature (Lemma B.2), or careful algorithm-dependent analysis.
Revised: As our main geometric result Theorem 3.6 already achieved tight bounds on the directional derivatives, further sample complexity improvement could potentially come out of utilizing second-order information such as the strong negative curvature (Lemma B.2), or careful algorithm-dependent analysis.

Original: While our result applies only to (complete) orthogonal dictionaries, a natural question is whether we can generalize to overcomplete dictionaries.
Revised: While our result applies only to (complete) orthogonal dictionaries, a natural question is whether we can generalize to overcomplete dictionaries.

Original: To date the only known provable algorithms for learning overcomplete dictionaries in the linear sparsity regime are based on the SOS method (Barak et al., 2015; Ma et al., 2016; Schramm & Steurer, 2017).
Revised: To date the only known provable algorithms for learning overcomplete dictionaries in the linear sparsity regime are based on the SOS method (Barak et al., 2015; Ma et al., 2016; Schramm & Steurer, 2017).

Original: We believe that our nonsmooth analysis has the potential of handling over-complete dictionaries, as for reasonably well-conditioned overcomplete dictionaries A , each a i (columns of A ) makes a (cid:62)
Revised: We believe that our nonsmooth analysis has the potential of handling over-complete dictionaries, as for reasonably well-conditioned overcomplete dictionaries A , each a i (columns of A ) makes a (cid:62)

Original: A approximately 1 -sparse and so a (cid:62) i AX gives noisy estimate of a certain row of X .
Revised: A approximately 1 -sparse and so a (cid:62) i AX gives noisy estimate of a certain row of X .

Original: So the same formulation as Eq.
Revised: So the same formulation as Eq.

Original: (1.1) intuitively still works.
Revised: (1.1) intuitively still works.

Original: We would like to leave that to future work.
Revised: We would like to leave that to future work.

Original: Nonsmooth phase retrieval and deep networks with ReLU mentioned in Section 1.1 are examples of many nonsmooth, nonconvex problems encountered in practice.
Revised: Nonsmooth phase retrieval and deep networks with ReLU mentioned in Section 1.1 are examples of many nonsmooth, nonconvex problems encountered in practice.

Original: Most existing theoretical results on these problems tend to be technically vague about handling the nonsmooth points: they either prescribe a rule for choosing a subgradient element, which effectively disconnects theory and practice because numerical testing of nonsmooth points is often not reliable, or ignore the nonsmooth points altogether, assuming that practically numerical methods would never touch these points—this sounds intuitive but no formalism on this appears in the relevant literature yet.
Revised: Most existing theoretical results on these problems tend to be technically vague about handling the nonsmooth points: they either prescribe a rule for choosing a subgradient element, which effectively disconnects theory and practice because numerical testing of nonsmooth points is often not reliable, or ignore the nonsmooth points altogether, assuming that practically numerical methods would never touch these points—this sounds intuitive but no formalism on this appears in the relevant literature yet.

Original: Besides our work, (Laurent & von Brecht, 2017; Kakade & Lee, 2018) also warns about potential problems of ignoring nonsmooth points when studying optimization of nonsmooth functions in machine learning.
Revised: Besides our work, (Laurent & von Brecht, 2017; Kakade & Lee, 2018) also warns about potential problems of ignoring nonsmooth points when studying optimization of nonsmooth functions in machine learning.

Original: A T ECHNICAL TOOLS A. H AUSDORFF DISTANCE
Revised: A T ECHNICAL TOOLS A. H AUSDORFF DISTANCE

Original: We need the Hausdorff metric to measure differences between nonempty sets.
Revised: We need the Hausdorff metric to measure differences between nonempty sets.

Original: For any set X and a point p in R n , the point-to-set distance is deﬁned as
Revised: For any set X and a point p in R n , the point-to-set distance is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: For any two sets X
Revised: For any two sets X

Original: X 2 ∈ R n the Hausdorff distance is deﬁned as
Revised: X 2 ∈ R n the Hausdorff distance is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: When X 1 is a singleton say X 1 = { p } .
Revised: When X 1 is a singleton say X 1 = { p } .

Original: Then
Revised: Then

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Moreover, for any sets X 1 , X 2 , Y 1 , Y 2 ⊂ R n ,
Revised: Moreover, for any sets X 1 , X 2 , Y 1 , Y 2 ⊂ R n ,

Original: [Equation]
Revised: [Equation]

Original: On the sets of nonempty, compact subsets of R n , the Hausdorff metric is a valid metric; particularly, it obeys the triangular inequality: for nonempty, compact subsets X, Y, Z ⊂ R n ,[Equation] H
Revised: On the sets of nonempty, compact subsets of R n , the Hausdorff metric is a valid metric; particularly, it obeys the triangular inequality: for nonempty, compact subsets X, Y, Z ⊂ R n ,

Original: [Equation]
Revised: [Equation]

Original: See, e.g., Sec. 7.1 of Sternberg (2013) for a proof.
Revised: See, e.g., Sec. 7.1 of Sternberg (2013) for a proof.

Original: Lemma A.1 (Restatement of Lemma A.1) .
Revised: Lemma A.1 (Restatement of Lemma A.1) .

Original: For convex compact sets X, Y ⊂ R n , we have
Revised: For convex compact sets X, Y ⊂ R n , we have

Original: [Equation]
Revised: [Equation]

Original: where h S ( u ) .
Revised: where h S ( u ) .

Original: = sup x ∈ S (cid:104) x , u (cid:105) is the support function associated with the set S .
Revised: = sup x ∈ S (cid:104) x , u (cid:105) is the support function associated with the set S .

Original: A. S UB -G AUSSIAN RANDOM MATRICES AND PROCESSES
Revised: A. S UB -G AUSSIAN RANDOM MATRICES AND PROCESSES

Original: Proposition A.2 (Talagrand’s comparison inequality, Corollary 8.6.3 and Exercise 8.6.5 of Vershynin (2018)) .
Revised: Proposition A.2 (Talagrand’s comparison inequality, Corollary 8.6.3 and Exercise 8.6.5 of Vershynin (2018)) .

Original: Let { X x } x ∈ T be a zero-mean random process on a subset T ⊂ R n .
Revised: Let { X x } x ∈ T be a zero-mean random process on a subset T ⊂ R n .

Original: Assume that for all x , y ∈ T we have
Revised: Assume that for all x , y ∈ T we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, for any t >
Revised: Then, for any t >

Original: [Equation]
Revised: [Equation]

Original: (A.8) with probability at least 1 − 2 exp (cid:0) − t 2 (cid:1) .
Revised: (A [Equation] [Equation] with probability at least 1 − 2 exp (cid:0) − t 2 (cid:1) .

Original: Here w ( T ) .
Revised: Here w ( T ) .

Original: =
Revised: =

Original: E g ∼N ( 0 , I ) sup x ∈ T (cid:104) x , g (cid:105) is the Gaussian width of T and rad( T ) =
Revised: E g ∼N ( 0 , I ) sup x ∈ T (cid:104) x , g (cid:105) is the Gaussian width of T and rad( T ) =

Original: sup x ∈ T (cid:107) x (cid:107) is the radius of T .
Revised: sup x ∈ T (cid:107) x (cid:107) is the radius of T .

Original: Proposition A.3 (Deviation inequality for sub-Gaussian matrices, Theorem 9.1.1 and Exercise 9.1. of Vershynin (2018)) .
Revised: Proposition A.3 (Deviation inequality for sub-Gaussian matrices, Theorem 9.1.1 and Exercise 9.1. of Vershynin (2018)) .

Original: Let A be an n × m matrix whose rows
Revised: Let A be an n × m matrix whose rows

Original: A i ’s are independent, isotropic
Revised: A i ’s are independent, isotropic

Original: [Equation]
Revised: [Equation]

Original: and sub-Gaussian random vectors in R m .
Revised: and sub-Gaussian random vectors in R m .

Original: Then for any subset T ⊂ R m , we have
Revised: Then for any subset T ⊂ R m , we have

Original: [Equation]
Revised: [Equation]

Original: Here K = max i (cid:13)(cid:13)
Revised: Here K = max i (cid:13)(cid:13)

Original: A i (cid:13)(cid:13) ψ 2 .
Revised: A i (cid:13)(cid:13) ψ 2 .

Original: B P ROOFS FOR S ECTION 3. B. P ROOF OF P ROPOSITION 3. 
Revised: B P ROOFS FOR S ECTION 3. B. P ROOF OF P ROPOSITION 3.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the last equality is obtained by conditioning on Ω and the fact that E Z ∼ N ( [Equation] ,σ 2 ) [ | Z | ] = (cid:112) 2 /πσ .
Revised: where the last equality is obtained by conditioning on Ω and the fact that E Z ∼ N (0 ,σ 2 ) [ | Z | ] = (cid:112) 2 /πσ .

Original: The subdifferential expression comes from
Revised: The subdifferential expression comes from

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: and the fact that ∂ E [ f ] ( q ) =
Revised: and the fact that ∂ E [ f ] ( q ) =

Original: ∂ E [ (cid:107) q Ω (cid:107) 2 ] =
Revised: ∂ E [ (cid:107) q Ω (cid:107) 2 ] =

Original: E [ ∂ (cid:107) q Ω (cid:107) 2 ] as the sub-differential and expectation can be exchanged for convex functions (Hiriart-Urruty & Lemar´echal, 2001).
Revised: E [ ∂ (cid:107) q Ω (cid:107) 2 ] as the sub-differential and expectation can be exchanged for convex functions (Hiriart-Urruty & Lemarchal, 2001).

Original: By the same exchangability result, we also have E [ ∂f ( q )] =
Revised: By the same exchangability result, we also have E [ ∂f ( q )] =

Original: ∂ E [ f ] ( q ) .
Revised: ∂ E [ f ] ( q ) .

Original: B. P ROOF OF P ROPOSITION 3. 
Revised: B. P ROOF OF P ROPOSITION 3.

Original: We ﬁrst show that points in the claimed set are indeed stationary points by taking the choice v Ω =in Eq.
Revised: We ﬁrst show that points in the claimed set are indeed stationary points by taking the choice v Ω =in Eq.

Original: (3.5), giving the subgradient choice E [ ∂f ] ( q ) =
Revised: (3.5), giving the subgradient choice E [ ∂f ] ( q ) =

Original: E [ q Ω / (cid:107) q Ω (cid:107) 2 

Revised: E [ q Ω / (cid:107) q Ω (cid:107) 2

Original: q Ω (cid:54) = 0 } ] .
Revised: q Ω (cid:54) = 0 } ] .

Original: Let q ∈ S and such that (cid:107) q (cid:107) 

Revised: Let q ∈ S and such that (cid:107) q (cid:107)

Original: = k .
Revised: = k .

Original: For all j ∈ supp( q ) , we have
Revised: For all j ∈ supp( q ) , we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for all j / ∈ supp( q ) , we always have [ q Ω ] j = 0 , so e (cid:62) j E
Revised: On the other hand, for all j / ∈ supp( q ) , we always have [ q Ω ] j = 0 , so e (cid:62) j E

Original: [ ∂f ] ( q ) = 0 .
Revised: [ ∂f ] ( q ) = 0 .

Original: Therefore, we have that E [ ∂f ] ( q ) =
Revised: Therefore, we have that E [ ∂f ] ( q ) =

Original: c ( θ, k ) q , and so
Revised: c ( θ, k ) q , and so

Original: [Equation]
Revised: [Equation]

Original: q [Equation] [Equation] Therefore q ∈ S is stationary.
Revised: Therefore q ∈ S is stationary.

Original: To see that {± e i : i ∈ [ n ] } are the global minima, note that for all q ∈ S n − 1 , we have
Revised: To see that {± e i : i ∈ [ n ] } are the global minima, note that for all q ∈ S n − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Equality holds if and only if (cid:107) q Ω (cid:107) 2 ∈ { 0 , 1 } almost surely, which is only satisﬁed at q ∈ {± e i : i ∈ [ n ] } .
Revised: Equality holds if and only if (cid:107) q Ω (cid:107) 2 ∈ { 0 , 1 } almost surely, which is only satisﬁed at q ∈ {± e i : i ∈ [ n ] } .

Original: To see that the other q ’s are saddles, we only need to show that there exists a tangent direction along which q is local max.
Revised: To see that the other q ’s are saddles, we only need to show that there exists a tangent direction along which q is local max.

Original: Indeed, for any other q , there exists at least two non-zero entries (with equal absolute value): WLOG assume that q 1 = q n > 0 .
Revised: Indeed, for any other q , there exists at least two non-zero entries (with equal absolute value): WLOG assume that q 1 = q n > 0 .

Original: Using the reparametrization in Appendix B. and applying Lemma B.2, we get that E [ f ] ( q ) is directionally differentiable along [ − q − n ; 1 − q 2 n q n ] , with derivative zero (necessarily, because 0 ∈ E [ ∂ R f ] ( q ) ) and strictly negative second derivative.
Revised: Using the reparametrization in Appendix B. and applying Lemma B.2, we get that E [ f ] ( q ) is directionally differentiable along [ − q − n ; 1 − q 2 n q n ] , with derivative zero (necessarily, because 0 ∈ E [ ∂ R f ] ( q ) ) and strictly negative second derivative.

Original: Therefore E [ f ] ( q ) is locally maximized at q along this tangent direction, which shows that q is a saddle point.
Revised: Therefore E [ f ] ( q ) is locally maximized at q along this tangent direction, which shows that q is a saddle point.

Original: The other direction (all other points are not stationary) is implied by Theorem 3.4, which guarantees that 0 / ∈ E
Revised: The other direction (all other points are not stationary) is implied by Theorem 3.4, which guarantees that 0 / ∈ E

Original: [ ∂ R f ] ( q ) whenever q / ∈ S .
Revised: [ ∂ R f ] ( q ) whenever q / ∈ S .

Original: Indeed, as long as q / ∈ S , q has a max absolute value coordinate (say n ) and another non-zero coordinate with strictly smaller absolute value (say j ).
Revised: Indeed, as long as q / ∈ S , q has a max absolute value coordinate (say n ) and another non-zero coordinate with strictly smaller absolute value (say j ).

Original: For this pair of indices, the proof of Theorem 3.4(a) goes through for index j (even if q ∈ S ( n +) ζdoes not necessarily hold because the max index might not be unique), which implies that 0 / ∈ E
Revised: For this pair of indices, the proof of Theorem 3.4(a) goes through for index j (even if q ∈ S ( n +) ζdoes not necessarily hold because the max index might not be unique), which implies that 0 / ∈ E

Original: [ ∂ R f ] ( q ) .
Revised: [ ∂ R f ] ( q ) .

Original: B. R EPARAMETRIZATION
Revised: B. R EPARAMETRIZATION

Original: For analysis purposes, we introduce the reparametrization w = q 1:( n − 1) in the region S ( n +), following (Sun et al., 2015) .
Revised: For analysis purposes, we introduce the reparametrization w = q 1:( n − 1) in the region S ( n +), following (Sun et al., 2015) .

Original: With this reparametrization, the problem becomes minimize w ∈ R n − 1 g ( w ) .
Revised: With this reparametrization, the problem becomes minimize w ∈ R n − 1 g ( w ) .

Original: = (cid:114) π 2 · m (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:20) w ; (cid:113) 1 − (cid:107) w (cid:107) 2 (cid:21) (cid:62) X
Revised: = (cid:114) π 2 · m (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:20) w ; (cid:113) 1 − (cid:107) w (cid:107) 2 (cid:21) (cid:62) X

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) subject to (cid:107) w (cid:107) ≤ (cid:114) n −n .
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) subject to (cid:107) w (cid:107) ≤ (cid:114) n −n .

Original: (B.8)
Revised: (B.8)

Original: The constraint comes from the fact that q n ≥ 1 / √ n and thus (cid:107) w (cid:107)≤ (cid:112) ( n − 1) /n .
Revised: The constraint comes from the fact that q n ≥ 1 / √ n and thus (cid:107) w (cid:107)≤ (cid:112) ( n − 1) /n .

Original: Lemma B.1.
Revised: Lemma B.1.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Direct calculation gives
Revised: Direct calculation gives

Original: [Equation]
Revised: [Equation]

Original: as claimed.
Revised: as claimed.

Original: [Equation]
Revised: [Equation]

Original: Lemma B.2 (Negative-curvature region) .
Revised: Lemma B.2 (Negative-curvature region) .

Original: For all unit vector v ∈ S n − 1 and all s ∈ (0 , 1) , let
Revised: For all unit vector v ∈ S n − 1 and all s ∈ (0 , 1) , let

Original: [Equation]
Revised: [Equation]

Original: (B.14) it holds that ∇

Original: [Equation]
Revised: [Equation]

Original: B
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: In other words, for all w (cid:54) = 0 , ± w / (cid:107) w (cid:107) is a direction of negative curvature.
Revised: In other words, for all w (cid:54) = 0 , ± w / (cid:107) w (cid:107) is a direction of negative curvature.

Original: Proof.
Revised: Proof.

Original: By Lemma B.1,
Revised: By Lemma B.1,

Original: [Equation]
Revised: [Equation]

Original: For s ∈ (0 , 1) , h v ( s ) is twice differentiable, and we have
Revised: For s ∈ (0 , 1) , h v ( s ) is twice differentiable, and we have

Original: [Equation]
Revised: [Equation]

Original: completing the proof.
Revised: completing the proof.

Original: [Equation]
Revised: [Equation]

Original: Lemma B.3 (Inward gradient) .
Revised: Lemma B.3 (Inward gradient) .

Original: For any w with (cid:107)w
Revised: For any w with (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any unit vector v ∈ R n − 1 , deﬁne h v ( t ) .
Revised: For any unit vector v ∈ R n − 1 , deﬁne h v ( t ) .

Original: =
Revised: =

Original: E [ g ] ( t v ) for t ∈ (0 , 1) .
Revised: E [ g ] ( t v ) for t ∈ (0 , 1) .

Original: We have from Lemma B.
Revised: We have from Lemma B.

Original: [Equation]
Revised: [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: − θ n −[-1 (cid:88) i = E Ω tv 
-]
Revised: − θ n −[-1 (cid:88) i = E Ω tv 
-]

Original: i 1 { i / ∈ Ω } (cid:113) 1 − t 2 v 2 i 1 { i / ∈ Ω } − t 2 (cid:13)(cid:13) v Ω c \ i (cid:13)(cid:13)(B.23) = θ (1 − θ ) n − 1 (cid:88) i = E Ω   v 2 i (cid:113) v 2 i + (cid:13)(cid:13) v Ω \ i (cid:13)(cid:13)− tv 2 i (cid:113) 1 − t 2 v 2 i − t 2 (cid:13)(cid:13) v Ω c \ i (cid:13)(cid:13)  (B.24) =
Revised: [Equation]

Original: θ (1 − θ ) t n − 1 (cid:88) 
Revised: (cid:88) i =

Original: [-i = v 
 i-] EΩ   (cid:113) t 2 v 2 i + t 2 (cid:13)(cid:13) v Ω \ i (cid:13)(cid:13)− (cid:113) 1 − t 2 v 2 i − t 2 (cid:13)(cid:13) v Ω c \ i (cid:13)(cid:13)  (B.25) =
Revised: [-i = v 
 i-] E

Original: [Equation]
Revised: [Equation]

Original: (1 − θ ) t n
Revised: tv

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:88) i = v 2 i E Ω   (cid:113) t 2 v 2 i 
Revised: i 1 { i /

Original: [Equation]
Revised: [Equation]

Revised: Ω } (cid:113)

Revised: [Equation]

Original: t 2 (cid:13)(cid:13) v Ω \
Revised: t [Equation] v [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: i (cid:13)(cid:13)− (cid:113) 1 − t 2 (cid:107) v (cid:107) 2 + t 2 (cid:13)(cid:13) v Ω  \ i (cid:13)(cid:13)  .
Revised: t [Equation] (cid:13)(cid:13) v Ω c \ i (cid:13)(cid:13) [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: Ω c \ i (cid:13)(cid:13)

Revised: [Equation]

Revised: 

Original: (B.26)
Revised: [Equation] [Equation] (cid:13)(cid:13) [Equation]

Revised: (cid:13)(cid:13)

Revised: [Equation]

Revised: (cid:113)

Original: We are interested in the regime of t so that
Revised: [Equation] i [Equation] [Equation]

Revised: [Equation]

Revised: Ω \

Revised: [Equation]

Revised: 

Revised: 

Original: [Equation]
Revised: [Equation]

Revised: We are interested in the regime of t so that

Original: [Equation]
Revised: [Equation]

Original: So ∇ t h v ( t ) ≥ 0 holds always for t ≤ 1 / (cid:113) 1 + (cid:107) v (cid:107) 2 ∞ .
Revised: So ∇ t h v ( t ) ≥ 0 holds always for t ≤ 1 / (cid:113) 1 + (cid:107) v (cid:107) 2 ∞ .

Original: By Lemma B.2, ∇ 2 h v ( t ) ≤ − θ (1 − θ ) over t ∈ (0 , 1) , which implies (cid:104)∇ 
Revised: By Lemma B.2, ∇ 2 h v ( t ) ≤ − θ (1 − θ ) over t ∈ (0 , 1) , which implies (cid:104)∇ t h [Equation] t 1 ) − ∇ t h [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Revised: , t

Revised: [Equation]

Original: (B.28)
Revised: (B.28)

Original: Taking t 1 = 1 / (cid:113) 1 + (cid:107) v (cid:107) 2 ∞ and considering t 2 ∈ [0 , t 1 ] , we have
Revised: Taking t 1 = 1 / (cid:113) 1 + (cid:107) v (cid:107) 2 ∞ and considering t 2 ∈ [0 , t 1 ] , we have

Original: [Equation]
Revised: [Equation]

Original: B.29)
Revised: (B.29)

Original: For any w , applying the above result to the unit vector w / (cid:107) w (cid:107) and recognizing that ∇ t h w / (cid:107) w (cid:107) ( t ) =
Revised: For any w , applying the above result to the unit vector w / (cid:107) w (cid:107) and recognizing that ∇ t h w / (cid:107) w (cid:107) ( t ) =

Original: D w / (cid:107) w (cid:107) g
Revised: D w / (cid:107) w (cid:107) g

Original: ( w ) =
Revised: ( w ) =

Original: D c w / (cid:107) w (cid:107) g
Revised: D c w / (cid:107) w (cid:107) g

Original: ( w ) , we complete the proof.
Revised: ( w ) , we complete the proof.

Original: (cid:4)
Revised: (cid:4)

Original: B. P ROOF OF T HEOREM 3. 
Revised: B. P ROOF OF T HEOREM 3.

Original: We ﬁrst show Eq.
Revised: We ﬁrst show Eq.

Original: (3.9) using the reparametrization in Appendix B.3.
Revised: (3.9) using the reparametrization in Appendix B.3.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: B.30) where the second equality follows by differentiating g via the chain rule.
Revised: B.30) where the second equality follows by differentiating g via the chain rule.

Original: Now, by Lemma B.3,
Revised: Now, by Lemma B.3,

Original: [Equation]
Revised: [Equation]

Original: For each radial direction v .
Revised: For each radial direction v .

Original: = w / (cid:107) w (cid:107) , consider points of the form t v with t ≤ 1 / (cid:113) 1 + (cid:107) v (cid:107) 2 ∞ .
Revised: = w / (cid:107) w (cid:107) , consider points of the form t v with t ≤ 1 / (cid:113) 1 + (cid:107) v (cid:107) 2 ∞ .

Original: Obviously, the function
Revised: Obviously, the function

Original: [Equation]
Revised: [Equation]

Original: is monotonically decreasing wrt t .
Revised: is monotonically decreasing wrt t .

Original: Thus, to derive a lower bound, it is enough to consider the largest t allowed.
Revised: Thus, to derive a lower bound, it is enough to consider the largest t allowed.

Original: In S ( n +) ζ, the limit amounts to requiring
Revised: In S ( n +) ζ, the limit amounts to requiring

Original: [Equation]
Revised: [Equation]

Original: So for any ﬁxed v and all allowed t for points in S ( n +) ζ, a uniform lower bound is
Revised: So for any ﬁxed v and all allowed t for points in S ( n +) ζ, a uniform lower bound is

Original: [Equation]
Revised: [Equation]

Original: So we conclude that for all q ∈ S ( n +)
Revised: So we conclude that for all q ∈ S ( n +)

Original: ζ
Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: We now turn to showing Eq.
Revised: We now turn to showing Eq.

Original: For e j with q j (cid:54) = 0 ,
Revised: For e j with q j (cid:54) = 0 ,

Original: [Equation]
Revised: [Equation]

Original: { j
Revised: { j

Original: [Equation]

Original: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: θ.
Revised: θ.

Original: (B.37)
Revised: (B.37)

Original: So for all j with q j (cid:54) = 0 , we have
Revised: So for all j with q j (cid:54) = 0 , we have

Original: [Equation]
Revised: [Equation]

Original:  =
Revised: 38) =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 

Original: [Equation]
Revised: [Equation]

Original: completing the proof
Revised: completing the proof.

Original: [Equation]

Original: C P ROOFS FOR S ECTION 3. C.C OVERING IN THE d E METRIC 
Revised: C P ROOFS FOR S ECTION 3. C.

Revised: C OVERING IN THE d

Revised: E METRIC

Original: For any θ ∈ (0 , 1) , deﬁne
Revised: For any θ ∈ (0 , 1) , deﬁne

Original: [Equation]
Revised: [Equation]

Original: with x ∼ iid BG( θ )
Revised: with x ∼ iid BG( θ )

Original: [Equation]
Revised: [Equation]

Revised: (C.

Original: [Equation]
Revised: [Equation]

Original: We stress that this notion always depend on θ , and we will omit the subscript θ when no confusion is expected.
Revised: We stress that this notion always depend on θ , and we will omit the subscript θ when no confusion is expected.

Original: This indeed deﬁnes a metric on subsets of S n − 1 .
Revised: This indeed deﬁnes a metric on subsets of S n − 1 .

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Over any subset of S n − 1 with a consistent support pattern, d E is a valid metric.
Revised: Over any subset of S n − 1 with a consistent support pattern, d E is a valid metric.

Original: Proof.
Revised: Proof.

Original: Recall that (cid:54) ( x , y ) .
Revised: Recall that (cid:54) ( x , y ) .

Original: = arccos (cid:104) x , y (cid:105) deﬁnes a valid metric on S n − 1 .
Revised: = arccos (cid:104) x , y (cid:105) deﬁnes a valid metric on S n − 1 .

Original: In particular, the triangular inequality holds.
Revised: In particular, the triangular inequality holds.

Original: For d E and p , q ∈ S n − 1 with the same support pattern, we have
Revised: For d E and p , q ∈ S n − 1 with the same support pattern, we have

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: 0 , notboth
Revised: not

Original: [Equation]
Revised: [Equation]

Original: where we have adopted the convention that (cid:54) ( 0 , v ) .
Revised: where we have adopted the convention that (cid:54) ( 0 , v ) .

Original: = 

Revised: =

Original: for any v .
Revised: for any v .

Original: It is easy to verify that d E ( p , q ) = 

Revised: It is easy to verify that d E ( p , q ) =

Original: ⇐⇒ p = q , and d E ( p , q ) =
Revised: ⇐⇒ p = q , and d E ( p , q ) =

Original: d E ( q , p ) .
Revised: d E ( q , p ) .

Original: To show the triangular inequality, note that for any p , q and r with the same support pattern, p Ω , q Ω , and r Ω are either identically zero, or all nonzero.
Revised: To show the triangular inequality, note that for any p , q and r with the same support pattern, p Ω , q Ω , and r Ω are either identically zero, or all nonzero.

Original: For the former case,
Revised: For the former case,

Original: [Equation]
Revised: [Equation]

Original: holds trivially.
Revised: holds trivially.

Original: For the latter, since (cid:54) ( · , · ) obeys the triangular inequality uniformly over the sphere,
Revised: For the latter, since (cid:54) ( · , · ) obeys the triangular inequality uniformly over the sphere,

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: completing the proof.
Revised: completing the proof.

Original: [Equation]
Revised: [Equation]

Original: Lemma C.2 (Vector angle inequality) .
Revised: Lemma C.2 (Vector angle inequality) .

Original: For n ≥ 2 , consider u , v ∈ R n so that (cid:54) ( u , v ) ≤ π/ 2 .
Revised: For n ≥ 2 , consider u , v ∈ R n so that (cid:54) ( u , v ) ≤ π/ 2 .

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The inequality holds trivially when either of u , v is zero.
Revised: The inequality holds trivially when either of u , v is zero.

Original: Suppose they are both nonzero and wlog assume both are normalized, i.e., (cid:107) u (cid:107)
Revised: Suppose they are both nonzero and wlog assume both are normalized, i.e., (cid:107) u (cid:107)

Original: = (cid:107) v (cid:107) = 1 .
Revised: = (cid:107) v (cid:107) = 1 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: ( u 
Revised: ( u , v ) =

Original: [Equation]
Revised: [Equation]

Original:  v 
Revised: − cos 2 (cid:54) ( u , v ) (C.12) = (cid:107) [Equation] (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (C.12)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: − (cid:104)

Original: [Equation]
Revised: [Equation]

Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Revised: (C.13)

Original:  ( Lagrange’s identity ) 
Revised: = (cid:88) [Equation] [Equation] ( Lagrange’s identity ) (C.14)

Original: [Equation]
Revised: [Equation]

Revised: (cid:88) Ω

Original: [Equation] 
Revised: [Equation] ( [ n [Equation] [Equation]

Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: (cid:105)

Revised: [Equation]

Revised: (C.15) =

Revised: (cid:88) Ω ∈ ( [ n ]

Revised: [Equation]

Revised: ) (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original:  Ω 
Revised: (cid:107) v Ω (cid:107) 2 sin [Equation] (cid:54) ( u Ω [Equation]

Revised: v Ω ) (C.16) ≤ (cid:88) Ω ∈ ( [ n ]2 ) sin 2 (cid:54)

Original: [Equation]
Revised: [Equation]

Original: (C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: If (cid:80) Ω ∈ ( [ n ]2 ) (cid:54) ( u Ω , v Ω ) > π/ 2 , the claimed inequality holds trivially, as (cid:54) ( u , v ) ≤ π/ 2 by our assumption.
Revised: If (cid:80) Ω ∈ ( [ n ]2 ) (cid:54) ( u Ω , v Ω ) > π/ 2 , the claimed inequality holds trivially, as (cid:54) ( u , v ) ≤ π/ 2 by our assumption.

Original: Suppose (cid:80) Ω ∈ ( [ n ]2 ) (cid:54) ( u Ω , v Ω ) ≤ π/ 2 .
Revised: Suppose (cid:80) Ω ∈ ( [ n ]2 ) (cid:54) ( u Ω , v Ω ) ≤ π/ 2 .

Original: Then, (cid:88)
Revised: Then, (cid:88)

Original: [Equation]
Revised: [Equation]

Original: by recursive application of the following inequality: ∀ θ 1 , θ 2 ∈ [0 , π/ 2] with θ 1 + θ 2 ≤ π/ 2 , 
Revised: by recursive application of the following inequality: ∀ θ 1 , θ 2 ∈ [0 , π/ 2] with θ 1 + θ 2 ≤ π/ 2 , sin

Original: [Equation]
Revised: [Equation]

Original: (C
Revised: (C.19)

Original: [Equation] So we have that when (cid:80) Ω ∈ ( [ n ]2 ) (cid:54) ( u Ω 
Revised: So we have that when (cid:80) Ω ∈ ( [ n ]2 ) (cid:54) ( u Ω , v Ω ) ≤ π/

Original: [Equation]
Revised: [Equation]

Original: v Ω

Original: [Equation]
Revised: [Equation]

Original: ≤ π/
Revised: C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: as claimed.
Revised: as claimed.

Original: [Equation]
Revised: [Equation]

Original: Lemma C.3 (Covering in maximum length-2 angles) .
Revised: Lemma C.3 (Covering in maximum length-2 angles) .

Original: For any η ∈ (0 , 1 / 3) , there exists a subset
Revised: For any η ∈ (0 , 1 / 3) , there exists a subset

Original: Q ⊂ S n − 1 of size at most (5 n log(1 /η ) /η )
Revised: Q ⊂ S n − 1 of size at most (5 n log(1 /η ) /η )

Original: 2 n − 1 satisfying the following: for any p ∈ S n − 1 , there exists some q ∈ Q such that (cid:54) ( p Ω , q Ω ) ≤ η for all Ω ⊂ [ n ] with | Ω |≤ 2 .
Revised: 2 n − 1 satisfying the following: for any p ∈ S n − 1 , there exists some q ∈ Q such that (cid:54) ( p Ω , q Ω ) ≤ η for all Ω ⊂ [ n ] with | Ω |≤ 2 .

Original: Proof.
Revised: Proof.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: our goal is to give an η -covering of S n − 1 in the d 2 metric.
Revised: our goal is to give an η -covering of S n − 1 in the d 2 metric.

Original: Step 1 We partition S n − 1 according to the support, the sign pattern, and the ordering of the non-zero elements.
Revised: Step 1 We partition S n − 1 according to the support, the sign pattern, and the ordering of the non-zero elements.

Original: For each conﬁguration, we are going to construct a covering with the same conﬁguration of support, sign pattern, and ordering.
Revised: For each conﬁguration, we are going to construct a covering with the same conﬁguration of support, sign pattern, and ordering.

Original: There are no more than 3 n · n ! such conﬁgurations.
Revised: There are no more than 3 n · n ! such conﬁgurations.

Original: Note that we only need to construct one such covering for each support size, and for each support size we can ignore the zero entries – the angle (cid:54) ( p Ω , q Ω ) is always zero when p , q have matching support and Ω contains at least one zero index.
Revised: Note that we only need to construct one such covering for each support size, and for each support size we can ignore the zero entries – the angle (cid:54) ( p Ω , q Ω ) is always zero when p , q have matching support and Ω contains at least one zero index.

Original: Therefore, the task reduces to bounding the covering number of
Revised: Therefore, the task reduces to bounding the covering number of

Original: [Equation]
Revised: [Equation]

Original: in d 2 for all n .
Revised: in d 2 for all n .

Original: Step 2 We bound the covering number of A n by induction.
Revised: Step 2 We bound the covering number of A n by induction.

Original: Suppose that
Revised: Suppose that

Original: [Equation]
Revised: [Equation]

Original: holds for all n (cid:48) ≤ n − 1 .
Revised: holds for all n (cid:48) ≤ n − 1 .

Original: (The base case m = 2 clearly holds.)
Revised: (The base case m = 2 clearly holds.)

Original: Let C n (cid:48)
Revised: Let C n (cid:48)

Original: ⊂ S n (cid:48) − 1 be the correpsonding covering sets.
Revised: ⊂ S n (cid:48) − 1 be the correpsonding covering sets.

Original: We now construct a covering for A n .
Revised: We now construct a covering for A n .

Original: Let R .
Revised: Let R .

Original: = 1 /η
Revised: = 1 /η

Original: = r k for some r ≥ 1 and k to be determined.
Revised: = r k for some r ≥ 1 and k to be determined.

Original: Consider the set
Revised: Consider the set

Original: [Equation]
Revised: [Equation]

Original: We claim that Q r,k with properly chosen ( r, k ) gives a covering of
Revised: We claim that Q r,k with properly chosen ( r, k ) gives a covering of

Original: A n
Revised: A n

Original: [Equation]
Revised: [Equation]

Original: Indeed, we can decompose [1 , R ) into [1 , r ) , [ r, r 2 ) , . . .
Revised: Indeed, we can decompose [1 , R ) into [1 , r ) , [ r, r 2 ) , . . .

Original: , [ r k − 1 , R ) .
Revised: , [ r k − 1 , R ) .

Original: Each consecutive ratio p
Revised: Each consecutive ratio p

Original: i +1 /p
Revised: i +1 /p

Original: i falls in one of these intervals, and we choose q so that q
Revised: i falls in one of these intervals, and we choose q so that q

Original: i +1 /q
Revised: i +1 /q

Original: i is the left endpoint of this interval.
Revised: i is the left endpoint of this interval.

Original: Such a q satisﬁes q ∈ Q r,k and
Revised: Such a q satisﬁes q ∈ Q r,k and

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: for all i ∈ [n − 1] .
Revised: for all i ∈ [

Original: [Equation]
Revised: [Equation]

Original: C
Revised: − 1]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By multiplying these bounds, we obtain that for all 1 ≤ i < j ≤ n ,p
Revised: By multiplying these bounds, we obtain that for all 1 ≤ i < j ≤ n ,

Original: [Equation]
Revised: [Equation]

Original: /p i
Revised: p i

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]

Original: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: Take r = 1 + η/ 2 n , we have r n − 1 =
Revised: Take r = 1 + η/ 2 n , we have r n − 1 =

Original: (1 + η/ 2 n ) n − 1 ≤ exp( η/ 2) ≤ 1 + η .
Revised: (1 + η/ 2 n ) n − 1 ≤ exp( η/ 2) ≤ 1 + η .

Original: Therefore, for all i, j , we have p j /p
Revised: Therefore, for all i, j , we have p j /p

Original: i
Revised: i

Original: q j /q i ∈ [1 , 1 + η ) , which further implies that (cid:54) (( p i , p j ) , ( q i , q j ))
Revised: q j /q i ∈ [1 , 1 + η ) , which further implies that (cid:54) (( p i , p j ) , ( q i , q j ))

Original: ≤ η by Lemma F.4.
Revised: ≤ η by Lemma F.4.

Original: Thus we have for all | Ω |≤ 2 that (cid:54) ( p Ω , q Ω ) ≤ η .
Revised: Thus we have for all | Ω |≤ 2 that (cid:54) ( p Ω , q Ω ) ≤ η .

Original: (The size-1 angles are all zero as we have sign match.)
Revised: (The size-1 angles are all zero as we have sign match.)

Original: For this choice of r , we have k = log R/ log r and thus
Revised: For this choice of r , we have k = log R/ log r and thus

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: and we have N ( A n,R ) ≤ (cid:101) N n .
Revised: and we have N ( A n,R ) ≤ (cid:101) N n .

Original: Step 3 We now construct the covering of A n \ A n,R .
Revised: Step 3 We now construct the covering of A n \ A n,R .

Original: For any p ∈
Revised: For any p ∈

Original: A n \ A n,R , there exists some i such that p
Revised: A n \ A n,R , there exists some i such that p

Original: i +1 /p
Revised: i +1 /p

Original: i ∈
Revised: i ∈

Original: [ R, ∞ ) , which means that the angle of the ray ( p i , p i +1 ) is in between [arctan( R ) , π/ 2) =
Revised: [ R, ∞ ) , which means that the angle of the ray ( p i , p i +1 ) is in between [arctan( R ) , π/ 2) =

Original: [ π/ 2 − η, π/ 2) .
Revised: [ π/ 2 − η, π/ 2) .

Original: As p is sorted, we have that
Revised: As p is sorted, we have that

Original: [Equation]
Revised: [Equation]

Original: l

Original: [Equation]

Original: [Equation]

Original: (C

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: So if we take q such that q i +1 /q
Revised: So if we take q such that q i +1 /q

Original: i ∈
Revised: i ∈

Original: [ R, ∞ ) , q also has the above property, which gives that
Revised: [ R, ∞ ) , q also has the above property, which gives that

Original: [Equation]
Revised: [Equation]

Original: p
Revised: all j [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore to obtain the cover in d 2 , we only need to consider the angles for Ω ⊂ { 1 , . .
Revised: Therefore to obtain the cover in d 2 , we only need to consider the angles for Ω ⊂ { 1 , . .

Original: , i } and Ω ⊂ { i + 1 , . . . , n } , which can be done by taking the product of the covers in A i and A n − i .
Revised: , i } and Ω ⊂ { i + 1 , . . . , n } , which can be done by taking the product of the covers in A i and A n − i .

Original: By considering all i ∈ { 1 , . . . , n − 1 } , we obtain the bound
Revised: By considering all i ∈ { 1 , . . . , n − 1 } , we obtain the bound

Original: [Equation]
Revised: [Equation]

Original: Step 4 Putting together Step 2 and Step 3 and using the inductive assumption, we get that
Revised: Step 4 Putting together Step 2 and Step 3 and using the inductive assumption, we get that

Original: [Equation]
Revised: [Equation]

Original: This shows the case for m = n and completes the induction.
Revised: This shows the case for m = n and completes the induction.

Original: Step 5 Considering all conﬁgurations of { support, sign pattern, ordering } , we have
Revised: Step 5 Considering all conﬁgurations of { support, sign pattern, ordering } , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma C.4 (Covering number in the d E metric) .
Revised: Lemma C.4 (Covering number in the d E metric) .

Original: Assume n ≥ 3 .
Revised: Assume n ≥ 3 .

Original: There exists a numerical constant
Revised: There exists a numerical constant

Original: C > 0 such that for any (cid:15) ∈ (0 , 1) , S n − 1 admits an (cid:15) -net of size exp( Cn log n(cid:15) )
Revised: C > 0 such that for any (cid:15) ∈ (0 , 1) , S n − 1 admits an (cid:15) -net of size exp( Cn log n(cid:15) )

Original: w.r.t.
Revised: w.r.t.

Original: d E deﬁned in Eq.
Revised: d E deﬁned in Eq.

Original: (C.1) : for any p ∈ S n − 

Revised: (C.1) : for any p ∈ S n −

Original: , there exists a q in the net with supp ( q )
Revised: , there exists a q in the net with supp ( q )

Original: =
Revised: =

Original: supp ( p ) and d E ( p , q ) ≤ (cid:15) .
Revised: supp ( p ) and d E ( p , q ) ≤ (cid:15) .

Original: We say such (cid:15) nets are admissible for S n − 1 wrt d E .
Revised: We say such (cid:15) nets are admissible for S n − 1 wrt d E .

Original: Proof.
Revised: Proof.

Original: Let η =
Revised: Let η =

Original: cid:15)/n 2 .
Revised: cid:15)/n 2 .

Original: By Lemma C.3, there exists a subset Q ⊂ S n − 1 of size at most
Revised: By Lemma C.3, there exists a subset Q ⊂ S n − 1 of size at most

Original: [Equation]
Revised: [Equation]

Original: log [Equation] such that for any p ∈ S n − 1 , there exists q ∈ S n − 1 such that supp( p ) =
Revised: such that for any p ∈ S n − 1 , there exists q ∈ S n − 1 such that supp( p ) =

Original: supp( q ) and (cid:54) ( p Ω , q Ω ) ≤ η for all | Ω |≤ 2 .
Revised: supp( q ) and (cid:54) ( p Ω , q Ω ) ≤ η for all | Ω |≤ 2 .

Original: In particular, the | Ω | = 1 case says that sign( p )
Revised: In particular, the | Ω | = 1 case says that sign( p )

Original: =
Revised: =

Original: sign( q ) , which implies that
Revised: sign( q ) , which implies that

Original: [Equation]
Revised: [Equation]

Original: Thus, applying the vector angle inequality (Lemma C.2), for any p ∈ S n − 1 and the corresponding q ∈ Q , we have
Revised: Thus, applying the vector angle inequality (Lemma C.2), for any p ∈ S n − 1 and the corresponding q ∈ Q , we have

Original: [Equation]
Revised: [Equation]

Original: Summing up, we get
Revised: Summing up, we get

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Therefore d E ( p , q ) ≤ (cid:15) .
Revised: Therefore d E ( p , q ) ≤ (cid:15) .

Original: [Equation]
Revised: [Equation]

Original: Below we establish the ”Lipschitz” property in terms of d E distance.
Revised: Below we establish the ”Lipschitz” property in terms of d E distance.

Original: Lemma C.5.
Revised: Lemma C.5.

Original: Fix θ ∈ (0 , 1) .
Revised: Fix θ ∈ (0 , 1) .

Original: For any (cid:15) ∈ (0 , 1) , let N (cid:15) be an admissible (cid:15) -net for S n − 1 wrt d E .
Revised: For any (cid:15) ∈ (0 , 1) , let N (cid:15) be an admissible (cid:15) -net for S n − 1 wrt d E .

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x m be iid copies of x ∼ iid BG( θ ) in R n .
Revised: , x m be iid copies of x ∼ iid BG( θ ) in R n .

Original: When m ≥ C(cid:15) − 2 n , the inequality sup[Equation] C
Revised: When m ≥ C(cid:15) − 2 n , the inequality sup

Original: [Equation]
Revised: [Equation]

Original: holds with probability at least 1 − exp (cid:0) − c(cid:15)
Revised: holds with probability at least 1 − exp (cid:0) − c(cid:15)

Original: 2 m (cid:1) .
Revised: 2 m (cid:1) .

Original: Here C and c are universal constants independent of (cid:15) .
Revised: Here C and c are universal constants independent of (cid:15) .

Original: Proof.
Revised: Proof.

Original: We call any pair of p , q ∈ S n − 1 with q ∈ N (cid:15) , supp ( p ) =
Revised: We call any pair of p , q ∈ S n − 1 with q ∈ N (cid:15) , supp ( p ) =

Original: supp ( q ) , and d E ( p , q ) ≤ (cid:15) an admissible pair.
Revised: supp ( q ) , and d E ( p , q ) ≤ (cid:15) an admissible pair.

Original: Over any admissible pair ( p , q ) , E [ R ] =
Revised: Over any admissible pair ( p , q ) , E [ R ] =

Original: d E ( p , q ) .
Revised: d E ( p , q ) .

Original: We next bound the deviation
Revised: We next bound the deviation

Original: R − E [ R ] uniformly over all admissible ( p , q ) pairs.
Revised: R − E [ R ] uniformly over all admissible ( p , q ) pairs.

Original: Observe that the process R is the sample average of m indicator functions.
Revised: Observe that the process R is the sample average of m indicator functions.

Original: Deﬁne the hypothesis class
Revised: Deﬁne the hypothesis class

Original: [Equation]
Revised: [Equation]

Original: and let d vc ( H ) be the VC-dimension of H .
Revised: and let d vc ( H ) be the VC-dimension of H .

Original: From concentration results for VC-classes (see, e.g., Eq (3) and Theorem 3.4 of Boucheron et al.
Revised: From concentration results for VC-classes (see, e.g., Eq (3) and Theorem 3.4 of Boucheron et al.

Original: (2005)), we have
Revised: (2005)), we have

Original: [Equation]
Revised: [Equation]

Original: for any t > 0 .
Revised: for any t > 0 .

Original: It remains to bound the VC-dimension d vc ( H ) .
Revised: It remains to bound the VC-dimension d vc ( H ) .

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: Observe that each set in the latter hypothesis class can be written as
Revised: Observe that each set in the latter hypothesis class can be written as

Original: [Equation]
Revised: [Equation]

Revised: sign

Revised: [Equation]

Revised: = sign

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: C

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: (cid:9)

Revised: [Equation]

Revised: (cid:9)

Revised: [Equation]

Revised: (cid:8)

Revised: [Equation]

Revised: (cid:55)→

Revised: [Equation]

Revised: (cid:8)

Revised: [Equation]

Revised: (cid:62)

Revised: [Equation]

Revised: [Equation]

Revised: (cid:62)

Revised: [Equation]

Revised: p

Revised: [Equation]

Original:  the union of intersections of two halfspaces.
Revised: S n [Equation] 1 (cid:9) . (C.46) the union of intersections of two halfspaces.

Original: Thus, letting
Revised: Thus, letting

Revised: H

Original: [Equation]
Revised: [Equation]

Original:  be the class of halfspaces, we have
Revised: C.47) be the class of halfspaces, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Note that H 0 has VC-dimension n + 1 .
Revised: Note that H 0 has VC-dimension n + 1 .

Original: Applying bounds on the VC-dimension of unions and intersections (Theorem 1.1, Van Der Vaart & Wellner (2009)), we get that
Revised: Applying bounds on the VC-dimension of unions and intersections (Theorem 1.1, Van Der Vaart & Wellner (2009)), we get that

Original: [Equation]
Revised: [Equation]

Original: (C
Revised: ≤ C

Original: [Equation]
Revised: [Equation]

Revised: n.

Revised: (C.49)

Original: Plugging this bound into Eq.
Revised: Plugging this bound into Eq.

Original: (C.43), we can set t =
Revised: (C.43), we can set t =

Original: (cid:15)/ 2 and make m large enough so that
Revised: (cid:15)/ 2 and make m large enough so that

Original: C 0 √ C 3 (cid:112) n/m ≤ (cid:15)/ 2 , completing the proof.
Revised: C 0 √ C 3 (cid:112) n/m ≤ (cid:15)/ 2 , completing the proof.

Original: (cid:4)
Revised: (cid:4)

Original: C. P OINTWISE CONVERGENCE OF SUB - DIFFERENTIAL
Revised: C. P OINTWISE CONVERGENCE OF SUB - DIFFERENTIAL

Original: Proposition C.6 (Pointwise convergence) .
Revised: Proposition C.6 (Pointwise convergence) .

Original: For any ﬁxed q ∈ S n − 1 ,
Revised: For any ﬁxed q ∈ S n − 1 ,

Original: P (cid:104)
Revised: P (cid:104)

Original: d H ( ∂f ( q ) , E [ ∂f ] ( q )) >
Revised: d H ( ∂f ( q ) , E [ ∂f ] ( q )) >

Original: C a (cid:112) n/m + C b t/ √ m (cid:105) ≤ 2 exp (cid:0) − t 2 (cid:1) ∀ t > 0 .
Revised: C a (cid:112) n/m + C b t/ √ m (cid:105) ≤ 2 exp (cid:0) − t 2 (cid:1) ∀ t > 0 .

Original: (C.50)
Revised: (C.50)

Original: Here C a , C b ≥ 0 are universal constants.
Revised: Here C a , C b ≥ 0 are universal constants.

Original: Proof.
Revised: Proof.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: Write X u .
Revised: Write X u .

Original: =
Revised: =

Original: h ∂f ( q ) ( u ) − E
Revised: h ∂f ( q ) ( u ) − E

Original: h
Revised: h

Original: ∂f
Revised: ∂f

Original: ( q ) ( u ) and consider the zero-mean random process { X u } deﬁned on S n − 1 .
Revised: ( q ) ( u ) and consider the zero-mean random process { X u } deﬁned on S n − 1 .

Original: For any u , v ∈ S n − 1 , we have
Revised: For any u , v ∈ S n − 1 , we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we write Q i .
Revised: where we write Q i .

Original: = sign (cid:0) q (cid:62) x i (cid:1) x
Revised: = sign (cid:0) q (cid:62) x i (cid:1) x

Original: i for all i ∈ [ m ] .
Revised: i for all i ∈ [ m ] .

Original: Next we estimate (cid:107) h Q
Revised: Next we estimate (cid:107) h Q

Original: i ( u ) − h Q
Revised: i ( u ) − h Q

Original: i ( v ) (cid:107) ψ 2 .
Revised: i ( v ) (cid:107) ψ 2 .

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: If h Q i ( u ) − h Q i ( v ) ≥ 0 and let z ∗ .
Revised: If h Q i ( u ) − h Q i ( v ) ≥ 0 and let z ∗ .

Original: = arg max z ∈ Q
Revised: = arg max z ∈ Q

Original: i (cid:104) z , u (cid:105) , we have
Revised: i (cid:104) z , u (cid:105) , we have

Original: [Equation]
Revised: [Equation]

Original:  57) and 
Revised: [Equation] 57) and (cid:107) [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we have used Lemma F.1 to obtain the last upper bound.
Revised: where we have used Lemma F.1 to obtain the last upper bound.

Original: If h Q i ( u ) − h Q i ( v ) ≤ 0 , h Q i ( v ) − h Q
Revised: If h Q i ( u ) − h Q i ( v ) ≤ 0 , h Q i ( v ) − h Q

Original: i ( u ) ≥ 

Revised: i ( u ) ≥

Original: and we can use similar argument to conclude that
Revised: and we can use similar argument to conclude that

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: Thus, { X u } is a centered random process with sub-Gaussian increments with a parameter C 4 / √ m .
Revised: Thus, { X u } is a centered random process with sub-Gaussian increments with a parameter C 4 / √ m .

Original: We can apply Proposition A.2 to conclude that
Revised: We can apply Proposition A.2 to conclude that

Original: [Equation]
Revised: [Equation]

Original: which implies the claimed result.
Revised: which implies the claimed result.

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF P ROPOSITION 3.5 (U NIFORM CONVERGENCE )
Revised: C. P ROOF OF P ROPOSITION 3.5 (U NIFORM CONVERGENCE )

Original: Throughout the proof, we let c, C denote universal constants that could change from step to step.
Revised: Throughout the proof, we let c, C denote universal constants that could change from step to step.

Original: Fix an (cid:15) ∈ (0 , 1 / 2) to be decided later.
Revised: Fix an (cid:15) ∈ (0 , 1 / 2) to be decided later.

Original: Let N (cid:15) be an admissible (cid:15) net for S n − 1 wrt d E , with | N (cid:15) | ≤ exp( Cn log( n/(cid:15) ))
Revised: Let N (cid:15) be an admissible (cid:15) net for S n − 1 wrt d E , with | N (cid:15) | ≤ exp( Cn log( n/(cid:15) ))

Original: (Lemma C.4).
Revised: (Lemma C.4).

Original: By Proposition C.6 and the union bound,
Revised: By Proposition C.6 and the union bound,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: provided that m ≥ Ct − 2 n .
Revised: provided that m ≥ Ct − 2 n .

Original: For any p ∈ S n − 1 , let q ∈ N (cid:15) satisfy supp ( q )
Revised: For any p ∈ S n − 1 , let q ∈ N (cid:15) satisfy supp ( q )

Original: =
Revised: =

Original: supp ( p ) and d E ( p , q ) ≤ (cid:15) .
Revised: supp ( p ) and d E ( p , q ) ≤ (cid:15) .

Original: Then we have d H
Revised: Then we have d H

Original: [Equation]
Revised: [Equation]

Original: by the triangular inequality for the Hausdorff metric.
Revised: by the triangular inequality for the Hausdorff metric.

Original: By the preceding union bound, term I is bounded by t/ 3 as long as the bad event does not happen.
Revised: By the preceding union bound, term I is bounded by t/ 3 as long as the bad event does not happen.

Original: For term II, we have
Revised: For term II, we have

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last line follows from Lemma F.5.
Revised: where the last line follows from Lemma F.5.

Original: As long as (cid:15) ≤ ct/ (cid:112) log(1 /t ) , the above term is upper bounded by t/ 3 .
Revised: As long as (cid:15) ≤ ct/ (cid:112) log(1 /t ) , the above term is upper bounded by t/ 3 .

Original: For term III, we have
Revised: For term III, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: sup

Original: [Table] [Equation] [Table] [Equation]
Revised: = (cid:114) π 2 ·

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: u

Original: [Table]

Original: [Table]

Original: ) =
Revised: ) =

Original: [Equation]

Original: π

Original: [Equation]

Original: [Equation]

Original: [Table]

Original: (cid:88)

Original: [Table]
Revised: [Table]

Original: s

Original: [Table] ( s [Table] ∈ { +1 , − 1 } dependent on p , q , x
Revised: π 2 · m

Original: [Table]
Revised: [Table]

Original: and u ) (

Original: [Table]

Original: = (cid:114) π

Original: [Equation]
Revised: [Equation]

Original: m

Original: [Table]
Revised: [Table]

Original: (cid:88)

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: (C.72)
Revised: (C.72)

Original: By Lemma C.5, with probability at least 1 − exp( − c(cid:15) 2 m ) , the number of different signs is upper bounded by 2 m(cid:15) for all p , q such that d E ( p , q ) ≤ (cid:15) .
Revised: By Lemma C.5, with probability at least 1 − exp( − c(cid:15) 2 m ) , the number of different signs is upper bounded by 2 m(cid:15) for all p , q such that d E ( p , q ) ≤ (cid:15) .

Original: On this good event, the above quantity can be upper bounded as follows.
Revised: On this good event, the above quantity can be upper bounded as follows.

Original: Deﬁne a set T .
Revised: Deﬁne a set T .

Original: = { s ∈ R m : s i ∈
Revised: = { s ∈ R m : s i ∈

Original: { +1 , − 1 , 0 } , (cid:107) s (cid:107) 0 ≤ 2 m(cid:15) } and consider the quantity sup s ∈ T (cid:107) Xs (cid:107) , where X =
Revised: { +1 , − 1 , 0 } , (cid:107) s (cid:107) 0 ≤ 2 m(cid:15) } and consider the quantity sup s ∈ T (cid:107) Xs (cid:107) , where X =

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x m ] .
Revised: , x m ] .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: uniformly (i.e., indepdent of p , q and u ).
Revised: uniformly (i.e., indepdent of p , q and u ).

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original:  g∼ N ( 0 , I m )
Revised: here g

Original: [Equation]
Revised: [Equation]

Revised: N ( 0 , I m )

Original: [Equation]
Revised: [Equation]

Original: .75) rad( T ) = √ 2 m(cid:15)
Revised: .75) rad

Original: [Equation]
Revised: [Equation]

Original: C [Equation] [Equation]
Revised: C.76)

Original: Noting that 1 / √ θ · X has independent, isotropic, and sub-Gaussian rows with a parameter C/ √ θ , we apply Proposition A.3 and obtain that
Revised: Noting that 1 / √ θ · X has independent, isotropic, and sub-Gaussian rows with a parameter C/ √ θ , we apply Proposition A.3 and obtain that

Original: [Equation]
Revised: [Equation]

Original: 77) with probability at least 1 − 2 exp (cid:0) − t 2 0 (cid:1) .
Revised: 77) with probability at least 1 − 2 exp (cid:0) − t 2 0 (cid:1) .

Original: So we have over all admissible ( p , q ) pairs,
Revised: So we have over all admissible ( p , q ) pairs,

Original: [Equation]
Revised: [Equation]

Original: Setting t 

Revised: Setting t

Original: = ct √ m and (cid:15)
Revised: = ct √ m and (cid:15)

Original: =
Revised: =

Original: ct (cid:112) θ/ log m , we have that
Revised: ct (cid:112) θ/ log m , we have that

Original: [Equation]
Revised: [Equation]

Original: provided that m ≥ C(cid:15)t − 2 n = Ct − 1 n (cid:112) θ/ log m , which is subsumed by the earlier requirement m ≥ Ct − 2 n .
Revised: provided that m ≥ C(cid:15)t − 2 n = Ct − 1 n (cid:112) θ/ log m , which is subsumed by the earlier requirement m ≥ Ct − 2 n .

Original: Putting together the three bounds Eq.
Revised: Putting together the three bounds Eq.

Original: (C.62), Eq.
Revised: (C.62), Eq.

Original: (C.67), Eq.
Revised: (C.67), Eq.

Original: (C.80), we can choose
Revised: (C.80), we can choose

Original: [Equation]
Revised: [Equation]

Original: 81) and get that d H ( ∂f ( p ) , E [ ∂f ] ( p )) ≤ t with probability at least
Revised: and get that d H ( ∂f ( p ) , E [ ∂f ] ( p )) ≤ t with probability at least

Original: [Equation]
Revised: [Equation]

Original:  provided that m ≥ Cnt − 2 log n log( m/t ) θt .
Revised: 84) provided that m ≥ Cnt − 2 log n log( m/t ) θt .

Original: A sufﬁcient condition is that m ≥ Cnt − 2 log 2 ( n/t ) for sufﬁciently large C .
Revised: A sufﬁcient condition is that m ≥ Cnt − 2 log 2 ( n/t ) for sufﬁciently large C .

Original: When this is satisﬁed, the probability is further lower bounded by 1 − exp( − cmθt 2 / log m ) .
Revised: When this is satisﬁed, the probability is further lower bounded by 1 − exp( − cmθt 2 / log m ) .

Original: C. P ROOF OF T HEOREM 3. 
Revised: C. P ROOF OF T HEOREM 3.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: By Proposition 3.5, with probability at least 1 − exp (cid:0) − cmθ 3 ζ 20 n − 3 log − 1 m (cid:1) we have
Revised: By Proposition 3.5, with probability at least 1 − exp (cid:0) − cmθ 3 ζ 20 n − 3 log − 1 m (cid:1) we have

Original: [Equation]
Revised: [Equation]

Original: E [
Revised: E [

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 86) provided that m ≥ Cn 4 θ − 2 ζ − 2 0 log ( n/ζ 0 ) .
Revised: 86) provided that m ≥ Cn 4 θ − 2 ζ − 2 0 log ( n/ζ 0 ) .

Original: We now show the properties Eq.
Revised: We now show the properties Eq.

Original: (3.12) and Eq.
Revised: (3.12) and Eq.

Original: (3.13) on this good event, focusing on S ( n +) ζbut obtaining the same results for all other 2 n − 1 subsets by the same arguments.
Revised: (3.13) on this good event, focusing on S ( n +) ζbut obtaining the same results for all other 2 n − 1 subsets by the same arguments.

Original: For Eq.
Revised: For Eq.

Original: (3.12), we have
Revised: (3.12), we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now sup (cid:104) ∂f ( q ) , e n /q n − e j /q j (cid:105) =
Revised: Now sup (cid:104) ∂f ( q ) , e n /q n − e j /q j (cid:105) =

Original: h ∂f ( q ) [Equation]
Revised: h ∂f ( q ) ( e n /q n − e j /q j ) (C.88)

Original:  e 
Revised: = E h ∂f ( q ) ( e n /q n − e j /q

Original: [Equation]
Revised: [Equation]

Original:  n − e
Revised: ) − E h ∂f ( q ) ( e n /q n − e

Original: [Equation]
Revised: [Equation]

Original: C
Revised: /q j ) + h ∂f ( q ) ( e n /q n

Original: [Equation]
Revised: [Equation]

Original: 88) =

Original: E
Revised: e

Original: [Equation]
Revised: [Equation]

Original: (C
Revised: /q

Original: [Equation]
Revised: [Equation]

Original: [Equation] ≤ E h ∂f ( q 
Revised: ) (C.89) ≤ E h ∂f ( q )

Original: [Equation]
Revised: [Equation]

Original: By Theorem 3.4(a)
Revised: By Theorem 3.4(a)

Original: [Equation]
Revised: [Equation]

Original: Moreover, (cid:107) e n /q n − e j /q j (cid:107) =
Revised: Moreover, (cid:107) e n /q n − e j /q j (cid:107) =

Original: [Equation]
Revised: [Equation]

Original: 1 /q 2 n + 1 /q
Revised: 1 /q 2 n + 1 /q

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:112) 1 /q
Revised: ≤ (cid:112) 1 /q

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n .
Revised: n .

Original: Meanwhile, we have
Revised: Meanwhile, we have

Original: [Equation]
Revised: [Equation]

Original: We conclude that
Revised: We conclude that

Original: [Equation]
Revised: [Equation]

Original: as claimed.
Revised: as claimed.

Original: For Eq.
Revised: For Eq.

Original: [Equation]
Revised: [Equation]

Original: we have by Theorem 3.4(b) that
Revised: we have by Theorem 3.4(b) that

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As we are on the good event
Revised: As we are on the good event

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: ) we haveinf
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: (C.104)
Revised: (C.104)

Original: Noting that (cid:107) q − n (cid:107) ≥ 1 √ 2 (cid:107) q − e n (cid:107) for all q with q n ≥ 0 completes the proof.
Revised: Noting that (cid:107) q − n (cid:107) ≥ 1 √ 2 (cid:107) q − e n (cid:107) for all q with q n ≥ 0 completes the proof.

Original: C. P ROOF OF P ROPOSITION 3. 
Revised: C. P ROOF OF P ROPOSITION 3.

Original: For any q ∈ S n − 1 , sup (cid:107) ∂f ( q ) (cid:107) = 
Revised: For any q ∈ S n − 1 , sup (cid:107) [Equation] [Equation] = d

Revised: [Equation]

Original: d H ( { 0 } , ∂f ( q )) 
Revised: { 0 } , ∂f ( q )) ≤ d

Original: ≤
Revised: [Equation]

Original: d H ( { 0 } , E [ ∂f ] ( q )) + d H ( ∂f ( q ) , E [ ∂f ] ( q )) (C.105) by the metric property of the Hausdorff metric. On one hand, we have sup (cid:107) E
Revised: [Equation]

Original:  [ ∂f ]
Revised: E [ ∂f ]

Original: [Equation]
Revised: [Equation]

Original: q )
Revised: q )

Revised: [Equation]

Revised: ∂f

Original: [Equation]
Revised: [Equation]

Original: = sup (cid:13)(cid:13)(cid:13)(cid:13)
Revised: q [Equation] E [ ∂f ] ( q )) [Equation]

Original: E Ω
Revised: C.105) by the metric property of the Hausdorff metric. On one hand, we have sup [Equation] =

Original: [Equation]
Revised: [Equation]

Original: q Ω
Revised: Ω

Original: [Equation]
Revised: [Equation]

Original: q Ω (cid:107)
Revised: C

Original: [Equation]
Revised: [Equation]

Original: On the other hand, by Proposition 3.5,
Revised: On the other hand, by Proposition 3.5,

Original: [Equation]
Revised: [Equation]

Original: (C [Equation] 107) with probability at least 1 − exp (cid:0) − c1 mθ log − 1 m (cid:1) , provided that m ≥ C 2 n 2 log n (simpliﬁed using θ ≥ 1 /n ).
Revised: (C.107) with probability at least 1 − exp (cid:0) − c mθ log − 1 m (cid:1) , provided that m ≥ C 2 n 2 log n (simpliﬁed using θ ≥ 1 /n ).

Original: Combining the two results complete the proof.
Revised: Combining the two results complete the proof.

Original: C. A DDITIONAL GEOMETRIES ON THE EMPIRICAL OBJECTIVE
Revised: C. A DDITIONAL GEOMETRIES ON THE EMPIRICAL OBJECTIVE

Original: Proposition C.7.
Revised: Proposition C.7.

Original: On the good event in Proposition 3.7, for all q ∈ S ( n +) ζ, we have
Revised: On the good event in Proposition 3.7, for all q ∈ S ( n +) ζ, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use Lebourg’s mean value theorem for locally Lipschitz functions 6 ,
Revised: We use Lebourg’s mean value theorem for locally Lipschitz functions 5 ,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., Theorem 2.3. of (Clarke, 1990).
Revised: e., Theorem 2.3. of (Clarke, 1990).

Original: It is convenient to work in the w space here.
Revised: It is convenient to work in the w space here.

Original: By subdifferential chain rules, g ( w ) is locally Lipschitz over (cid:110) w : (cid:107) w (cid:107) <
Revised: By subdifferential chain rules, g ( w ) is locally Lipschitz over (cid:110) w : (cid:107) w (cid:107) <

Original: (cid:113) n − 1 n (cid:111) .
Revised: (cid:113) n − 1 n (cid:111) .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: for a certain t 0 ∈ (0 , 1) and a certain v ∈ ∂g
Revised: for a certain t 0 ∈ (0 , 1) and a certain v ∈ ∂g

Original: ( t 0 w ) .
Revised: ( t 0 w ) .

Original: Now for any q and the corresponding w ,
Revised: Now for any q and the corresponding w ,

Original: [Equation]
Revised: [Equation]

Original: It follows
Revised: It follows

Original: [Equation]
Revised: [Equation]

Original: where at the last inequality we have used Proposition 3.7.
Revised: where at the last inequality we have used Proposition 3.7.

Original: Continuing the calculation, we further have
Revised: Continuing the calculation, we further have

Original: [Equation]
Revised: [Equation]

Original: completing the proof.
Revised: completing the proof.

Original: [Equation]
Revised: [Equation]

Original: Proposition C.8.
Revised: Proposition C.8.

Original: Assume θ ∈
Revised: Assume θ ∈

Original: [1 /n, 1 / 2] .
Revised: [1 /n, 1 / 2] .

Original: When m ≥ Cθ − 2 n log n , with probability at least 1 − exp (cid:0) − cmθ 3 log − 1 m (cid:1) , the following holds: for all q ∈ S ( n +)
Revised: When m ≥ Cθ − 2 n log n , with probability at least 1 − exp (cid:0) − cmθ 3 log − 1 m (cid:1) , the following holds: for all q ∈ S ( n +)

Original: ζ
Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: Here C, c > 0 are universal constants.
Revised: Here C, c > 0 are universal constants.

Original: Proof.
Revised: Proof.

Original: We ﬁrst establish uniform convergence of f ( p ) to E [ f ] ( p ) .
Revised: We ﬁrst establish uniform convergence of f ( p ) to E [ f ] ( p ) .

Original: Consider the zero-centered random process X p .
Revised: Consider the zero-centered random process X p .

Original: =
Revised: =

Original: f ( p ) − E [ f ] ( p ) on S n − 1 .
Revised: f ( p ) − E [ f ] ( p ) on S n − 1 .

Original: Similar to proof of Proposition C.6, we can show that for all p , q ∈ S n −
Revised: Similar to proof of Proposition C.6, we can show that for all p , q ∈ S n −

Original: [Equation]
Revised: [Equation]

Original: C [Equation]
Revised: (C.114)

Original: Applying Proposition A.2 gives that
Revised: Applying Proposition A.2 gives that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − exp ( − cm ) , provided that m ≥ Cn .
Revised: with probability at least 1 − exp (cid:0) − cmθ 2 (cid:1) , provided that m ≥ Cθ − 2 n .

Original: Now we consider E [ f ] ( q ) − E [ f ] ( e n ) .
Revised: Now we consider E [ f ] ( q ) − E [ f ] ( e n ) .

Original: For convenience, we ﬁrst work in the w space and note that E [ f ] ( q ) − E [ f ] ( e n ) =
Revised: For convenience, we ﬁrst work in the w space and note that E [ f ] ( q ) − E [ f ] ( e n ) =

Original: E [ g ] ( w ( q )) − E [ g ] ( 0 ) .
Revised: E [ g ] ( w ( q )) − E [ g ] ( 0 ) .

Original: By Lemma B.3, E [ g ] is monotonically increasing in every radial direction v until (cid:107) w (cid:107) 2 + (cid:107) w (cid:107) 2 ∞ ≤ 1 , which implies that
Revised: By Lemma B.3, E [ g ] is monotonically increasing in every radial direction v until (cid:107) w (cid:107) 2 + (cid:107) w (cid:107) 2 ∞ ≤ 1 , which implies that

Original: [Equation]
Revised: [Equation]

Original: For w with (cid:107) w (cid:107) = 1 / 2 ,
Revised: For w with (cid:107) w (cid:107) = 1 / 2 ,

Original: [Equation]
Revised: [Equation]

Original: ( using θ ≤1 / 2 and (cid:107) w (cid:107) = 1 / 2) (
Revised: using θ ≤

Original: [Equation]
Revised: [Equation]

Revised: 2 and (cid:107)

Original: [Equation]
Revised: [Equation]

Original: So, back to the q space,
Revised: So, back to the q space,

Original: [Equation]
Revised: [Equation]

Original: Combining the results in Eq.
Revised: Combining the results in Eq.

Original: (C.115) and Eq.
Revised: (C.115) and Eq.

Original: (C.121), we conclude that with high probability
Revised: (C.121) [Equation] we conclude that with high probability

Original: [Equation]
Revised: [Equation]

Original: So when f ( q ) − f ( 0 ) ≤ 3 / 100 , (cid:107) q − n (cid:107) ≤ 1 / 2 , which is equivalent to (cid:107) w (cid:107) ≤ 1 / 2 in the w space.
Revised: So when f ( q ) − f ( 0 ) ≤ 2 / 25 · θ , (cid:107) q − n (cid:107) ≤ 1 / 2 , which is equivalent to (cid:107) w (cid:107) ≤ 1 / 2 in the w space.

Original: Under this constraint, by Lemma B.3,
Revised: Under this constraint, by Lemma B.3,

Original: [Equation]
Revised: [Equation]

Original: So, emulating the proof of Eq.
Revised: So, emulating the proof of Eq.

Original: (3.9) in Theorem 3.4, we have that for q ∈ S ( n +) ζwith (cid:107) q − n [Equation]
Revised: (3.9) in Theorem 3.4, we have that for q ∈ S ( n +) ζwith (cid:107) q − n (cid:107) ≤ 1 / 2 ,

Original: [Equation]
Revised: [Equation]

Original: where at the last inequality we use q n = (cid:113) 1 − (cid:107) w (cid:107) 2 ≥ √ 3 / 2 when (cid:107) w (cid:107) ≤ 1 / 2 .
Revised: where at the last inequality we use q n = (cid:113) 1 − (cid:107) w (cid:107) 2 ≥ √ 3 / 2 when (cid:107) w (cid:107) ≤ 1 / 2 .

Original: Moreover, we emulate the proof of Eq.
Revised: Moreover, we emulate the proof of Eq.

Original: (3.13) in Theorem 3.6 to obtain that
Revised: (3.13) in Theorem 3.6 to obtain that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − exp (cid:0) − cmθ 3 log − 1 m (cid:1) , provided that m ≥ Cθ − 2 n log n .
Revised: with probability at least 1 − exp (cid:0) − cmθ 3 log − 1 m (cid:1) , provided that m ≥ Cθ − 2 n log n .

Original: The last step of our proof is invoking the mean value theorem, similar to the proof of Proposition C.7.
Revised: The last step of our proof is invoking the mean value theorem, similar to the proof of Proposition C.7.

Original: For any q , we have
Revised: For any q , we have

Original: [Equation]
Revised: [Equation]

Original: C [Equation] 127) for a certain t ∈ (0 , 1) and a certain v ∈ ∂g ( t w ) .
Revised: for a certain t ∈ (0 , 1) and a certain v ∈ ∂g ( t w ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: completing the proof.
Revised: completing the proof

Original: [Equation]
Revised: [Equation]

Original: D P ROOFS FOR S ECTION 3. D. S TAYING IN THE REGION S ( n +)ζ
Revised: D P ROOFS FOR S ECTION 3. D. S TAYING IN THE REGION S ( n +)

Revised: [Table]

Revised: [Equation]

Original: Lemma D.1 (Progress in S ( n +)
Revised: Lemma D.1 (Progress in S ( n +)

Original: [Equation]
Revised: [Equation]

Original: \ S ( n +)) .
Revised: \ S ( n +)) .

Original: Set
Revised: Set

Original: η = t 0 / (100 √ n ) for t 0 ∈ (0 , 1) .
Revised: η = t 0 / (100 √ n ) for t 0 ∈ (0 , 1) .

Original: For any ζ 0 ∈ (0 , 1) , on the good events stated in Proposition 3.7 and Theorem 3.6, we have for all q ∈ S ( n +) ζ\ S ( n +)and q + being the next step of Riemannian subgradient descent that 
Revised: For any ζ 0 ∈ (0 , 1) , on the good events stated in Proposition 3.7 and Theorem 3.6, we have for all q ∈ S ( n +) ζ\ S ( n +)and q + being the next step of Riemannian subgradient descent that [Equation] n

Original: [Equation]
Revised: [Equation]

Original: In particular, we have q + ∈ S ( n +) ζ.
Revised: In particular, we have q + ∈ S ( n +) ζ.

Original: Proof.
Revised: Proof.

Original: We divide the index set [ n − 1] into three sets
Revised: We divide the index set [ n − 1] into three sets

Original: [Equation]
Revised: [Equation]

Original: We perform different arguments on different sets.
Revised: We perform different arguments on different sets.

Original: We let g ( q ) ∈ ∂
Revised: We let g ( q ) ∈ ∂

Original: R f ( q ) be the subgradient taken at q and note by Proposition 3.

Revised: R f ( q ) be the subgradient taken at q and note by Proposition 3.

Original: that (cid:107) g
Revised: that (cid:107) g

Original: (cid:107) ≤ 2 , and so | g i | ≤ 2 for all i ∈ [ n ] .
Revised: (cid:107) ≤ 2 , and so | g i | ≤ 2 for all i ∈ [ n ] .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: For any j ∈ I 0 ,
Revised: For any j ∈ I 0 ,

Original: [Equation]
Revised: [Equation]

Original: ,n
Revised: ,n

Original: [Equation]
Revised: [Equation]

Original: Provided that η ≤ 1 / (4 √ n ) , 1 − 2 η √ n ≥ 1 / 2 , and so(
Revised: Provided that η ≤ 1 / (4 √ n ) , 1 − 2 η √ n ≥ 1 / 2 , and so

Original: [Equation]
Revised: [Equation]

Original: (D.7) where the last inequality holds when η ≤ 1 / √ 40 n .
Revised: where the last inequality holds when η ≤ 1 / √ 40 n .

Original: For any j ∈ I 1 ,
Revised: For any j ∈ I 1 ,

Original: [Equation]
Revised: [Equation]

Original: 8) where the very last inequality holds when η ≤ 1 / (26 √ n ) .
Revised: where the very last inequality holds when η ≤ 1 / (26 √ n ) .

Original: Since q ∈ S ( n +) ζ\ S ( n +), I 2 is nonempty.
Revised: Since q ∈ S ( n +) ζ\ S ( n +), I 2 is nonempty.

Original: For any j ∈ I 2 ,q
Revised: For any j ∈ I 2 ,

Original: [Equation]
Revised: [Equation]

Original: (cid:19)

Original: [Equation]

Original: (D.9)
Revised: (D.9)

Original: Since g j /q j ≤ 2 √ 3 n , 1 − ηg j /q j ≥ 1 / 2 when η ≤ 1 / (cid:0) 4 √ 3 n (cid:1) .
Revised: Since g j /q j ≤ 2 √ 3 n , 1 − ηg j /q j ≥ 1 / 2 when η ≤ 1 / (cid:0) 4 √ 3 n (cid:1) .

Original: Conditioned on this and due to that g j /q j − g n /q n ≥ 0 , it follows
Revised: Conditioned on this and due to that g j /q j − g n /q n ≥ 0 , it follows

Original: [Equation]
Revised: [Equation]

Original: D.
Revised: D.

Original: [Equation]
Revised: [Equation]

Original: If q 2 n /q 2 j ≤ 2 , q 2+ ,n /q 2+ ,j ≤ 5 / 2 provided that
Revised: If q 2 n /q 2 j ≤ 2 , q 2+ ,n /q 2+ ,j ≤ 5 / 2 provided that

Original: [Equation]
Revised: [Equation]

Original: As q / ∈ S ( n +), we have q 2 n / (cid:107) q − n (cid:107) 2 ∞ ≤ 2 , so there must be a certain j ∈ I 2 satisfying q 2 n /q 2 j ≤ 2 .
Revised: As q / ∈ S ( n +), we have q 2 n / (cid:107) q − n (cid:107) 2 ∞ ≤ 2 , so there must be a certain j ∈ I 2 satisfying q 2 n /q 2 j ≤ 2 .

Original: We conclude that when
Revised: We conclude that when

Original: [Equation]
Revised: [Equation]

Original: the index of largest entries of q + , − n remains in I 2 .
Revised: the index of largest entries of q + , − n remains in I 2 .

Original: On the other hand, when η ≤ 1 / (100 √ n ) , for all j ∈ I 2 ,
Revised: On the other hand, when η ≤ 1 / (100 √ n ) , for all j ∈ I 2 ,

Original: [Equation]
Revised: [Equation]

Original: So when η = t/ (100 √ n ) for any t ∈ (0 , 1) ,
Revised: So when η = t/ (100 √ n ) for any t ∈ (0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: completing the proof.
Revised: completing the proof.

Original: [Equation]
Revised: [Equation]

Original: Proposition D.2.
Revised: Proposition D.2.

Original: For any ζ 0 ∈ (0 , 1) , on the good events stated in Proposition 3.7 and Theorem 3.6, if the step sizes satisfy
Revised: For any ζ 0 ∈ (0 , 1) , on the good events stated in Proposition 3.7 and Theorem 3.6, if the step sizes satisfy

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:27) for all k, (D.15) the iteration sequence will stay in S ( n +) ζprovided that our initialization q (0) ∈ S ( n +) ζ.
Revised: (cid:27) for all k, (D.15) the iteration sequence will stay in S ( n +) ζprovided that our initialization q (0) ∈ S ( n +) ζ.

Original: Proof.
Revised: Proof.

Original: By Lemma D.1, if the current iterate q ∈ S ( n +) ζ\ S ( n +), the next iterate q + ∈ S ( n +) ζ, provided that η ≤ 1 / (100 √ n ) .
Revised: By Lemma D.1, if the current iterate q ∈ S ( n +) ζ\ S ( n +), the next iterate q + ∈ S ( n +) ζ, provided that η ≤ 1 / (100 √ n ) .

Original: Now if the current q ∈ S ( n +), i.e., q 2 n /q 2 j ≥ 2 for all j ∈ [ n − 1] , we can emulate the analysis of the set I 1 in proof of Lemma D.1.
Revised: Now if the current q ∈ S ( n +), i.e., q 2 n /q 2 j ≥ 2 for all j ∈ [ n − 1] , we can emulate the analysis of the set I 1 in proof of Lemma D.1.

Original: Indeed, for any j ∈ [ n − 1] , q
Revised: Indeed, for any j ∈ [ n − 1] , q

Original: [Equation]
Revised: [Equation]

Original: n [Equation] (D.16) where the last inequality holds provided that η ≤ (1 − ζ 0 ) / (9 √ n ) .
Revised: (D.16) where the last inequality holds provided that η ≤ (1 − ζ 0 ) / (9 √ n ) .

Original: Combining the two cases ﬁnishes the proof.
Revised: Combining the two cases ﬁnishes the proof.

Original: (cid:4)
Revised: (cid:4)

Original: D. P ROOF OF T HEOREM 3. 
Revised: D. P ROOF OF T HEOREM 3.

Original: As we have η ( k ) ≤[-
-]
Revised: As we have η ( k ) ≤[-
-]

Original: [Equation]
Revised: [Equation]

Original: n and q (0) ∈ S ( n +) ζ, the entire sequence (cid:8) q ( k ) (cid:9) k ≥ 0 will stay in S ( n +) ζby Proposition D.2.
Revised: and [Equation] , the entire sequence (cid:8) q ( k ) (cid:9) k ≥ 0 will stay in S ( n +) ζby Proposition D.2.

Original: For any q and any v ∈ ∂ R f ( q ) , we have (cid:104) v , q (cid:105) = 0 and therefore (cid:107) q − η v (cid:107) 2 =
Revised: For any q and any v ∈ ∂ R f ( q ) , we have (cid:104) v , q (cid:105) = 0 and therefore (cid:107) q − η v (cid:107) 2 =

Original: (cid:107) q (cid:107) 2 + η 2 (cid:107) v (cid:107) 2 ≥ 1 .
Revised: (cid:107) q (cid:107) 2 + η 2 (cid:107) v (cid:107) 2 ≥ 1 .

Original: (D.17)
Revised: (D.17)

Original: So q − η v is not inside B n .
Revised: So q − η v is not inside B n .

Original: Since projection onto B n is a contraction
Revised: Since projection onto B n is a contraction

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where we have used the bounds in Proposition 3.7 and Theorem 3.6 to obtain the last inequality.
Revised: where we have used the bounds in Proposition 3.7 and Theorem 3.6 to obtain the last inequality.

Original: Further applying Proposition C.
Revised: Further applying Proposition C.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Summing up the inequalities until step K (assumed ≥ 5 ), we have
Revised: Summing up the inequalities until step K (assumed ≥ 5 ), we have

Original: [Equation]
Revised: [Equation]

Original: Substituting the following estimates
Revised: Substituting the following estimates

Original: [Equation]
Revised: [Equation]

Original:  and noting 16 (cid:13)(cid:13) q ( K ) − e n (cid:13)(cid:13) 2 ≤ 32 , we have 
Revised: (D.24) and noting 16 (cid:13)(cid:13) q ( K ) − e n (cid:13)(cid:13) 2 ≤ 32 , we have f (cid:0) [Equation] best

Original: [Equation]
Revised: [Equation]

Revised: n

Original: [Equation]
Revised: [Equation]

Revised: (D.25)

Original: Noting that
Revised: Noting that

Revised: K ≥

Original: [Equation]
Revised: [Equation]

Original: and when K ≥ 1 , K 1 − 2 α ≥
Revised: (cid:15)

Original: [Equation]
Revised: [Equation]

Original:  yielding that
Revised: (D.26) and when K ≥ 1 , K 1 − 2 α ≥ 1 , yielding that

Original: K ≥ 
Revised: K ≥ (cid:32)

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: So we conclude that when
Revised: So we conclude that when

Original: [Equation]
Revised: [Equation]

Revised: f

Original: f (cid:0) q best (cid:1) − f ( e n ) ≤ (cid:15) .
Revised: [Equation] q best [Equation] − f ( e n ) ≤ (cid:15) .

Original: When this happens, by Proposition C.8,
Revised: When this happens, by Proposition C.8,

Original: [Equation]
Revised: [Equation]

Original: Plugging in the choice ζ 0 = 1 / (5 log n ) in Eq.
Revised: Plugging in the choice ζ 0 = 1 / (5 log n ) in Eq.

Original: (D.28) gives the desired bound on the number of iterations.
Revised: (D.28) gives the desired bound on the number of iterations.

Original: E P ROOFS FOR S ECTION 3. E. P ROOF OF L EMMA 3. 
Revised: E P ROOFS FOR S ECTION 3. E. P ROOF OF L EMMA 3.

Original: Lemma E.1.
Revised: Lemma E.1.

Original: For all n ≥ 3 and ζ ≥ 0 , it holds that
Revised: For all n ≥ 3 and ζ ≥ 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: We note that a similar result appears in (Gilboa et al., 2018) but our deﬁnitions of the region S ζ are slightly different.
Revised: We note that a similar result appears in (Gilboa et al., 2018) but our deﬁnitions of the region S ζ are slightly different.

Original: For completeness we provide a proof in Lemma F.3.
Revised: For completeness we provide a proof in Lemma F.3.

Original: We now prove Lemma 3.9.
Revised: We now prove Lemma 3.9.

Original: Taking ζ = 1 / (5 log n ) in Lemma E.1, we obtain
Revised: Taking ζ = 1 / (5 log n ) in Lemma E.1, we obtain

Original: [Equation]
Revised: [Equation]

Original: By symmetry, all the 2 n sets (cid:110) S
Revised: By symmetry, all the 2 n sets (cid:110) S

Original: ( i +) 1 / (5log n ) , S ( i − )
Revised: ( i +) 1 / (5log n ) , S ( i − )

Original: 1 / (5log n ) : i ∈ [ n ] (cid:111) have the same volume which is at least 1 / (4 n ) .
Revised: 1 / (5log n ) : i ∈ [ n ] (cid:111) have the same volume which is at least 1 / (4 n ) .

Original: As q (0) ∼ Uniform( S n − 1 ) , it falls into their union with probability at least 2 n · 1 / (4 n ) = 1 / 2 , on which it belongs to a uniformly random one of these 2 n sets.
Revised: As q (0) ∼ Uniform( S n − 1 ) , it falls into their union with probability at least 2 n · 1 / (4 n ) = 1 / 2 , on which it belongs to a uniformly random one of these 2 n sets.

Original: E. P ROOF OF T HEOREM 3. 
Revised: E. P ROOF OF T HEOREM 3.

Original: Assume that the good event in Proposition 3.7 happens and that in Theorem 3.6 happens to all the 2 n sets (cid:110) S ( i +) 1 / (5log n ) , S ( i − ) 1 / (5log n ) : i ∈ [ n ] (cid:111) , which by setting ζ 0 = 1 / (5 log n ) has probability at least 1 − exp( − cmθ 3 ζ 20 n [Equation] 3 log [Equation]
Revised: Assume that the good event in Proposition 3.7 happens and that in Theorem 3.6 happens to all the 2 n sets (cid:110) S ( i +) 1 / (5log n ) , S ( i − ) 1 / (5log n ) : i ∈ [ n ] (cid:111) , which by setting ζ 0 = 1 / (5 log n ) has probability at least 1 − exp

Original: 1 m )

Original: [Equation]

Original: ( −

Original: [Equation]

Original: ) = 1 − exp( − c

Original: [Equation]

Original: n

Original: [Equation]
Revised: [Equation]

Original: log m

Original: [Equation]
Revised: [Equation]

Original: (E.3)
Revised: (E.3)

Original: By Lemma 3.9, random initialization will fall these 2 n sets with probability at least 1 / 2 .
Revised: By Lemma 3.9, random initialization will fall these 2 n sets with probability at least 1 / 2 .

Original: When it falls in one of these 2 n sets, by Theorem 3.8, one run of the algorithm will ﬁnd a signed standard basis vector up to (cid:15) accuracy.
Revised: When it falls in one of these 2 n sets, by Theorem 3.8, one run of the algorithm will ﬁnd a signed standard basis vector up to (cid:15) accuracy.

Original: With R independent runs, at least S .
Revised: With R independent runs, at least S .

Original: = 14 R of them are effective with probability at least 1 − exp (cid:0) − ( R/ 4) 2 / ( R/ 4 · 2) (cid:1)
Revised: = 14 R of them are effective with probability at least 1 − exp (cid:0) − ( R/ 4) 2 / ( R/ 4 · 2) (cid:1)

Original: = 1 − exp ( − R/ 8) , due to Bernstein’s inequality.
Revised: = 1 − exp ( − R/ 8) , due to Bernstein’s inequality.

Original: After these effective runs, the probability any standard basis vector is missed (up to sign) is bounded by
Revised: After these effective runs, the probability any standard basis vector is missed (up to sign) is bounded by

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds whenever S ≥ 2 n log n .
Revised: where the second inequality holds whenever S ≥ 2 n log n .

Original: F A UXILIARY CALCULATIONS
Revised: F A UXILIARY CALCULATIONS

Original: Lemma F.1.
Revised: Lemma F.1.

Original: For x ∼ BG( θ ) , (cid:107) x (cid:107) ψ 2 ≤ C a .
Revised: For x ∼ BG( θ ) , (cid:107) x (cid:107) ψ 2 ≤ C a .

Original: For any vector u ∈ R n and x ∼ iid BG( θ ) , (cid:13)(cid:13) x (cid:62) u (cid:13)(cid:13) ψ 2 ≤ C b (cid:107) u (cid:107) .
Revised: For any vector u ∈ R n and x ∼ iid BG( θ ) , (cid:13)(cid:13) x (cid:62) u (cid:13)(cid:13) ψ 2 ≤ C b (cid:107) u (cid:107) .

Original: Here C a , C b ≥ 0 are universal constants.
Revised: Here C a , C b ≥ 0 are universal constants.

Original: Proof.
Revised: Proof.

Original: For any λ ∈ R ,
Revised: For any λ ∈ R ,

Original: [Equation]
Revised: [Equation]

Original: So (cid:107) x (cid:107) ψ 2 is bounded by a universal constant.
Revised: So (cid:107) x (cid:107) ψ 2 is bounded by a universal constant.

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: , (F.2) as claimed.
Revised: , (F.2) as claimed.

Original: [Equation]
Revised: [Equation]

Original: Lemma F.2.
Revised: Lemma F.2.

Original: Let a 1 , . .
Revised: Let a 1 , . .

Original: , a m be iid copies of a ∼ iid BG( θ ) .
Revised: , a m be iid copies of a ∼ iid BG( θ ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: for any t ≥ 0 .
Revised: for any t ≥ 0 .

Original: Here C a , C b ≥ 0 are universal constants.
Revised: Here C a , C b ≥ 0 are universal constants.

Original: Proof.
Revised: Proof.

Original: Consider the zero-centered random process deﬁned on S n − 1 :
Revised: Consider the zero-centered random process deﬁned on S n − 1 :

Original: X q . =
Revised: X q . =

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Then, for any p , q ∈ S n − 1 ,
Revised: Then, for any p , q ∈ S n − 1 ,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: (F.
Revised: (F.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where we use the estimate in Lemma F.1 to obtain the last inequality.
Revised: where we use the estimate in Lemma F.1 to obtain the last inequality.

Original: Note that X q is a mean-zero random process, and we can invoke Proposition A.2 with w ( S n − 1 )
Revised: Note that X q is a mean-zero random process, and we can invoke Proposition A.2 with w ( S n − 1 )

Original: =
Revised: =

Original: C 4 √ n and rad (cid:0) S n − 1 (cid:1) =
Revised: C 4 √ n and rad (cid:0) S n − 1 (cid:1) =

Original: 2 to get the claimed result.
Revised: 2 to get the claimed result.

Original: [Equation]
Revised: [Equation]

Original: Lemma F.3.
Revised: Lemma F.3.

Original: For all n ≥ 3 and ζ ≥ 0 , it holds that
Revised: For all n ≥ 3 and ζ ≥ 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where we write ψ ( t ) .
Revised: where we write ψ ( t ) .

Original: = 1 √ 2 π (cid:82) t − t exp (cid:0) − s 2 / 2 (cid:1) ds .
Revised: = 1 √ 2 π (cid:82) t − t exp (cid:0) − s 2 / 2 (cid:1) ds .

Original: Now we derive a lower bound of the volume ratio by considering a ﬁrst-order Taylor expansion of the last equation around ζ = 

Revised: Now we derive a lower bound of the volume ratio by considering a ﬁrst-order Taylor expansion of the last equation around ζ =

Original: (as we are mostly interested in small ζ ).
Revised: (as we are mostly interested in small ζ ).

Original: By symmetry, ¯ h (0) = 1 / (2 n ) .
Revised: By symmetry, ¯ h (0) = 1 / (2 n ) .

Original: Moreover, we have
Revised: Moreover, we have

Original: [Equation]
Revised: [Equation]

Original: Now we provide an upper bound for the second term of the last equation.
Revised: Now we provide an upper bound for the second term of the last equation.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Now for any λ ∈ (0 , 1 / 2) , exp
Revised: Now for any λ ∈ (0 , 1 / 2) , exp

Original: [Equation]
Revised: [Equation]

Original: Taking logarithm on both sides, rearranging the terms
Revised: Taking logarithm on both sides, rearranging the terms

Original: [Equation]
Revised: [Equation]

Original: and setting λ = 1 / 4 , we obtain
Revised: and setting λ = 1 / 4 , we obtain

Original: [Equation]
Revised: [Equation]

Original: provided that n ≥ 3 .
Revised: provided that n ≥ 3 .

Original: Now we show that ¯ h ( ζ ) ≥ ¯ h (0) + ¯ h (cid:48) (0) ζ by showing that ¯ h (cid:48)(cid:48) ( ζ ) ≥ 0 .
Revised: Now we show that ¯ h ( ζ ) ≥ ¯ h (0) + ¯ h (cid:48) (0) ζ by showing that ¯ h (cid:48)(cid:48) ( ζ ) ≥ 0 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Using integration by part
Revised: Using integration by part

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: ∞
Revised: ∞

Original: [Equation]
Revised: [Equation]

Original: and similarly (cid:90) ∞
Revised: and similarly (cid:90) ∞

Original: [Equation]
Revised: [Equation]

Original: Noting that
Revised: Noting that

Original: [Equation]
Revised: [Equation]

Original: and combining the above integral results [Equation] we conclude that ¯ h 
Revised: and combining the above integral results, we conclude that ¯ h (cid:48)(cid:48)

Original: [Equation]{+( ζ ) ≥ 
+}
Revised: {+( ζ ) ≥ 
+}

Original: ≥ 0 and complete the proof.
Revised: and complete the proof.

Original: (cid:4)
Revised: (cid:4)

Original: Lemma F.4.
Revised: Lemma F.4.

Original: Let ( x 1 , y 1 ) , ( x 2 , y 2 ) ∈
Revised: Let ( x 1 , y 1 ) , ( x 2 , y 2 ) ∈

Original: R 2 > 0 be two points in the ﬁrst quadrant satisfying y 1 ≥ x 1 and y 2 ≥ x 2 , and y 2 /x 2 y 1 /x 1 ∈ [1 , 1 + η ] for some η ≤ 1 , then we have (cid:54) (( x 1 , y 1 ) , ( x 2 , y 2 ))
Revised: R 2 > 0 be two points in the ﬁrst quadrant satisfying y 1 ≥ x 1 and y 2 ≥ x 2 , and y 2 /x 2 y 1 /x 1 ∈ [1 , 1 + η ] for some η ≤ 1 , then we have (cid:54) (( x 1 , y 1 ) , ( x 2 , y 2 ))

Original: ≤ η .
Revised: ≤ η .

Original: Proof.
Revised: Proof.

Original: For i = 1 , 2 , let θ i be the angle between the ray ( x i , y i ) and the x -axis.
Revised: For i = 1 , 2 , let θ i be the angle between the ray ( x i , y i ) and the x -axis.

Original: Our assumption implies that θ i ∈
Revised: Our assumption implies that θ i ∈

Original: [ π/ 4 , π/ 2) and θ 2 ≥ θ 1 , thus (cid:54) (( x 1 , y 1 ) , ( x 2 , y 2 ))
Revised: [ π/ 4 , π/ 2) and θ 2 ≥ θ 1 , thus (cid:54) (( x 1 , y 1 ) , ( x 2 , y 2 ))

Original: = θ 2 − θ 1 , so we have (F.30) tan 
Revised: = θ 2 − θ 1 , so we have (F.30) tan (cid:54) (( x 1 , [Equation] 1 ) , ( x 2 , y 2 )

Original: [Equation]
Revised: [Equation]

Original: Therefore (cid:54) (( x 1 , y 1 ) , ( x 2 , y 2 )) ≤ arctan( η ) ≤ η .
Revised: Therefore (cid:54) (( x 1 , y 1 ) , ( x 2 , y 2 )) ≤ arctan( η ) ≤ η .

Original: (cid:4)
Revised: (cid:4)

Original: Lemma F.5.
Revised: Lemma F.5.

Original: For any p , q ∈ S n − 1 with the same support pattern such that d E ( p , q ) ≤ (cid:15) ≤ 12 , we have for all u ∈ S n − 1 that
Revised: For any p , q ∈ S n − 1 with the same support pattern such that d E ( p , q ) ≤ (cid:15) ≤ 12 , we have for all u ∈ S n − 1 that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix some threshold t > 0 to be determined.
Revised: Fix some threshold t > 0 to be determined.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: The second to last inequality uses Cauchy-Schwarz, and the last inequality uses the fact that u (cid:62) x
Revised: The second to last inequality uses Cauchy-Schwarz, and the last inequality uses the fact that u (cid:62) x

Original: = u (cid:62) Ω x Ω is (cid:107) u Ω (cid:107) 

Revised: = u (cid:62) Ω x Ω is (cid:107) u Ω (cid:107)

Original: -sub-Gaussian conditioned on Ω and thus 1-sub-Gaussian marginally.
Revised: -sub-Gaussian conditioned on Ω and thus 1-sub-Gaussian marginally.

Original: Taking t = (cid:113) 2 log 1 (cid:15) 2 , the above bound simpliﬁes to(cid:115)
Revised: Taking t = (cid:113) 2 log 1 (cid:15) 2 , the above bound simpliﬁes to

Original: [Equation]
Revised: [Equation]

Original: where we have used θ ≤ 1 / 2 and (cid:15) ≤ 1 / 2 .
Revised: where we have used θ ≤ 1 / 2 and (cid:15) ≤ 1 / 2 .

Original: (cid:4)
Revised: [Equation]

Original: G R ESULTS ON ORTHOGONAL DICTIONARIES
Revised: G R ESULTS ON ORTHOGONAL DICTIONARIES

Original: [Figure]
Revised: [Figure]

Revised: H F ASTER ALTERNATIVE ALGORITHM FOR LARGE - SCALE INSTANCES

Revised: The Riemannian subgradient descent is cheap per iteration but slow in overall convergence, similar to many other ﬁrst-order methods.

Revised: We also test a faster quasi-Newton type method, GRANSO,that employs BFGS for solving constrained nonsmooth problems based on sequential quadratic optimization (Curtis et al., 2017).

Revised: For a large dictionary of dimension n = 400 and sample complexity m = 10 n 2 (i.e., 1 . 6 × 10 6 ), GRANSO successfully identiﬁes a basis after 1500 iterations with CPU time 4 hours on a two-socket Intel Xeon E5-2640v4 processor (10-core Broadwell, 2.40 GHz)—this is approximately 10 × faster than the Riemannian subgradient descent method, showing the potential of quasi-Newton type methods for solving large-scale problems.

Original: H E XPERIMENT WITH IMAGES
Revised: I E XPERIMENT WITH IMAGES

Original: To experiment with images, we follow a typical setup for dictionary learning as used in image processing (Mairal et al., 2014).
Revised: To experiment with images, we follow a typical setup for dictionary learning as used in image processing (Mairal et al., 2014).

Original: We focus on testing if complete (i.e., square and invertible) dictionaries are reasonable sparsiﬁcation bases for real images, instead on any particular image processing or vision tasks. 
Revised: We focus on testing if complete (i.e., square and invertible) dictionaries are reasonable sparsiﬁcation bases for real images, instead on any particular image processing or vision tasks. [Figure]

Original: Setup Two natural images are picked for this experiment, as shown in the ﬁrst row of Fig.
Revised: Setup Two natural images are picked for this experiment, as shown in the ﬁrst row of Fig.

Original: 3, each of resolution 512 × 512 .
Revised: 3, each of resolution 512 × 512 .

Original: Each image is divided into 8 × 8 non-overlapping blocks, resulting in 64 × 64 = 4096 blocks.
Revised: Each image is divided into 8 × 8 non-overlapping blocks, resulting in 64 × 64 = 4096 blocks.

Original: The blocks are then vectorized, and stacked columnwise into a data matrix
Revised: The blocks are then vectorized, and stacked columnwise into a data matrix

Original: Y ∈ R 64 × 4096 .
Revised: Y ∈ R 64 × 4096 .

Original: We precondition the data to obtain
Revised: We precondition the data to obtain

Original: [Equation]
Revised: [Equation]

Original: so that nonvanishing singular values of Y are identically one.
Revised: so that nonvanishing singular values of Y are identically one.

Original: We then solve formulation (1.1) round (5 n log n ) times with n = 64 using the BFGS solver based on GRANSO, obtaining round (5 n log n ) vectors.
Revised: We then solve formulation (1.1) round (5 n log n ) times with n = 64 using the BFGS solver based on GRANSO, obtaining round (5 n log n ) vectors.

Original: Negative equivalent copies are pruned and vectors with large correlations with other remaining vectors are sequentially removed until only 64 vectors are left.
Revised: Negative equivalent copies are pruned and vectors with large correlations with other remaining vectors are sequentially removed until only 64 vectors are left.

Original: This forms the ﬁnal complete dictionary.
Revised: This forms the ﬁnal complete dictionary.

Original: [Figure] Results The learned complete dictionaries for the two test images are displayed in the second row of Fig.
Revised: Results The learned complete dictionaries for the two test images are displayed in the second row of Fig.

Original: Visually, the dictionaries seem reasonably adaptive to the image contents: for the left image with prevalent sharp edges, the learned dictionary consists of almost exclusively oriented sharp corners and edges, while for the right image with blurred textures and occasional sharp features, the learned dictionary does seem to be composed of the two kinds of elements.
Revised: Visually, the dictionaries seem reasonably adaptive to the image contents: for the left image with prevalent sharp edges, the learned dictionary consists of almost exclusively oriented sharp corners and edges, while for the right image with blurred textures and occasional sharp features, the learned dictionary does seem to be composed of the two kinds of elements.

Original: Let the learned dictionary be A .
Revised: Let the learned dictionary be A .

Original: We estimate the representation coefﬁcients as A − 1 Y .
Revised: We estimate the representation coefﬁcients as A − 1 Y .

Original: The third row of Fig.contains the histograms of the coefﬁcients.
Revised: The third row of Fig.contains the histograms of the coefﬁcients.

Original: For both images, the coefﬁcients are sharply concentrated around zero (see also the fourth row for zoomed versions of the portions around zero), and the distribution resembles a typical zero-centered Laplace distribution—which is a good indication of sparsity.
Revised: For both images, the coefﬁcients are sharply concentrated around zero (see also the fourth row for zoomed versions of the portions around zero), and the distribution resembles a typical zero-centered Laplace distribution—which is a good indication of sparsity.

Original: Quantitatively, we calculate the mean sparsity level of the coefﬁcient vectors (i.e., columns of A − 1 Y ) by the metric (cid:107)·(cid:107) 1 / (cid:107)·(cid:107) 2 : for a vector v ∈ R n , (cid:107) v (cid:107) 1 / (cid:107) v (cid:107) 2 ranges from 1 (when v is one-sparse) to √ n (when v is fully dense with elements of equal magnitudes), which serves as a good measure of sparsity level for v .
Revised: Quantitatively, we calculate the mean sparsity level of the coefﬁcient vectors (i.e., columns of A − 1 Y ) by the metric (cid:107)·(cid:107) 1 / (cid:107)·(cid:107) 2 : for a vector v ∈ R n , (cid:107) v (cid:107) 1 / (cid:107) v (cid:107) 2 ranges from 1 (when v is one-sparse) to √ n (when v is fully dense with elements of equal magnitudes), which serves as a good measure of sparsity level for v .

Original: For our two images, the sparsity levels by the norm-ratio metric are 5 . 9135 and 6 .
Revised: For our two images, the sparsity levels by the norm-ratio metric are 5 . 9135 and 6 .

Original: 4339 , respectively, while the fully dense extreme would have a value √ 64 = 8 , suggesting the complete dictionaries we learned are reasonable sparsiﬁcation bases for the two natural images, respectively.
Revised: 4339 , respectively, while the fully dense extreme would have a value √ 64 = 8 , suggesting the complete dictionaries we learned are reasonable sparsiﬁcation bases for the two natural images, respectively.